I think you can go one step further. Never forget you're serving your customers, and your software has other raison d'etre. You only write software to provide value to them, so think of testing the same way.
Each test has the opportunity cost of writing some part of a new feature for your customers. But so does every minute spent of fixing bugs that would have been caught with more testing, at a fraction of the cost.
There are many ways to improve code quality. Using an automated test suite is only one of them, and while it's one that is widely useful, it is of very limited value in some circumstances and I think for some developers it instils a false sense of security. Not having an automated test suite that covers a particular part of your code does not imply that the code is "untested" or of no value. It just means some other approach is needed in that case.
Not having automated test covering a piece of code does not imply that it's untested at the time it's written, but it sure as hell implies that it's not getting tested when seemingly unrelated feature X gets refactored and unknowingly breaks it.
Tests are only marginally important at the time you're writing the code they test. The real value comes months later when something else causes the test to fail, and now you a: know the code is broken, and b: have a clear specification what what that code was supposed to do.
Sorry, but I simply can't agree with most of that. I do agree that automated tests are more valuable during maintenance than during initial development, though I think they help then too. It's the other details of your comments I'm disputing below.
Firstly, even if automated testing isn't appropriate for a particular part of the code, there should still be other forms of quality checking going on that would pick up a broken feature before the code is accepted, and certainly before the product ships. If this doesn't happen, you're relying on a limited set of automated tests as a substitute for things like proper code reviews and pre-release QA, in which case IMNSHO you're already doomed to ship junk on bad days.
Secondly, if you can break one piece of code by changing a completely unrelated bit of functionality elsewhere, you have other fundamental problems: your code isn't clearly organised with an effective modular design, and your developers demonstrably don't understand how the code works or the implications of the changes they are going to make before they dive in and start editing (or even afterwards). Again, you're already doomed: no amount of unit testing is going to save you from bugs creeping in under such circumstances.
Finally, unit tests are not a clear specification of anything, ever, other than the behaviour of a specific test.
Basically, if you consider automated unit testing a substitute for any of
(a) maintaining a clean design
(b) doing an impact analysis before making changes to existing code
(c) writing and updating proper documentation, including clear specifications, or
(d) proper peer review and QA processes
then I think you're suffering from precisely the false sense of security I mentioned earlier. In many contexts, unit tests can be great for sounding alarm bells early and giving some basic confidence, but even in the most ideal circumstances they can never replace those other parts of the development process.
QA itself is a process failure. If the testers have ever repeated an action more than twice, they should be automated, and you're back to automated testing.
The only QA I've ever worked with that was worthwhile spent their time writing automated tests - they were programmers concentrated in test. Otherwise, you're literally saying 'It would be cheaper to pay this room full of people to do what a machine can do instead of paying 1/10th their number to write the same thing as a test', which is essentially never true.
More concretely, if you use testing to drive your refactors and architecture--as opposed to, say, finding pain points in normal code or actual design time in preproduction--I would be concerned that you are "guardrail programming", as a gentleman put in a talk I saw recently.
When we drive, we don't have guardrails to bounce us back on the road every time we veer off--they're there to protect us against accidents or when something goes seriously wrong with our vehicle. If you told somebody that you drove from city A to city B by hugging the guardrail, they'd say you were nuts.
Similarly, depending on unit tests to do design is strange--they're there to be sure that your code functions according to contract.
I disagree completely, and your comment makes me think you've never seriously used unit testing.
Writing tests makes you think about how pieces of your code interact with each other, dependencies etc.
As an example, if you're trying to test Function A and are finding you need tens of lines of setup code to be able to do so, then that would be a warning sign that you may want to think about refactoring out some of those dependencies
I've seen code in productions apps with comments along the lines of "this isn't optimal, but it's easier to test". Every time I do, I die a little inside.
Unit tests are THE most overrated buzzword of the last 10 years.
If you mean that unit tests have accumulated a lot of dogma over the past few years, and you are saying they are "overrated" because you still need to think about how, what, and why you are testing, I agree.
If you are using your post as an excuse for not using automated testing at all, I completely disagree. That's the bad kind of developer laziness.
On the other hand, I do have to concede that when competing against people who don't use unit testing on the open market, I come off looking like a wizard in terms of what I can accomplish in a reasonable period of time and the sorts of things I can do (successful major changes to large existing code bases you wouldn't even dream of starting), so maybe I shouldn't try so hard to encourage others to use them sensibly. So, I mean, yeah, totally overrated. Have I also mentioned how overrated syntax highlighting is? You should totally just shut it off. Also, fixing compiler warnings are for wusses, and what moron keeps putting -Werr in compilers?
How much of that complexity is self-inflicted? Most of the unit testing advocates I know are also the worst architecture astronauts.
Every line in a codebase has a cost, including tests. I'd rather deal with a code base that's as trim as possible.
I've done unit tests before, but I don't find that they help that much, because they don't solve the most common source of actual production issues: things you didn't think of.
I find they do help there. Having unit tests makes me trust my code better. Confronted with a "it does not behave as I would expect" issue, that trust helps me focus attention away from the implementation of those functions.
Problem with that is that, to get that trust, I need to know that unit tests exist, and, preferably have spent time writing or reading them. Question then is whether that time would not be spent better on reading the existing code. I think that, often, the answer to that is "no", but I cannot really argue that.
Perhaps, it is because writing unit tests puts you explicitly in "break this code (that may not have been written) mode". Writing a unit test that calls a function with some invalid arguments and verifies that it throws is often simpler than reading the code to verify that. Also, unit tests may help in the presence of bug foxes and/or changing requirements. Bug report/Requirements change => code change => unit tests break => free reminder "oops, if we change that, feature X will break".
How do you then know that everything works fine when you do large scale refactoring? Test everything manually? (genuine question, not trying to be snarky).
He doesn't. And I'm not being snarky either. People will say they do, but they don't have any assurance of it. And furthermore, over time they'll learn to stop making these sorts of changes because they don't work, become very cynical about what can be done, and internalize the limitations of not using testing as the limitations of programming itself.
And then these people will be very surprised when I pull off a fairly large-scale invasive refactoring successfully, and deliver product no engineer thought possible.
I'm not hypothesizing; this has been my career path over the past five years, and I have names and faces of the cynical people I'm referring too. You can not do the things I do without testing support. I know you can't, because multiple people who have more raw intelligence than I try and fail.
It is equally true you can't be blind about dogma, 100% coverage being a particularly common bugaboo, but I completely reject the idea that the correct amount of automated testing is zero for any non-trivial project.
I get the impression that the code base on which you pulled off the "large-scale invasive refactoring" was not initially under test, else why would the cynical engineers think it could not be done. So did you have to bring the legacy code under test first?
I'm curious as to what exactly you mean. Can you give some examples? If your're frequently making large-scale changes, I'd spend more time worrying about why you're having such a hard time nailing the requirements down.
If you've only worked on projects with nailed-down requirements, you're probably not working on the sorts of projects most HN people face. The requirements change because the world changes, or our understanding of it. That's the nature of the startup. Stable codebases serving stable needs don't need as much refactoring, that's true. And in those cases units might be a waste of time. But for those of us (the majority, I'd wager, at least around here) who work on fast-moving, highly speculative projects, they are an absolute godsend.
A framework previously designed to work on a single device was ripped apart and several key elements were made to run over a network remotely instead. (That may sound trivial in a sentence, but if anyone ever asks you to do this, you should be very concerned.) The framework was never designed to do this (in fact I dignify it with the term "framework"), and tight coupling and global variables were used throughout. This was not a multi-10-million line behemoth, but it was the result of at least a good man-century of work.
As mentioned in my other post, first I had to bring it under test as is, then de-globalize a lot of things, then run the various bits across the network. Also testing the network application. Also, by the way, releases were still being made and many (though not all) of the intermediate stages needed to still be functional as single devices, and also we desire the system to be as reverse-compatible as possible across versions now spanning over a year of releases. (You do not want to be manually testing that your network server is still compatible with ~15 previous versions of the client.) And there's still many other cases I'm not even going into here where testing was critical.
The task I'm currently working on is taking a configuration API that has ~20,000 existing references to it that is currently effectively in "immediate mode" (changes occur instantly) and turning into something that can be managed transactionally (along with a set of other features) without having to individually audit each of those 20,000 references. Again, I had to start by putting the original code under a microscope, testing it (including bug-for-bug compatibility), then incrementally working out the features I needed and testing them as I go. The new code needs to be as behavior-similar as possible, because history has shown small deviations cause a fine spray of subtle bugs that are really difficult to catch in QA.
I could not do this without automated testing. (Perhaps somebody else could who is way smarter, but I have my doubts.) The tests have already caught so many things. Also, my first approach turned out wrong so I had to take another, but was fortunately able to carry the tests over, because the tests were testing behavior and not implementation. (Also it was the act of writing those tests that revealed the performance issues before the code shipped.)
This isn't a matter of large-scale requirement changes on a given project. This is a matter of wanting to take an existing code base and add new features that nobody thought of when the foundation of the code was being laid down 5-7 years ago. (In fact, had they tried to put this stuff in at the time it would have all turned out to be a YAGNI violation and would have been wrong anyhow.) Also, per your comment in another close-by thread, the foundation was all laid down prior to my employment... not that that would have changed anything.
The assumption that large-scale changes could only come from changing requirements is sort of what I was getting at when I was talking about how the limitations of not-using-testing can end up internalized as the limitations of programming itself.
Might I also just say one more time that testing can indeed be used very stupidly, and tests with net negative value can be very easily written. I understand where some opposition can come from, and I mean that perfectly straight. It is a skill that must be learned, and I am still learning. (For example: Code duplication in tests is just as evil as it is in real code. One recurring pattern I have for testing is a huge pile of data at the top, and a smaller loop at the bottom that drives the test. For instance, testing your user permissions system this way is great; you lay out what your users are, what the queries are, and what the result should be in a big data structure, then just loop through and assert they are equal. Do not type the entire thing out manually.) But it is so worth it.
1) How many lines of code is in that man-century project?
Is the number of lines of code ~proportional to the number of man hours, or lines(man-hours) function is ~ logarithmic?
2) How does your typical project look like (or how does that project look like) in terms of testing vs coding?
Do you spend few months of covering old code by tests and only then start testing?
Or you do "add tests - add features - add tests - add features - ..." cycle?
What's the proportion between time spent on writing tests and writing code?
3) What's the proportion of time you spend directly working (analyzing requirements/testing/writing code) and generally learning (books, HN, etc.)?
4) Do you do most of the work yourself or you mostly leading your team?
5) How do you pick your projects, and when you pick them - what are your relationships with the clients: Fixed contract? Hourly contract? Employment?
So, at the end of the day, you never actually did design-from-scratch work, and instead used tests to verify incremental design improvements (key part: verify not create)?
Starting from scratch does not take into account all the growing pains the previous software hat that made it into the quagmire you have learned to hate.
The sum total of the improvements were not incremental. Testing helped give me a more incremental path, but from the outside you would not have perceived them as such.
I don't do large-scale refactoring. Seriously. Small pieces? Sure.
But I've never, in 15 years of development, had to rewrite half of an application I've already written.
Spending a large amount of extra time and energy, things I don't have an excess of to begin with, for a "might" or a "maybe" seems like a rather poor choice to me.
Aside from the unit tests pb, suboptimal but easy to test code is critical in a lot of situations, like time critical bug fixes or last minute feature addition on a production site.
Most of the time, testing takes more time than writing the code, so throwing optimality under the bus can be the best choice. If it's Good Enough nobody's going to rewrite, but I wouldn't see it as something inherently negative or shameful, it's just a question of priorities.
On the other hand, if your code is full of architectural compromises, special cases and privilege escalation tricks just to allow you to test everything in some particular way, maybe the tail is wagging the dog?
There are many ways we try to improve code quality and make sure we get it right. Automated test suites are only one of them. Software design needs to take multiple factors into account, and letting one of them arbitrarily dominate all others is a dangerous path to take.
If I have a function/module/method buried deeply inside my system such that testing it requires either ten lines of setup code or backdoors ("special cases and privilege escalation tricks") in the deployed code, that might say something interesting about my architecture in either case. Is the code really only ever going to be called from that one place and in that one way, and if so, exactly how valuable is it? Sure, it might be that the only place I currently want to call (say) a weighted modulo 11 checksum is in credit card validation and the context there is I have a third-party payment gateway and a valid order object and all that stuff, but I would still be looking at surfacing the actual calculation in a library module somewhere that I can test it without doing all this setup. I grant you that architecture is only ever easy in retrospect - that's why we refactor - but I don't think that represents an architectural compromise.
If all your algorithms are as trivial as calculating a weighted modulo 11 checksum, then the sort of case I'm thinking of doesn't apply. However, in real code, we sometimes have to model situations and solve problems that are inherently complex. The algorithms and data structures we work with will necessarily reflect that essential complexity, and ultimately so will our code.
Beyond a certain point, I think automated tests that give simple yes/no answers are no longer a particularly effective way to test certain types of complex algorithm. Sometimes there are just too many possible inputs and interactions between different effects to get a sensible level of coverage and draw any useful conclusions from that kind of testing alone. You might still have some automated tests, but they are more like integration tests than unit tests at that point.
Internally, you could try writing almost-unit-tests for the implementation details, but then you get into the usual concerns about backdoor access and tying tests too closely to implementation details that might change frequently. Alternatively, some form of careful algorithm design with systematic formal proof might be called for. Maybe instrumenting the code and checking the actual values at key points will highlight errors that aren't yet manifesting as faults, things that a boolean automated test would miss because they haven't violated some arbitrary threshold but which form an unexpected pattern to a knowledgable human observer. However, in these cases, you really want the code to be as simple as possible, and hooks to permit internal access to run some automated test cases as well could cause an awful lot of clutter.
> If all your algorithms are as trivial as calculating a weighted modulo 11 checksum, then the sort of case I'm thinking of doesn't apply.
My estimate is that 98% of all programming everywhere is as algorithmically trivial as calculating a weighted modulo 11 checksum - probably more so - and it acquires its bugginess from accidental complexity due to poor factoring, and from conflicts at interfaces. Test-driven development is pretty good, in my experience, at helping ameliorate both these problems.
Of course, that doesn't mean I actually do it 100% or even 80% of the time. I'm happy to agree that it's no panacea: testing threads and UIs are particular pain points for me, and usually I substitute with either Thinking Really Hard or just Not Changing Stuff As Much
Formal proof for me is stuff I learnt at college, forgot subsequently, and keep meaning to reread up on. Thank you for prompting it back up my TODO list
> My estimate is that 98% of all programming everywhere is as algorithmically trivial as calculating a weighted modulo 11 checksum - probably more so - and it acquires its bugginess from accidental complexity due to poor factoring, and from conflicts at interfaces.
I think it depends a lot on your field.
If you're working in a field that is mostly databases and UI code, with a typical schema and most user interaction done via forms and maybe the occasional dashboard-type graphic, then 98% might even be conservative.
On the other hand, if you're doing some serious data munging within your code, 98% could be off by an order of magnitude. That work might be number crunching in the core of a mathematical modelling application, more advanced UI such as parsing a written language or rendering a complex visualisation, other I/O with non-trivial data processing like encryption, compression or multimedia encoding, and no doubt many other fields too.
Generalising from one person's individual experience is always dangerous in programming. I've noticed that developers who come from the DB/business apps world often underestimate how many other programming fields there are. Meanwhile, programmers who delight in mathematical intricacies and low-level hackery often forget that most widely-used practical applications, at least outside of embedded code, are basically a database with some sort of UI on top. And no, the irony that I have just generalised from my own personal experience is not lost on me. :-)
This can lead to awkward situations where practical problems that are faced all the time by one group are casually dismissed by another group as a situation you should never be in that is obviously due to some sort of bad design or newbie programmer error. I'm pretty sure a lot of the more-heat-than-light discussions that surround controversial processes like TDD ultimately come down to people with very different backgrounds making very different assumptions.
Sure, but context shapes behavior. People should be eating better too, but that's a lot easier to do when you have a fridge full of vegetables than a cupboard full of Doritos. Test-driven development forces me to think about code from the outside first.
My original disagreement was more along the lines of "unit testing is more important for design than for QA" and less "unit testing is important".
I certainly support unit testing, as its essential--and anyone telling you otherwise is bonkers--to ensuring that code follows contract.
That said, if unit testing was great for design but didn't spot errors, it'd be useless. Whereas, if it was useless for design and good for errors, that's okay, because I can do the design work myself.
