Mar 27, 2011

Testing: Why Bother? - Introduction

I stumbled upon an interesting discussion this week in the testing-in-python mailing list, asking how to explain why testing is important to a "we do not write tests" audience. Because this is something I had to do a few times already, I thought about writing a blog post about it (actually, a series of posts...)

Testing? What do you mean by that?
Well, there's a lot of things that can be related as "testing" when talking about software development. There's TDD, Unit tests, Integration tests, having a QA engineer test your code, and more.
In these posts, I'm talking about justifying writing any kind of automated tests that can be run through a CI (Continuous Integration) system after each commit. It doesn't matter if they were originally written during practicing TDD, the important thing is making sure there's something that's always there to test nothing was broken after each little changeset in the code.

So, what's the deal? Why do I need to write tests?
Well, the advantages of testing your software are usually clear and aren't really debatable.
Every one will agree that when you write and run automated tests for your code, you become more confident that your code works, and when you run your tests after each change you make you can gain confidence that you didn't break anything.
The problem is, though these advantages are pretty much obvious and clear to anyone, the majority of developers do NOT write automated tests.

But, if testing is so awesome, why wouldn't everyone write tests?
Well, there are 2 kinds of anti-testing developers (that's how I'm going to refer to them from now on).
One type of anti-testers, are the kind that simply lack the education. Most programmers don't learn how to write tests as part as their formal education, and therefore when they start at their first software industry job, they usually don't write tests unless exposed to this subject as part of the company's training and mentoring. Because concepts like TDD and CI are relatively new, a lot of old-fashioned team leaders and R&D managers are not exposed to the importance of testing as well, and they don't demand this from new employees and don't include this in training them to the job.
These kind of anti-testing developers are the relatively easy kind. They did not already develop antagonism towards testing, and they may be easily persuaded by fellow pro-testing developers, teammates, managers or external preachers.
The other type of anti-testing developers, is the type that I'm targeting these blog posts towards. These are developers that are already exposed to the concepts, maybe even experimented with testing a little bit in the past, but claim to have some solid reasons why testing is a waste of time for them, and simply do not believe in it. There's usually a list of excuses they give to justify their hostility towards testing.

The list of excuses
In my experience as a pro-testing preaching developer, I've come across some various types of resistance. I've collected a list of common excuses for justifying the anti-testing approach:

In my next posts, I'll try to attend all of these excuses, and clarify why I think they are plain misconceptions.

Can you think of any more excuses? Please let me know in the comments.


  1. My excuse is that my code is changed by customer so much during the time of writing code that my tests will be of no use because program will have to run differently during it development. (I'm working in company that is creating software on contracts with other companies and not something to sell to masses, in PHP)

  2. In my experience the value you can derive from testing depends a lot on the situation.

    If I have a clear spec in mind (ie. I'm writing a parser or something) I really like to use TDD. The advantages are just too great to pass.

    If I need to code something more vague (ie. HTML5 drawing app), the value of testing diminishes a bit or at least changes form.

    In a recent solo project of mine I ended up doing a compromise. I modularized the code extremely well. I also added contracts (ie. attr must not be null and within this and that range...) to the vulnerable bits.

    This worked out pretty nicely. Some of the code is nigh-impossible to test (how do you test interactive canvas perf while using the app?). It might be possible somehow but perhaps not worth the effort.

    I believe I would have benefited a lot by having a nice acceptance testing framework at my disposal during the development. This would have helped me to solidify the UI part of the app early on.

    I've sketched out chaining syntax for this (given('elem').is('selected').then('assert here')). Just need to get around implementing it.

    I think another cool way to complement this would be to come up with some kind of UI fuzzer. It would just use the app randomly till it runs into an error. This is pretty much what your users will do anyhow. Esp. in case of JS this should provide some tangible value.

  3. Thank you for the comments. Very interesting.
    @tomt160, your excuse is similar to the fifth one in the list. I'll try to address it in the relevant post.
    @Juho yours is a mixture of the second excuse (too hard) and the last (hard to tell how code should behave). Anyhow it seems that you have found ideas to test at least part of the code so that's pretty good.

  4. I can sign under the 1, 2, 5, 7, 8 excuses. And for the 8th I would change the word code to user.

  5. Well, for those who like to use excuse number 5 and 7. We recently completed a project and put in production developed by one guy, close to 1000 hours of work + another 300 hours of other folks who integrated this into our platforms. The code put together by this guy translated a proprietary print language into IBMs AFP mainframe print language (thousand of pages of documentation...old and complex technology) to actually print on mainframe printers. We processed thru this engine easily 300 million pages of statements in production. This software incurred during our production runs on ONE bug... and I will say it was a spec issue, not really a coder error.
    The whole code was written using heavy refactoring AND TDD. I used a contractor to put this piece together, a guy from Ukraine. I use this story with all engineers in the company when they complain about how tough it is to do TDD, or question the value.