As the title says.
Apropros unit testing. Is it increasing software quality? It it decreasing time-to-market? Is it having the opposite effects? Is it “best practice” (and therefore doomed to fail 90% of the time)? What are the alternatives? How do they compare? All this and more to discuss in this new monster argumentdiscussion thread!
Cas 
in the business software i write as a day job unit tests are invaluable.
They provide both reality checks as the projects get bigger or aged… and allow new developers to come in and make changes with the confidence that they have not broken existing functionality.
In most cases we unit test our utility code and interface points…
Can you write unit tests for a games OpenGL interface? i dont know… would be interested to know if there is a framework out there for it.
Now as a complete confession. In my game im working on … i have not written a single test yet. Frankly the code is too in flux and im the only developer. So there is likely a reasonable argument that size of team equals greater value from unit tests.
@princec: unit tests are invaluable in LibStruct - but… that’s the first project I am forced to use them to check for regressions. There is simply too much that can go wrong.
In all my other projects, I couldn’t care less about proper unit tests, let alone decent coverage.
I’ve never used unit-testing before, but it sounds useful for those reasons jmguillemette mentioned.
Most of the bugs that I make for myself are due to threading, and apparently unit testing threading problems is very difficult:
We use on in the day job on a huge enterprise scale. However, we’re a rapidly moving software team so the problem is fragility and false negatives. That along with the worth in regression is pretty suspect, looking at the statistics across bugs coming back in, support requests vs the coverage in the code there isn’t any correlation between high coverage = low bugs/support.
Cheers,
Kev
This was my finding too: no correlation between coverage and reported issues.
What I did discover was a very high correlation between tests written and time it took to get the software released. Unit testing at least doubled time-to-market but found virtually no bugs that trivial QA didn’t find and never found bugs that weren’t suitable for unit testing (which is most of them).
My conclusion is that TDD is a self-licking lollipop which falls into the category “best practices” (this is worth a read). It makes managers look like they’ve got their arses covered; it makes developers look like they’re busy; it makes all the right “noises”. What it doesn’t seem to be doing is getting software out quicker, or particularly any higher quality than by any other means.
What those other means might be I leave to further commentary
@Riven - there’s a time and a place for unit tests and I think you’d be one of those who knows when and where they’re needed 
Cas
I think unit testing suddenly becomes important in a lot of cases when using dynamic languages, which are popular, thus leading to popularity of practices like TDD etc. Here’s one such post: http://williamedwardscoder.tumblr.com/post/54327549368/dynamic-languages-are-unmaintainable-and-unit
I agree that it quickly becomes much less useful the more the language itself checks everything for you, disallows you from doing a lot of things that don’t make sense, and provides inline documentation of itself.
I write test units (like things) in one and exactly one situation and never than other. That situation generalizes to code that writes code.
Another argument for techniques like TDD which is usually pushed is that they help you design cleaner and better public API’s but personally not seen any advantage here over manually doing so (in fact its faster to do so manually).
When it comes to writing games (as opposed to libraries and engine code), unit testing is a complete waste of time. Jonathan Blow sums up the problem pretty nicely (including arguing pretty strongly against spending time commenting code and most types of optimisations).
Frankly, I often forget what is the difference between unit and functional tests, so I have to recall that permanently… :-\ Can anybody also mention about mock frameworks. Nice thing in the unit testing, isn’t it?
I’ve seen one area where it has a vague positive, which is that if you’re doing TDD effectively, you’ll have a separate programmer writing the tests against the APIs he’s given. Most code instantly fails at the first hurdle by simply having no specification or documentation to speak of, and the tester can simply throw it back and say, “I can’t test this as I don’t know what you think it’s supposed to do.”
Of course, largely totally irrelevant outside of fully reusable, releasable, versionable library code… which is most code.
Cas
You can use “unit tests” without going completely in a TDD process.
I try to have a pragmatic approach in my job or for personnal projects that is the following:
Critical because i haven’t the right to introduce a bug
Difficult because i will have difficulty to enhance theses parts without taking some risks to break something.
In this stage, i not write a billions of unit tests and i try to keep reasonnable.
After this developpement process, become the maintenance time when a bug is found.
When a bug is found in production, i consider we never see it in all stages we can have (dev, internal test, client test)
So it’s not a “simple bug”, and in this case, we wrote a unit test that can check this precise bug that protect us to have it again.
With this approach, i can reduce costs of unit tests (in term of time and effort) and trying to maximize theses benefits.
Best regards,
Séb.
I find unit tests helpful in my day job where I build frameworks or utilities for other people. The other side of my job is developer support and we have tests there too, to ensure our samples still work with the latest server code and API changes (which are always “backwards compatible”). Hell, I even have some unit tests for my user guides, just to check the code snippets compile and run without errors.
I find it very hard to write unit tests at home. In my games I have almost no unit tests, except for set pieces like AI, collision detection and so on.
I’m doing a lot of server side development for my day job that usually runs as a kind of job or web service or is used by others to access some remote data. For these things, i write unit tests to see if my code does what it’s supposed to do during development and they also help others to understand how to use that stuff. You could easily accomplish this with some test classes with simple main-methods as well. But using unit tests instead makes it easier for others to identify them in the project.
But it almost never happens that they unveil some bug that otherwise would have gone unnoticed…well, except for today, but that doesn’t count… TDD as a pure doctrine is rather pointless IMHO.
See: white box vs. black box testing. White box is good. Black box is usually a waste of time.
The Rust language relies very heavily on unit and integration testing. Countless bugs have been caught and fixed before being merged into the master branch. Because of this it lowers the barrier for contribution considerably, as your changes can be easily verified without wasting enyone’s time. The same bug very rarely happens twice, and when it does you can be sure it’s never the same cause (because regression tests).
In fact, unit testing, integration testing, and benchmarking functionality has been well-defined and built into the compiler to avoid the bloat and fragmentation among external frameworks (and it also allows tests to be written alongside implementations without making a mess). Even code examples in the documentation are built and run to make sure that it all stays up to date.
I’d say it’s a very important practice in large open source projects to lower the barrier for contributions, but for small projects with only a handful of people who understand most of the system, it won’t catch enough bugs to be worth the time it takes to write all the tests.
… or even rewrite the tests.
IME tests take about as long as the original code to develop… and maintain. Is that really worth it?
Cas
In environments where regressions cost thousands or millions, I think I’d unit-test every nook and cranny.
I think I’d be using a language it was harder to make mistakes in, too
Cas
I used to work for small companies that tested their code by clicking though a web UI, filling out forms, observing what happened. Slowly but surely, almost every piece of business logic (at least subtily) broke.
It involved administration of the healthcare of about a million people, so it… mattered, although lives where not at stake. Ever since I don’t quite take the reasoning seriously that unit tests are too much of a burden because they merely catch dirt stupid, easily discovered bugs. These silly bugs won’t be discovered until either the app falls apart or some bulk operation went bezerk and did real damage.
Things will always break in complex ways, but that doesn’t mean we shouldn’t safeguard the simple logic from off by one errors and the like.