The Unit Testing Advocacy & Derision Thread

General Discussions
#1

I personally think unit testing is a waste of time. I was fortunate enough to work on a large project which had a vast amount of unit testing in it. I noticed after a year working there that we spent more time on the unit tests than on the code itself, and in all the time I worked there, the unit tests only found 3 bugs ahead of production. After roughly doubling development time (conservative guess; I’d put it at triple).

IMHO unit testing is papering over the cracks in design and specification. Wherever unit testing is being used today would be better served by, for example, design by contract.

Flame on!

Cas :slight_smile:

#2

Ooo, controversial!

What language was this in btw? And does your opinion cover all usages of unit tests, or just with that particular language?

Also, were these unit tests, or functional tests but written in a unit testing framework like junit?

#3

Yeah!

But before I make any kind of comment, I want to clear something up. By unit test you mean only the test of something small (like a single method)? Or do you also include functional tests in your war plan? (a test representing for example an entire use case which touches large chunks of your code to test if it works as expected given certain parameters).

I have to ask because the definition of unit test is a bit vague nowadays. Many people don’t actually refer to testing something small, but only to a test that can be invoked through junit or testng (I have to admit, I make that same mistake).

#4

'twas JUnit (and using JMock or something for a few of them), on a plain old Java desktop application.

@gimbal - I mean unit testing as is traditionally defined - a unit being one software component, which in Java, tends to be a single class*. This leads to designs that aid unit testing, which in turn leads to designs that I would not ordinarily design, which again, I’m not sure is necessarily a good thing. Though of course proponents of unit testing claim that designing classes just so they can be tested with unit tests is a Good Thing.

As soon as you’re getting into the testing of the interactions between two or more classes you’re into system testing territory and I’ve seen plenty of people abusing unit tests which do exactly that, and is yet another reason why unit testing usually seems to fail to do quite what it was supposed to do.

And of course there is frequently the observation that testing components completely in isolation does not seem to actually find many bugs. I merely observed this in my own experiences of the project I was working on. Before I get too smug about that though it would be wise to consider that the reason the unit tests never found any bugs is because they’d been in place for so long. Perhaps.

Cas :slight_smile:

  • Though really a single file is probably more appropriate, as classes frequently contain inner classes to help them along.
#5

I’ve never used unit testing in any other language so I couldn’t comment but I suspect the reasoning and outcomes are the same. And they were supposed to be unit tests but many of them were largely irrelevant or accidentally testing more than one public class by design.

Cas :slight_smile:

#6

Since this is about unit tests in their intended form, I can only agree with you. Unit testing small things tend to cause an insane amount of them to be created in any half-decent application which eventually only become a maintenance nightmare. Nice idea with good intentions, but in my opinion not really “realistic” in the long run. I especially dislike ruling that exists around it, like “if you fix a bug you MUST prove it with a unit test”. Such black and white rules tend to lead to utterly useless tests…

I much prefer to stick to the larger functional tests; in fact they are the tool I use to actually develop most (enterprise) applications; all from the tests (which tend to be in the form of one “happy flow” and a great number of tests that deal with failure paths). I tend to have 90% of the code properly tested before I even deploy a single thing, the remaining 10% is usually configuration and environment related (like FTP transfers not being handled properly, etc. etc.) or something as minor as a mistake in a web resource, and not so often for example an architectural mistake.

The real benefit comes when RFCs come into play that actually alter architecture. After modifying the application I end up with a number of tests that fail because their test conditions no longer apply; usually only tests from the same test suite by the way. Through adapting them to the changes I have a far greater insurance that I actually did it right than if I didn’t have the tests to inform me of my misgivings (and perhaps even design flaws; when it is difficult to get something into a test I treat that as something suspect). Plus I have again the benefit of seeing it all happen by simply running a specific test through testng and seeing what happens under the set conditions, either through logging generated or by stepping through. I wouldn’t have to for example deploy, go through a web interface and follow a very specific routine to get things to happen, the test does it as a sort of prebuilt script.

I don’t diss unit tests entirely, I just don’t use them for what the programming gods intended. In for example graphically oriented applications; be it a game or for example a Swing application, I tend to use them to quickly try out code before I start to see them work when firing up the application/game itself. Just to check the numbers and make sure I didn’t screw up on calculations and such. Call them disposable unit tests - saves plenty of time.

#7

Humm…I’m old skool. I tend to use this antiquated notion once called design. Then implement and test hedge cases and stick a fork in it.

Seriously, this is one of those notions than have gone way out of control. It should be pretty obvious when some white or black box tested is needed or not. I tend to think that some modern CS notions all stem from assuming that most programmers are monkeys that couldn’t get work attempt to recreate the works of W.S. The problem is that following these kinds of practices tend to create monkeys rather than computer scientist…so it a self fulfilling kinda thing.

#8

Its certainly an interesting opinion. Me personally I don’t see how unit tests would be used as a replacement for design (if I understand it properly). scratches ear. Like anything they’re a tool which you can use for good or for bad, the birth of the concept and the overly excited adoption (which was pretty similar to how XML was abused) didn’t change anything IMO. Ook, eeeeeep!

runs off to find a banana

#9

I think Roquen is on my wavelength.

Cas :slight_smile:

#10

@gimbal: With a proper understanding of the problem and a design spec of sub-system, it should be obvious where (and if) given types of testing are needed or not. Too much focus is placed on proper “software engineering techniques” (monkey programming) and too little on problem solving, which IMHO is really what computer science is all about.

#11

Aren’t we simply talking about over-doing unit testing?

When I write some complex method or class, I like to write a small test that verifies some normal cases and some corner cases. This gets rid of most off-by-one bugs. I never unit-test highlevel constructs, as the number of potential (conflicting) parameters spiral out of control: it’s indeed a waste of time. That doesn’t mean that it isn’t a valuable tool to quickly check to see whether a method fails to do what you think it should do, as it might also reveal that your understanding of the method is wrong. Whatever it is, this trivial testing will either find obvious bugs or inform you on how to use it.

You might also argue that unittests are a basic form of documentation. :persecutioncomplex:

#12

Agreed. I think this overlaps a bit with the “design pattern” topic of the general rant thread :slight_smile:

Unit tests… no, too small. A javadoc suffices more. But the larger tests I was yapping about certainly are a form of documentation - for me; after a year I forget the details of the code, but the automated tests always get me back on track quickly.

#13

A unit test is just more code, not really an explanation of what you’re trying to achieve. It looks to me as if unit testing is being overzealously applied to make up for a lack of take-up from design-by-contract techniques. And failing. Everywhere I see it being used in a corporate environment, it’s all about ass-covering, “not breaking the build” (haha as if), following procedure (“we do it this way here”), etc. rather than any intelligent application of unit testing as per Riven’s usage - that is, just testing a few little odd bits here and there that are complicated.

Cas :slight_smile:

#14

The most important test is looking at the sourcecode. No joke, this is seriously undervalued.

Anyway. I think it the testmethod depends heavily on how the development is structured.
The less a programmer has influence and responsibility for the complete product, the more paralel testing is needed.
For the programmer is working under a different contract (of involvement) than a developer in a 2 person team.

Ive seen code that could never have run. Wich means that the programmer never ever called the method before in a testenvironment.
More like coding in a style “if eclipse does not show red lines, its fine to check it in, the tester will give feedback if it does not work”
NOTHING should ever leave the dev- computer without beeing run at least once in a standard case.

Tesing a component by itself is important when integrating this component adds a huge complexity to it, such that initiating and tracing bugs is more timewasting than
writing a test-driver (and testcases to check borderconditions) for it.

Most bugs arise either in novel aproches to solve a problem (doing something methodically wrong)
or during maintainance when repetatively extending parts to cover additional requirements - using other parts as a template
(copy pasting and adjusting other codeparts -> there is the danger of ASCII degeneration, where copying Text might loose single bits)

#15

design-by-contract is awesome. It’s a shame that I’ve never liked any languages that fully incorporate the notion, but that’s neither here-nor-there.

As an additional aside, the JSR that deals with allowing more type annotations have a number of examples that ‘hint’ that there may be some type contracts headed our way, such as @NonNull, @Nullable and @Immutable.

#16

If you seen this applied the way you described here, it wasn’t right. Unit testing is mostly used in TDD (test driven developpement) and it has been demonstrated that it can cut down the total developpement time of rather big project by factor 2. On corporate project, assuming that every programer isn’t a A+ class, it helps to avoid 4000 lines methods and enforce decoupling of the different parts of the application since it force the application to be testable, it’s a mean to obtain quality, not just “ass-covering”.

Here’s a good reference :
http://www.agiledata.org/essays/tdd.html

Enjoy :slight_smile:

#17

My empirical observations were that it increased coding time by at least a factor of two. Every line of code is a line of code that needs maintaining; the unit tests were as big as the code they were testing.

My key observation was the number of actual bugs the unit tests found. That is, about 3, in a year. The buglist itself though stretched for pages and pages. This leads me to think that all the real work of testing working software lies elsewhere.

Cas :slight_smile:

#18

You see all kinds of very questionable studies…or at the bare minimum don’t match anything in my experience. An example is the stupid “programmer pair” thing. I find it very hard to believe that it’s a win.

#19

You aren’t doing it right.

Anyone who makes a programming language compiler/parser: tests cases are absolutely critical. They aren’t just some checkbox item. They are critical in supporting the thousands of different types of code that you’re expected to handle. Every major compiler has a large bank of tests cases to vet out every release. Web browsers are a variant of this.

Also, data structures. At various points, I’ve needed to design custom internal data structures for a given application. The test cases were very useful first in writing the data structure and testing out edge cases, but the bigger benefit was in stopping other developers from breaking things.

Certain code doesn’t mesh will with unit tests: user interface code is a good example. All the code needed to wire up interfaces and handle user input and update screen elements doesn’t lend itself to meaningful test cases.

#20

Language front-ends, back-ends and everything in the middle is a very different kind of animal…and yeah, in this case you need LOTS of test unit like constructs. But that’s a whole different animal. It’s pretty rare to have sets of sub-systems with this kind of inter/intra dependance.