Why software tests need concrete failure cases

A green test only proves itself

A test is a small program that calls another program and compares the result with an expectation. If it passes, that means: in this one case, what came out was what the test expected.

An uncomfortable possibility follows from this: a test can be green because it checks nothing. It calls something, expects nothing concrete, and stays green even when the code under test no longer exists.

// Proves nothing: only checks that nothing was thrown
public function testBerechnung(): void
{
    $ergebnis = $this->rechner->brutto(1000);
    self::assertNotNull($ergebnis);
}

// Proves something: states input and expected result
public function testBruttoMitRegelsatz(): void
{
    self::assertSame(1190, $this->rechner->brutto(1000));
}

// Proves the failure case: this is where the real value lies
public function testNegativerBetragWirdAbgewiesen(): void
{
    $this->expectException(InvalidArgumentException::class);
    $this->rechner->brutto(-1);
}

The counter-check is part of it

A new test for a bug that has been found only counts as one if it fails WITHOUT the fix. That sounds obvious, but it is skipped regularly — and then you have a test that would not have noticed the bug.

We therefore do this explicitly: revert the fix, run the test, see red, reapply the fix, see green. Only then does it count. The same applies to tests with placeholders: if the placeholder dictates the result that is supposed to be checked, the test checks the placeholder.

Which cases are worth it

• Business rules with boundaries: the last valid and the first invalid value, not an arbitrary one from the middle.
• Everything to do with money, permissions or deadlines.
• Every bug that has occurred once — so that it does not come back.
• Steps that change several records at once: either everything applies or nothing does.

What is rarely worth it: tests for code without decisions, such as pure passing through of values. There the test costs maintenance and finds nothing.