Satan's Test Case


It all started with an email that I can only paraphrase as the following:

Good morning,

Could you kindly review the unit tests for our project? We wish to verify that the our test cases fully cover the program’s logic.

~ Developer Friend

This sounds straightforward enough, right? No diabolical intention or winking emojis, so I responded:

Sure, I’ll review your tests and report back to you with what I find.

~ Me

The project included a small README document that had instructions on how to run the test suite. So far so good, this is something that every application should have.

I ran the testing command as instructed and was delightfully surprised to see the following output.

<OK> Configuring environment.
<OK> Starting tests.
<OK> 32 Tests run. 32 successes. 0 Failures.
<OK> Cleaning up.
<OK> Done.

It seemed like I was off to a good start so I opened up the file for the test case and started scratching my head.

When we design test cases it is not uncommon to “mock” various parts of the full application so that individual parts can be tested against what is assumed to be properly functioning code.

The test cases in this particular test suite were written against a mock application.

A mock that is, of the entire application.

I thought, no, I hoped, that I was wrong. Whoever had written the tests for this project had created fake test functions that each returned the expected value, and then asserted that it was true in the test case.

Why in the world something like this was written I have no clue, but it existed and I had to break the news to the owner of the application that they didn’t actually have any test coverage. Looking at the code for the application itself, I have two theories for why this mistake ended up being created.

  • First, the setup of the software was too tightly coupled. That is, the way it was written wasn’t modular enough to let parts of it be easily tested. Typically, if you have to mock large parts of your application during testing this is an indicator that you have bigger problems than you know.

  • And second, it was clear that whoever wrote the test cases for this code wasn’t the person who authored it. Through some extreme misunderstanding (of the principles of unit testing) I can only suspect that they failed to understand the design of the application itself and what the task assigned to them was intended to accomplish.

Test Case Anatomy

Enough about code problems, lets talk about what a good test case looks like.

Although there are numerous ways to write tests in every programming language, there are a few common ideas that most programming languages encourage. Under typical, prefered circumstances, test cases have the following layout.

  1. A file that contains one or more groups of tests.
  2. A class name or section title that groups a collection of individual tests cases.
  3. A series of cases that each test that a single function or functionality operates as expected.

Then, each test case has the following structure.

  1. A setup section that instantiates some prerequisite data.
  2. An execution section that runs the piece of code being tested.
  3. A verification section that checks that the produced result meets some kind of fixed constraint.

Also, here are a few general rules of thumb when it comes to writing tests.

  • If you open up a test case in your favorite text editor or IDE, and you have to scroll to see the entire test, then there is something wrong with either the tests or the software itself.
  • Comparing calculated values in tests should be avoided. If you want to make sure that a function calculates the correct value you shouldn’t be risking the possibility that the calculation itself is incorrect.

Here is an example of what a good test case should look like.

def test_fail_authentication_if_user_is_not_active(self):
    user = User.objects.create_user('foo', 'bar', 'baz')
    user.is_active = False
    user.save()
    self.model.objects.create(key='foobar_token', user=user)
    response = self.csrf_client.post(
        self.path, {'example': 'example'},
        HTTP_AUTHORIZATION=self.header_prefix + 'foobar_token'
    )
    assert response.status_code == status.HTTP_401_UNAUTHORIZED

Code snippet courtesy of Django REST Framework

Most software developers have either heard of, or encountered a set of unit tests in which the author placed every assertion within a single method. While this is annoying and is also poor practice, it could be much worse.