Testing is a controversial topic. People have strong convictions about testing approaches.
Test Driven Development is the most prominent example. Clear empirical evidence is missing which invites strong claims. I advocate for an economic perspective towards testing. Secondarily, I claim that focussing too much on unit tests is not the most economic approach. I coin this testing philosophy Lean Testing.
The main argument is as follows. Different kinds of tests have different costs and benefits. You have finite resources to distribute into testing. You want to get the most out of your tests, so use the most economic testing approach. For many domains, e.g. GUIs, other kinds of tests than unit tests give you a better bang for the buck.
Confidence and Tests
The article ‘Write tests. Not too many. Mostly integration’ and the related video by Kent C. Dodds express the ideas behind Lean Testing well. He introduces three dimensions with which to measure tests:
• Cost (cheap vs. expensive)
• Speed (fast vs. slow)
• Confidence (low vs. high or click doesn’t work vs. checkout doesn’t work)
The following is the ‘Testing Trophy’ suggesting how to distribute your testing resources.
Compared to Fowler’s ‘Testing Pyramid’ confidence as a dimension is added. Another difference is that unit tests do not cover the largest area.
One of Kent C. Dodds’ major insights is that you should actually consider the confidence a test gives you.
Return on Investment of Tests
The Return on investment (ROI) of an end-to-end test, and to a lesser degree an integration test, is substantially higher than that of a unit test. This is because an end-to-end test covers a much greater area of the code base. Even taking into account higher costs, it provides disproportionally more confidence.
Plus, end-to-end tests test the critical paths that your users actually take. Whereas unit tests may test corner cases that are never or very seldomly encountered in practice. The individual parts may work but the whole might not. The previous points can be found in ‘Unit Test Fetish’ by Martin Sústrik.
Further, Kent C. Dodds claims that integration tests provide the best balance of cost, speed and confidence. I subscribe to that claim. We don’t have empirical evidence showing that this is actually true, unfortunately. Still, my argument goes like this: End-to-end tests provide the greatest confidence. If they weren’t so costly to write and slow to run we would use lots more end-to-end tests. Although better tools like Cypressmitigate these downsides. Unit tests are less costly to write and faster to run but they test only a small part that might not even be critical. Integration tests lie somewhere between unit tests and end-to-end tests so they provide the best balance.
Aside: Terminology
The term ‘integration test’, and even more so ‘end-to-end test’, seems to generate intense fears in some people. Such tests are supposed to be brittle, hard-to-setup and slow-to-run. The main idea is to just not mock so much.
In the React context of Kent C. Dodd’s article integration testing refers to not using shallow rendering. An integration test covers several components at once. Such a test is easier to write and more stable since you do not have to mock so much and you are less likely to test implementation details.
In the backend world, an integration test would run against a real database and make real HTTP requests (to your controller endpoints). It is no problem to spin up a Docker database container beforehand and have its state reset after each test. Again, these tests run fast, are easy to write, reliable and resilient against code changes.
Code Coverage
Another point is that code coverage has diminishing returns. In practice, most agree as most projects set the lower bound for coverage to around 80%. There is actually supporting research such as ‘Exploding Software-Engineering Myths’. What follows are general arguments.
Even with 100% code coverage you trust your dependencies. They can, in principle, have 0% code coverage.
For many products, it is acceptable to have the common cases work but not the exotic ones (‘Unit Test Fetish’). If you miss a corner case bug due to low code coverage that affects 0.1% of your users you might survive. If your time to market increases due to high code coverage demands you might not survive. And “just because you ran a function or ran a line does not mean it will work for the range of inputs you are allowing” (source).
Code Quality and Unit Tests
There is the claim that making your code unit-testable will improve its quality. Many arguments and some empirical evidence in favor of that claim exist so I will put light on the other side.
Photo by zhu xihua on Unsplash
The article ‘Unit Test Fetish’ states that unit tests are an anti-architecturedevice. Architecture is what makes software able to change. Unit tests ossify the internal structure of the code. Here is an example:
Imagine you have three components, A, B and C. You have written an extensive unit test suite to test them. Later on you decide to refactor the architecture so that functionality of B will be split among A and C. You now have two new components with different interfaces. All the unit tests are suddenly rendered useless. Some test code may be reused but all in all the entire test suite has to be rewritten.
This means that unit tests increase maintenance liabilities because they are less resilient against code changes. Coupling between modules and their tests is introduced! Tests are system modules as well. See ‘Why Most Unit Testing is Waste’ for these points.
There are also some psychological arguments. For example, if you value unit-testability, you would prefer a program design that is easier to test than a design that is harder to test but is otherwise better, because you know that you’ll spend a lot more time writing tests. Some further points can be found in ‘Giving Up on Test-First Development’.
The article ‘Test-induced Design Damage’ by David Heinemeier Hansson claims that to accommodate unit testing objectives, code is worsened through otherwise needless indirection. The question is if extra indirection and decoupled code is always better. Does it not have a cost? What if you decouple two components that are always used together. Was it worth decoupling them? You can claim that indirection is always worth it but you cannot, at least, dismiss harder navigation inside the code base and during run-time.
Conclusion
Lean Testing takes an economic point of view to reconsider the Return on Investment of unit tests. Consider the confidence a test provides. Integration tests provide the best balance between cost, speed and confidence. Be careful about code coverage as too high aspirations there are likely counter-productive. Be skeptical about the code-quality improving powers of making code unit-testable.
To make it clear, I do not advocate to never write unit tests. I hope that I provided a fresh perspective on testing. As a future article, I plan to present how to concretely implement a good integration test for both a frontend and backend project.
If you desire clear, albeit unnuanced, instructions, here is what you should do: Use a typed language. Focus on integration and end-to-end tests. Use unit tests only where they make sense (e.g. pure algorithmic code with complex corner cases). Be economic. Be lean.
https://blog.usejournal.com/lean-testing-or-why-unit-tests-are-worse-than-you-think-b6500139a009