Test-driven development is hard - here's the untold truth
Image credits to https://testsigma.com/blog/ai-driven-test-automation/These days you read a ton of articles about all the advantages of test-driven development (TDD), and hear a lot of talks at tech conferences that tell you to: "Do the tests!", and how cool it is to do them.
And you know what? Unfortunately, they are right (not necessarily about the "cool" part, but about the useful part).
Tests are a must!
The typical advantages we list when it comes to talking about TDD are real:
You write better software.
You have protection from breaking the world when new features are introduced.
Your software is self-documented.
You avoid over-engineering.
Even if I've always agreed with these advantages, there was a time when I thought that I didn't need TDD to write good and maintainable software.
Of course, now, I know I was wrong, but why did I have this idea despite the shiny magic of the pros? There's only one reason: it costs a lot.
The Cost
It costs a lot! Someone is probably thinking: "But, it costs even more if you don't do the tests" - and this is right, too.
These two costs come at different times:
You do TDD - you have a cost now.
You don't do TDD - you will have a cost in the future.
So, how do we come out of this impasse?
The most effective way to get something done is by doing it as naturally as possible.
The nature of people is to be lazy (here, software developers are the best performers) and greedy, so you have to find a way of reducing the costs now. It's easy to say, but so hard to do!
Here, I will share my experience and what has worked for me in turning the benefit/cost ratio in my favor. But, before I do that, let's analyze some typical difficulties in applying TDD.
Are You Able to Test the Sum of Two Numbers?
Generally speaking, theory is not optional. You have to master it to master the practice. However, trying to apply all the theoretical knowledge you've previously acquired at once could have an overwhelming effect.
The typical theory lesson on TDD starts with something like this:
Typical starting lesson on TDDAnd here you are like: "That's easy!"
Then comes this:
Red ➡ green ➡ refactor cycle.
Unit, acceptance, regression, and integration tests.
Mocking, stubs, and fakes.
If you are lucky (or maybe unlucky), someone will tell you about contract testing.
And, if you are very lucky (or maybe very unlucky) you will touch legacy codebase refactoring.
The going gets tough, but you are an experienced developer and all these concepts are not that hard to handle for you.
Then class ends; you go home, and throughout the next days you diligently do some code katas to fix the concepts you just learned. So far so good.
The Struggle is Real
Next comes a real-world project, with real deadlines and real timing costs - but you are motivated to apply your shiny new TDD.
You start thinking about the architecture of your software and start writing tests for the first class and the class itself - let's call it Class1.
Now, you think about the first user of Class1, let's call it UsageOfAClass, and again you test and write it. Class1 is a collaborator of UsageOfAClass, so are you going to mock it? Ok, let's mock it.
But, what about the real interactions of Class1 and UsageOfAClass? Maybe you should test them all as well? Let's do it.
At this point, inside of you, you start hearing a little voice that says: "I would develop much faster if I didn't have to write these tests". You don't listen to this evil voice and proceed straight to the next test.
Class2 will be used by UsageOfAClass and it persists itself inside a db. So, do we have to test Class2, its interaction with UsageOfAClass, and the persistence in the db?
But wait… did anyone mention how to cope with I/O testing during the TDD theory class?
The theory behind TDD is not that hard to understand but applying it to the real world can be really complex if you don't approach it the right way.
Just Do It
We should always keep in mind that theory must be bent to our needs and not the contrary.
The main goal is to get the job done. So my advice is: just do it!
Start simple and just do your task up to the end. Then, when you get stuck in some theoretical mind loop like:
Is this a unit or an integration test?
Should I mock it or not?
Here I should write a new collaborator, so a brand new suite of infinite unit tests, just to write "hey, banana".
Just forget about theory for a while and take a step forward. Just do it as it comes! Once you are done with your task, have a look back at your work. Looking back at the completed job, it will be much easier to analyze what would have been the right thing to do.
Practical TDD
Just do it. By the way, I think this is also the right approach to TDD.
What was wrong with how we built Class1, Class2, and UsageOfAClass? The approach.
This is a bottom-up approach:
Analyze the problem.
Figure out an architecture.
Start building it from unit components.
This approach is the best friend of over-engineering. You typically build the system to prevent changes that you think will come in the future, without knowing if they actually will.
Then, when a requirement changes, it typically happens in a way that doesn't fit your structure, no matter how good it is.
For me, the key to drastically reducing the immediate cost of writing with TDD has been to take a top-down approach:
Bring a user story.
Write a very simple test of a use case.
Make it run.
Go back to step 2 until all use cases are complete.
During this process, don't worry too much about architecture, clean code (well, remember at least to use decent variables' names), or any kind of complication that is not currently needed.
Just do what you know you need now, up to the end.
Tests of the story clearly state what the current and known requirements are.
Once you are done, take a look at your big ball of spaghetti mud code, get over the shame, and look deeper at what you have done:
It works! And tests prove it.
All of the system is there and only what is actually needed to get the job done.
Now, you have an overview of all the parts of your system, so you can refactor with the knowledge of the domain that you couldn't have had when you started from scratch. And tests will make sure that nothing will break while refactoring.
Refactoring
The best way for me to start refactoring is to identify areas of responsibility and separate them into private methods. This step helps identify responsibilities and their inputs and outputs.
After that, classes of collaborators are almost there and you just need to move them into different files.
As you proceed, write tests for the classes that pop out from the process and iterate until you are satisfied with the result. And remember, if you get stuck somewhere, just do it!
If you do something bad, once you are done you will have more information on how to get over the mistake next time you face it. Getting the job done is the priority, to the best of your current abilities.
This way, if you analyze your errors to learn from them, you will also refine your abilities.
The Next User Story
Continue developing your product following these steps:
Take a story.
Make it work completely in a "test-code" cycle.
Refactor.
While adding features, you will continue to change your software and maybe even its structure. But, as the system grows, the cost of change will maintain a linear growth due to the two main features of TDD:
Architecture discovery (that helps to control the complexity).
Protection from breaking changes.
The system will not be over-engineered, as architecture is going to emerge as stories are completed.
You don't think about possible future requirements; if you end up needing it, then the cost to implement it will be low.
What Can Make It Go Wrong?
The size of the story. What you build up to the end must be the right size. Not too big (otherwise it will take too much time to get any feedback) or too small (otherwise you won't have the overview).
What if the story is too big?
Split it up in pieces that can be built from the start to the end.
Do you agree with me or do you think that all of this is a bunch of rubbish? Let me know what you think in the comments; it would be great to start a conversation on TDD and share our experiences.
I want to thank Matteo Baglini for helping me to find my way through a practical approach to software development and TDD.
Thank you for reading!