The Problem with End-to-End Tests
Salt is great. According to Wikipedia, salt is essential for life in general, and saltiness is one of the basic human tastes.
Most people cook their food with table salt. However, if you add too much or too little salt to your meal, then you’ll likely have a bad time eating such meal.
This doesn’t make salt bad. Does it? No. It just needs to be added in the right quantity.
Other than getting the quantity right, there are medical conditions for which you’re advised to either reduce your salt intake, or do away with it completely.
So, is salt bad just because someone was asked to do away with it?
I’m no medical practitioner, but maybe not.
The same may be said for end-to-end tests. They are great for testing software, but they have their own share of problems. Too much or too little isn’t good for you either.
Before going on, do you know what end-to-end tests are?
What are End-to-end Tests?
Of all tests, I think end-to-end tests are pretty easy to understand quite intuitively.
If you had a banking application for which a user could withdraw money from their account such as shown below:
You may write an end-to-end test that tests that a user can actually withdraw money i.e without bugs.
In a frontend Application, an end to end test automates a browser and checks that certain functionalities work, while pretending to be a user.
In pseudo-code, the test file may look like this:
const testWithdrawal = (browser) => {
browser
.goToURL("http://localhost:3000")
.click("button-1")
.expect("#amount")
.toContainText(2490701)
}The test acts like a real user. It fires up a browser, and visits the url where the app is loaded. It also simulates a click on the withdrawal button, then makes an assertion to confirm that the final amount is equal to the previous amount minus 10000.
Pretty easy to understand.
So, what’s the actual problem with end-to-end tests?
What Problems?
End-to-end tests are powerful kind of tests. They save you a lot of time while testing functionalities as a whole.
In fact, if you ever looked up different types of tests for a frontend application, you’re likely to come across a “testing pyramid” that looks like this:
Essentially, end-to-end tests are paraded as the most useful kind of tests to write.
This is true, but if you’re new to testing in general you may interpret the previous statement wrongly.
You may not see the need for other kinds of tests. Or even worse, you may be tempted to wipe out unit and (or) integration tests in favour of just having the “ultimately useful kind of tests” — end-to-end tests.
Just before you go ahead with the wrong interpretation about end-to-end tests, please consider the following:
1. End-to-end Tests are Slow
Think about it. Firing up the browser, loading up the application, setting some state, simulating user clicks etc.. That can take some time!
As a matter of fact, it isn’t uncommon to have a suite of end-to-end tests take about 30 minutes to run.
While you may think that’s not a lot of time, remember that while developing applications, most times you want immediate feedback.
Does this new feature break existing functionality? Does it work as expected?
You want an answer quickly, and if it takes you 30 minutes every time, you’ll have to develop some patience.
If you make the mistake of thinking end-to-end tests are the only kind of tests you need, then you may have written too much of them which may take hours to completely run your test suites.
You’ll agree that’s not the best way to get iterative feedback very quickly.
2. End-to-end tests fail more frequently.
Assuming you know what a unit test is, a well written unit test run a hundred times will likely succeed as long as the tests are deterministic and no change has been made to the codebase.
The above statement is true for unit tests but the same can’t be said for end-to-end tests.
So, what do I mean by they fail more frequently?
You may have an end-to-end test fail whereas the code being tested is working as expected. No bugs. Nothing wrong. Then why did the end-to-end test fail?
Well, the code may have taken too long to be executed, an API may have taken longer time to respond or be temporarily down, on and on it goes.
This isn’t a problem solely with end-to-end tests. With many external dependencies comes these problems.
End-to-end tests are great, but be aware they may fail even when your code works just as expected.
You know, it’s like having that weird boss who’s happy with you today, and tomorrow frowns at you like they’d seen a ghost. You did nothing wrong, you’re the same person, they just aren’t very reliable with their mood.
You get the point now.
3. Debugging End-to-end Tests can be Hard
Debugging. We all love debugging, right? Hell, no. If I don’t have to spend so much time debugging code, I’d be shipping new features in minutes. Uh, that’s a stretch.
Debugging end-to-end tests can sometimes be a real pain. Apart from having to open the browser and trying really hard to reproduce the bug, you could experience more frustration if your test failed on another machine, e.g a CI server.
To curb such frustrations, it is very common to have end-to-end tests run on a reproducible environment like a docker container.
You see how much your knowledge base grows to accommodate end-to-end tests? Well, that can be difficult for junior developers too. Anyway, that’s for their good, I guess. They’d be forced learn on the job but be aware of the extra learning curve.
4. End-to-end tests are more difficult to write
I wouldn’t stress this point as you likely already know this. It is just worth re-iterating that a suite of end-to-end tests will generally take more time to set up and write as opposed to other kinds of tests, namely, unit tests, integration tests, and snapshot tests.
Amidst the Problems…
Don’t get the essence of this article wrong. I’m not against end-to-end tests. They are fabulous. However, if you’re new to testing it’s easy to think they are the ultimate saviours that do not have any downsides.
Well, they aren’t, and this is exactly why a good test suite should have the right blend of tests.
It’s like making a nice bowl of soup. Water’s great, but if pour too much water while making the soup, you end up with painful dinner. Every ingredient must be added in the right proportion. Even though “right proportion” is mostly subjective, there’s usually an agreeable portion everyone can come to terms with.
The same may be said of writing tests in general.
Write more unit tests, fewer snapshot tests / integration tests, and even fewer end-to-end tests.
Conclusion
With your understanding of why the “great” end-to-end tests shouldn’t make up for your entire test suite, go write applications with a balanced test suite, and hopefully with fewer bugs.
Selah!
