Some time ago I was discussing with my colleagues about whether a set of end-to-end suite tests were meant to test the same thing than an integration suite test, at that moment the discussion ended in a agree to disagree terms, I remember to mention that I would investigate about the topic later and write an article somewhere.
After reading for quite long time about both terms I think that in that moment we were discussing with an erroneous base idea (maybe that’s why we couldn't get into a clear conclusion), basically end-to-end tests can’t be compared to integration tests because in reality the first is a type of the second one.
Googling a bit about the topic I found with the exact same mistake, a lot of people were writing about end-to-end tests and they were interchanging these with integration tests implying that they were synonyms.
Now we know that end-to-end tests are a kind of integration tests, we have a clear doubt arising, When we said end-to-end tests were doing the same thing as integration tests, with what type of testing were we comparing them? The answer is simple, System testing.
Perfect, now we have two things to compare end-to-end versus system tests. both of them are integration tests, what differentiate them is the context, they test the same thing (with some variants) from different places and different ways. End-to-end test the general functionality of the system and also the flow of the screens, for this article we are going to focus in end-to-end test in the web (the main reason why I made emphasis in screens flow). However, the term can be used for mobile, desktop or anything that works as a communication interface for a system. System testing is done in the server context (backend), they also test the functionality of the system, the interoperability between components and their dependencies (they are done directly from the code and not through a communication interface).
Which of the two should we use? (in case we have the opportunity to use only one of them), this depends mainly in the nature of the project, I’ll list some of the benefits and disadvantages of each one and I’ll give my opinion about the question above. However,
being the nature of the project to be the reason with more weight to take the decision.
Let’s start with end-to-end:
Aside from testing the whole system, you also test the screens flow.
Implicitly you tests a lot of internal things about the system with the results, for example, check that the push notification system is working correctly or that the login/logout process do the expected behavior.
It tests directly what the final user (or consumer) will interact with.
You can run these tests in multiple browsers automatically. With this you make sure that everything is working not caring about what browser we are using (this is something that has improved with the time, before the use of hacks to make something work in a specific browser was the day to day in web development).
The more tests you have the easier it will be to create new tests cases (this of course if you make correct usage of the PageObject pattern and maintain your test code in good shape with good practices).
They take too much time to run, normally we will have to run them isolated to save some time when developing, for example, if you are working in something related to the login process you should only run the login test suite.
They aren’t 100% consistent, even though the browsers issue has improved a lot, sometimes you could have weird behaviors that make the tests fail. The problem is that you might re-run them and the problem will not appear again.
If the design of the views change is highly probable that you will have to discard your tests because they are highly coupled with the HTML & CSS, it is possible to achieve maintain them but this depends of a great organization not only in your HTML/CSS but also in the correct abstraction of their use in the tests.
If you have a Continuous Integration / Continuous Deployment flow (which you should), the inconsistency of tests will become a real problem because they can sometimes fail without any apparent reason delaying your deployment process.
Now let’s list the things for System testing:
You can test the business logic without intermediates or interferences.
They are really fast so they can be run multiple times without problems (in complex systems they could take a bit of time and it highly depends on the nature of the system and what we are testing, however they will always be faster than end-to-end tests without doubts.
Deleting the data created by the tests is pretty easy, you can even create a new database for the tests (as part of the process of the tests themselves) and when a test fail or all of them pass successfully you just drop the database.
Normally they are faster to implement than end-to-end tests (however we have to take into account that the velocity of implementation of end-to-end increases the more tests we have) this is highly dependant of what is exactly being tested though.
The only way that these tests become obsolete is if the business logic changes completely, in this case you are better off starting a whole new project than trying to change this one with the test suite (Did someone experience this in real life? I doubt it, not me).
If you have unit tests is hard to decide to start using integration tests, because you can “feel” that is just too much (personally, it's never too much) however this is something that happens a lot.
You have to be careful when creating your tests trying to do your best not to create direct dependencies with frameworks or things like that (this is specifically in the tests, of course your tests will probably be using code that has dependencies to a framework but in your test code you should only be using your domain language and maybe some mocks also, of course).
Sometimes they are hard to create when the project is already created and in an advanced phase because if a project is in this state and it doesn’t have test it’s highly probable that testing was not in mind when designing the solution for it (this is really a problem of architecture of the project but still related to this kind of tests).
In the server context it’s common to prefer unit tests over system testing, supposing that you only have budget to use one of them.
My opinion about whether to use end-to-end or system testing would be to use system testing (mostly), the reason for this is simple, most of the time the critical part of your application would be in the server (the business logic). However I repeat for third time that one of the most important things when choosing is the nature of the project, an example in where I would choose end-to-end over system testing is a mobile game, this game will only save some basic information in the server like the score and an username, in this case is way more important the user experience with the game itself than the primitive server logic, exceptionally I would prefer to find ways to test the game interface of course.
If I could go back in time and have the discussion again with my colleagues what I would do first is change our erroneous base idea that we took about the comparison and I’m 100% sure that we would have ended in an immediate shared conclusion about the issue.