Skip to main content

How to Achieve More Stable End to End Tests

Recently myself and another colleague wrote an acceptance test for a feature that had yet to be implemented. For this end to end test we used Selenium, after all we wanted to test the whole feature so this made sense. Our test performed some simple user input, performed a calculation and checked the response. The problem with the test was it was very brittle. If the application had not recently been used, the massive data set the application relied on would not be cached.

To get around this we added a few Thread.Sleep() statements into the test. This worked rather well for the majority of test runs, but sometimes these pauses were not long enough. On the other hand sometimes the data was cached, meaning these sleeps would be unnecessary. One resource which has recently done the rounds was regarding useful advice about using WaitForPageLoad() and WaitForCondition(). WaitForCondition will only execute once a condition has been met, such as a element becoming visible. This meant that for the times when the dataset was in memory the test would be executed immediately, while the times when the data was being loaded, the test would simply wait until the test was ready to move on. This was a very simple, yet highly effective tweak to our tests. The execution time went from roughly thirty seconds, to just less than ten seconds in one case.

This was not the end of the battle to achieve more stable Selenium tests. Some of our tests were still rather flaky. Some mornings we would enter work, notice the red build and discover that the several failed tests were down to Selenium timeouts. During the daytime however, we rarely had these issues. In order to fix these problems I increased the frequency of builds. The idea being the more we run our builds the more chance we have of spotting the errors. After all, if something was to fail at 2am, I am unlikely to care. 2pm however, and the team will be all over it. By making the problem more visible, we would be forced to fix the outstanding issues.

The aim was to make the tests as fast as possible, while maintaining stability. One thing the excellent Growing Object-Oriented Software (Goos) touches on is the aspect of not needing to perform end to end testing at the GUI all the time. The benefit of not touching the UI is huge. Your tests are faster, they're more stable and a heck of lot easier to write. The other nice benefit of testing from an API point of view, rather than the browser is it forces you to decouple your app from the views. If you're not writing fat models and skinny controllers, you'll have adapt in order to test as much of your application as possible without hitting the UI.

What about the remaining part of your feature that is not covered by the application? I like to imagine this part as the tip of an iceberg. As this area is small enough the actual UI testing you need should be minimal. So here we can let Selenium do what it is good at. Click things. Selenium is great at this. All you need to do at this level is check for 404s, incorrect page titles and a few other mundane aspects of the UI. There should be no need to check if your actual application is correct at this level. For correctness, you should have a large suite of fast, isolated, unit tests.

Another point to consider is how often your view actually changes, in comparison to your actual underlying API. A designer should be free to move things, rename content, add images and so forth without breaking tests. As long as there is a calculate button somewhere on the page, and said button takes you to a result page, who cares about everything else? Likewise the code underneath can be consistently changing behind the scenes, as long as the API remains constant, our tests should always be valid.

For the technical low down on some of the ways we are achieving more stable end to end tests, check out six tips to speed up Selenium tests.

Comments

Popular posts from this blog

Constant Object Anti Pattern

Most constants are used to remove magic numbers or variables that lack context. A classic example would be code littered with the number 7. What does this refer to exactly? If this was replaced with DaysInWeek or similar, much clarity is provided. You can determine that code performing offsets would be adding days, rather than a mysterious number seven.Sadly a common pattern which uses constants is the use of a single constant file or object. The beauty of constants is clarity, and the obvious fact such variables are fixed. When a constant container is used, constants are simply lumped together. These can grow in size and often become a dumping ground for all values within the application.A disadvantage of this pattern is the actual value is hidden. While a friendly variable name is great, there will come a time where you will want to know the actual value. This forces you to navigate, if only to peek at the value within the constant object. A solution is to simple perform a refactor …

Three Steps to Code Quality via TDD

Common complaints and problems that I've both encountered and hear other developers raise when it comes to the practice of Test Driven Development are: Impossible to refactor without all the tests breakingMinor changes require hours of changes to test codeTest setup is huge, slow to write and difficult to understandThe use of test doubles (mocks, stubs and fakes is confusing)Over the next three posts I will demonstrate three easy steps that can resolve the problems above. In turn this will allow developers to gain one of the benefits that TDD promises - the ability to refactor your code mercifully in order to improve code quality.StepsStop Making Everything PublicLimit the Amount of Dependencies you Use A Unit is Not Always a Method or ClassCode quality is a tricky subject and highly subjective, however if you follow the three guidelines above you should have the ability to radically change implementation details and therefore improve code quality when needed.

DRY vs DAMP in Tests

In the previous post I mentioned that duplication in tests is not always bad. Sometimes duplication becomes a problem. Tests can become large or virtually identically excluding a few lines. Changes to these tests can take a while and increase the maintenance overhead. At this point, DRY violations need to be resolved.SolutionsTest HelpersA common solution is to extract common functionality into setup methods or other helper utilities. While this will remove and reduce duplication this can make tests a bit harder to read as the test is now split amongst unrelated components. There is a limit to how useful such extractions can help as each test may need to do something slightly differently.DAMP - Descriptive and Meaningful PhrasesDescriptive and Meaningful Phrases is the alter ego of DRY. DAMP tests often use the builder pattern to construct the System Under Test. This allows calls to be chained in a fluent API style, similar to the Page Object Pattern. Internally the implementation wil…