Tuesday, 28 July 2015

You Still Need Manual Tests

This blog has numerous examples of why unit, integration and contract testing is essential. However you still need manual tests. It is foolish to believe that all testing can be covered by automated tests despite the bias in this area.

Why?

  • Manual tests can catch anything you may have missed at an automation level.
  • Manual tests can be unique. Use exploratory testing to try different scenarios.
  • Manual tests that fail become automated tests, so they can never regress.
  • Manual tests exercise the full stack. Many areas such as DI conventions, logging, and other framework related configuration are better suited to manual verification.
  • UI changes require visual verification - automation is near impossible here.

More Than Just Functionality

Over the years, manual testing has caught numerous bugs, issues or things I've just plain missed. When you are deep in a problem it can be hard to see the wood for the tress. A second party manually testing provides an unbiased check of your code for a second time.

The key with manual tests is to ensure any issue is converted into an automated test. This offsets the fact that manual testing is expensive both in terms of time and cost. By doing so any regressions will be prevented.

QA includes more than functional testing. Security, performance and usability to name a few are equally important. Do not avoid the manual test step. Automated tests are only as good as the tests themselves. Embrace manual and automated testing for the best of both worlds.

Tuesday, 21 July 2015

The Benefits of Contract Testing

I previously claimed that you need some integrated tests but as few as possible. There are huge benefits to this approach, but there is a problem. How do you stop your test doubles falling out of line with the real implementations? The answer is to use Contract Tests.

Steps

  1. Create a base test fixture, this is where your tests live. All assertions belong here.
  2. Subclass this base class with each implementation.
  3. Override each setup step to provide the implementation that is to be tested.
  4. All tests should pass for each instance.

Example

In this example there is a SQL repository and an in memory repository. It is not possible to change either in any manner that causes them to behave differently. We can safely use the in memory repository for tests, with confidence that the test double matches the contract of the real implementation.

The test double implementations can be executed on every test run. While real implementations can be relegated to execution prior to commit or during continuous integration. This trade off allows for fast feedback cycles while ensuring all tests are run against production like implementations.

References

Tuesday, 14 July 2015

Integration Tests

It is well documented you need a balance between different categories of automated tests. The split is usually in the form.

  • 70% unit
  • 20% integration
  • 10% acceptance

While unit tests make up the majority of tests, there is a limit to their effectiveness. As soon as you leave the system boundary you need integration tests. Examples of when integration tests are required is code that interacts with databases, web services or the file system.

These integration tests should not test logic, this is a mistake. They will become brittle and slow to execute otherwise. Instead of checking domain logic, test at a lower level. Go as low as you can without leaking implementation details of the subject under test. By going as low as possible you will radically reduce the number of integration tests required. Less tests means easier maintenance. Less tests also means faster tests.

Example

Assuming a SQL database, invoke the repository and test as lightly as possible. Do not indirectly test this repository by invoking the code higher levels in the stack. Avoid concerning yourself with what is happening behind the scenes. Simply test that you can insert a record, and retrieve the newly inserted record. Any other code that is involved at higher levels can suffice at a unit level.

Assertions should be loose enough to verify that the code is working, but not asserting basic correctness. In other words prefer assertions that check for the presence of results, rather than what those results look like. If the value is of concern, convert into a fast, isolated unit test.

Integration Tests are a Scam

The term Integrated Tests is my preference given that integration tests are a scam. This slight change in terminology helps keep these tests focused. Rather than spiraling out of control, they are small in number and simply verify that "something is working". This is done by pushing all tests of logic to the unit level.

The key point here is that integration tests are required. Strongly resist the urge to write all tests at the integrated level. Likewise do not fall into the trap that thinking all tests must be done at the unit level. The key here is balance.

There is a fatal flaw with integration tests however. They can be wrong. Given tests at a unit level will stub out anything that is out of process, how do you stop such tests falling out of sync with the real implementation? This is where Contract Tests come into play.

Tuesday, 7 July 2015

Static Code

Static code is considered a bad thing by developers. This is especially true when working with legacy code. The use of static code is often seen as a smell and should not be used.

This is not as black and white as it first seems. Static code can be problematic when global state is involved. Not only is it hard to change, static code is very hard to test in an automated fashion. Bad examples of static code include persistence, third party services, and out of process calls. These examples should avoid static code where possible.

One guideline that served me very well in my early days of TDD was treating static code as a death to testability. Unfortunately some developers don't move on from this guideline and treat any use of static code as bad.

In fact static code can have a benefit. If a method within a class can be promoted to a public static method (PSM) it shows that the code is stateless. This allows the "extract class" refactoring to be performed. Without a PSM such refactoring is much more difficult. IDEs can automate this step and if in a dynamic language you can simply lean on the runtime to catch issues.

The steps to perform this refactor are easy. If at any stage this is not possible the method contains state.

  1. Make the method public.
  2. Make the method static.
  3. Move the public static method to the new class.
  4. Update usage of the previous calls.
  5. Optionally remove the static modifier and update previous call sites.

If the code cannot be promoted to a PSM then state exists. Increasingly the code I write leads itself to a functional paradigm despite not be written in a strictly functional language. Small, focused classes that tend to be immutable. The use of PSM makes transition to this style of code easy. There is no reason to avoid the use of static code as an intermediate step to get to this position.