Quality is paramount – Quality assurance can only be delivered by thorough process and product testing.  Testing takes time and effort and adds costs to projects, but is something we absolutely recommend to clients and something we at Lightflows build into our product and software development cycle.

For a complex or business-critical web application the benefits of building an automated acceptance test suite are considerable.

Running a comprehensive set of tests against a proposed release build before deployment to production gives everyone confidence that the release will go smoothly, new features will work as expected, old features will continue to work, and customers and users will not be disrupted.

What does automated acceptance testing look like?

When the tests are running, it can be a little like watching a pianola play ‘The Entertainer’ – the keys are being pressed fast and furiously and exactly the same way every time but there is no one sitting at the keyboard.

Imagine having to sit down and run through using your own web application to test that everything works as expected in every scenario you can think of. Would you click every button, check every input on a form, check every error message? Would you test that everything works every time a change is made to the application? Even if you intended to check everything, would you remember?

Having this kind of laborious testing automated can save person-hours which might be spent on more interesting or productive work, but above all, if it is built and maintained faithfully, it catches mistakes that humans, being human, would not catch.

Instead of having a person go through every user story or follow every route through a flow diagram, we write some code to test the application and run that code instead. The code opens a browser, navigates to pages, enters text into forms, logs into applications, clicks on buttons, select from dropdowns, everything, in fact, that a human would, but faster of course, and the same way every time.

What kind of errors are prevented?

Errors prevented by this kind of testing are the most important ones: those that would affect the users, customers, the business itself.

Acceptance tests are written from the point of view of the user of the application. They don’t care about the technical details behind the scenes, whether the code is running on the front end or the back end, or is written in this language or that. They should not care about the implementation details at all.

Acceptance tests simply measure whether the application meets business requirements. Can you log in and place an order? Can you look up a customer by name, retrieve their orders and sort them by order value?

This kind of testing is not the be-all and end-all of testing, nor should it be the only testing carried out during the development of a new feature or a new application, but if you have it, it is guaranteed to ‘save your bacon’ sooner or later.

Because the acceptance test comes from the business or end-user perspective it will catch missing functionality that no amount of any other type of testing could catch. Imagine that the specification and design includes a requirement for a confirmation dialogue before records are deleted, but for some reason, it failed to get into the development pipeline. Unit and functional tests all pass, you can, of course, delete records as expected, but only the acceptance tests catch that the confirmation dialogue is missing.

The law of unintended consequences will at some point mean that in a large codebase, in the less than the perfect world we work in, changes made to one part of the code will break something somewhere else. Even the most experienced and knowledgeable developers will eventually fall into this trap. Testing new code and new functionality is necessary and valuable, but on the occasion where something else, that this developer is unfamiliar with, is badly impacted by the change and ceases to work, it is only the automated acceptance test that will catch the mistake before any damage is done.

Acceptance testing is too much hassle?

It is certainly true that building and maintaining a suite of automated acceptance tests is time-consuming and requires an attention to detail so that coverage is meaningful and no pathways or elements important to the business are missed.

Is this cost worth the benefit?  Certainly, automated acceptance tests are not universally viewed as a net benefit, but how important is it to you that your application functions as expected without disruption to its users? Is it worth taking the chance that nothing is inadvertently missed, that the law of unintended consequences won’t apply this time or the next?

The discipline of creating and maintaining the test suite will itself ensure that nothing critical is missed from the specification, that the focus of development is on the end result from the user perspective.

Building automated testing into the build and deploy pipeline ensures that software is not released that has easily avoidable business-critical bugs in it.

Dos and some more Dos of acceptance testing

  • Build the tests from the specification, you are testing the fit of the application to the business need.
  • Build the tests in a modular way so that commonly repeated tasks are extracted into functions or classes which can be used over and again.
  • Make the tests subject to version control and whatever release or tag system you have. Obviously you should run version 3.0 of the tests against release 3.0 of the software!
  • Build a successful pass of the acceptance tests into your build pipeline. Run them against the feature or development branch by all means, but you must run them on the merged release candidate.

Run the tests against a known set of data, a separately maintained test database, or a set of data constructed by the test suite itself. The data structure must match the production database but the content cannot be mutable in the way that lives data is. When the tests call for a user to login to the application, you do not want to use real users or real passwords, or to have dummy test users lying around in the live database, even a copy of the production database, ready to be removed by an admin doing a clear out. The test data must be a faithful representation of real data but should have no real or sensitive information in it and should not be subject to uncontrolled change.

If you are running the tests so that they can be observed in real-time, record them as part of the sign-off, although don’t expect any entertainment value from the resulting video.

In Conclusion

Automated acceptance testing is not for the faint-hearted or the lackadaisical, but it will undoubtedly improve the quality of the delivered product of almost any software development team. And there is always the satisfaction of seeing all those green ticks when your latest pull request is accepted.