In my last post The Three T’s of Test Engineering I discussed three key challenges to increasing test automation with the overall goal of reducing time to value and maintaining quality. These challenges related directly to the engineering activity itself (i.e. automating tests), but unfortunately mastering these challenges is not the only roadblock for achieving the desired agility to support digital transformation.

According to the World Quality Report 2018 Test Environments and Data are probably the second biggest bottleneck (after automation) toward achieving greater QA and Testing. To illustrate the problems, imagine your software release is like an opening night performance by a Symphony Orchestra, you have a large ensemble of musicians playing instruments from different sections that need to come together in perfect harmony.

Each musician plays a part within a section (e.g. a system) and each section of instruments is dependent on one or more other sections to receive their cue. Now, assume that your strategy to practice for the opening night performance was to assemble the whole orchestra in a concert hall (that was representative of the concert hall to be used for the performance), do you think that you would have the whole performance ready in six months?

A more likely approach would be to have each musician practice their own piece, then meet the rest of their instrumental section and practice together. As each section learns their piece you may start integrating them with other sections to see how they play together and once everyone was familiar with the performance you would arrange for several dress rehearsals.
For the individual musicians’ rehearsals, you would expect them to do this locally (e.g. at home) not pay for a persistent environment (a concert hall) and they would rehearse around their schedule (on demand). You would give them their sheet music and it would probably be a snippet of the real performance, with the conductor being the only one holding the complete symphony.
Practicing alone will only help to a point, e.g. the musician may struggle to gauge the right tempo and will need to be able to recognise their cues but if you recorded the other instrument sections each musician could practice to their own schedule without impacting or waiting on others.

Finally as the rehearsals advance the musicians would want to practice in an environment that closely replicates the opening performance, so you would arrange the seating plan, the acoustics, the conductor, etc… to be the same each time, it would probably not be possible to leave it all set up in situ, so you would define the process: e.g. a seating plan, acoustic configuration, etc so you could consistently setup and clear away (ready for the next time), it may not always be the same people doing the setup and clear down so the instructions would need to provide a step-by-step guide. The setup and clear down would be time consuming and prone to human-error so you may look to use a robot that could be programmed to complete the task the same way each time, on demand.

Testing software is like this, the product is not a monolithic entity it is an ensemble of systems that will be ready at different times, so to reduce the dependencies between the systems and speed up the testing process we need to interact with these other systems even if they are not quite ready: just like the musician playing along with the other instrument sections without them actually being there. In testing terms this would be called Service Virtualisation.

Maintaining one or more persistent environments becomes costly, you wouldn’t do this in the orchestra metaphor, yet according to the World Quality Report 2018 31% of organisations still do this, from my experiences I suspect that this is much higher for so-called End-to-End environments. Relying on a person or team to setup and teardown environments introduces yet another bottleneck (their schedule) and potential inconsistencies (human error), nowadays we see much more infrastructure as code and tools that can automate the build and tear down process – like the robot above. This is a subset of a process called Robotic Process Automation.

Software needs data, the data typically has various states, once you’ve tested a feature the data may change state and no longer be valid for the next test, there are only two main choices to provisioning data:

1. Create it from scratch
2. Use production data

Both have benefits and challenges and require some thought and effort to provision to the tester for when they need it. This is often referred to as Test Data Management
At P2 Consulting we can’t help you test your orchestral rehearsals, but we can help you with Service Virtualisation, Robotic Process Automation and Test Data Management.

To learn more about how we can help your business with our full range of Testing Services, please contact us today.

Luke Barfield, Head of Testing and QA
[email protected]
+44 (0) 7837 259780