Behavior-driven development in dependent pipelines.

Status Quo

It is very common for automated tests to be executed from a test developers desktop, namely because it has a browser and can render pages locally. To allow testing on a variety of platforms and browsers, BrowserStack is commonly used, which is great! However, this implies that the tests are being run from source, and are therefore volatile.

Execution from a Pipeline

With test execution being performed from source, it is a natural progression to use this same method in a pipeline. While this provides a level of automation, it continues to be volatile based on what is happening in source control.

Is there a better way?

Software development has carved this path in the past. If we think of test automation as being a dependency of our deployment, it makes sense that the process software development uses to address the same challenge can be used in test automation. The key is to build a runnable artefact from your test automation source and store that in a a registry, to be consumed by the software development pipeline that needs to be tested.

So how do they do it?

Software has dependencies on 3rd party components. The source of these components is not brought into the development process, instead, versioned artifacts are included using dependency management tools, such as NPM, PiP, Maven, NuGet, etc.

These tools can also be used to bring a versioned test artifact into your dependent pipelines to test your changes, however, many test automation tools, i.e. Selenium, have runtime dependencies which can be difficult to fulfill in you pipeline agent, especially one that you do not host yourself.

Containers to the Rescue (again)

By packing the runnable artefact, along with runtime dependencies into a container image, the tests can be executed within the pipeline. Combined with docker-compose, both the application and the tests can be executed using multiple source IP addresses.

Like all things in 12-Factor principles, deployments have no relationship to source control, only artifacts.