Automate Integration Tests in CICD Pipeline

5 min readApr 4, 2019

Recently I helped our team to automate integration test in our CICD using TeamCity and Octopus, I’m sure it’s not new but it was for me, I thought it’d be a good story to share how I did it.

What we try to achieve

The goal we try to achieve is to have a repeatable environment independent continuous delivery, with this in place the team should be in a better position to deliver their features or changes from local to production fully automated. Therefore it is key to have automated testing that is beyond just giving functional assurance but also can assure correctness of application configuration after deployment, as well as the infrastructure the application is running within, such as network connectivity between its dependencies.

The test itself needs to be environment independent, meaning, it can be run against any environment DEV, UAT, or even production, and it should be automatically kicked off after deployment.

Solution

Our integration test project is written in C# using XUnit, and our CICD pipeline uses TeamCity and Octopus. Initially, I planned to setup TeamCity to have the Run Integration Tests after the Deploy Dev step (like below) as it looks naturally fit until I realized the complexity and difficulty of getting Octopus variables especially Octopus output variables in TeamCity is overwhelming. It can be done but it requires more effort than having tests run directly in Octopus, so I chose the latter.

Following is a diagram showing the solution at high-level, it doesn’t include details like how a service itself is deployed or resources it required are provisioned, as they are not the focus of this story.

Prepare test project

There isn’t any restrictions about what can be used to write a test project it can be any one of the common test frameworks, such as NUnit, XUnit, MSTest or even SpecFlow, etc because the principle is the same, which is to find a way upload all compiled test assemblies to Octopus server and run with the test runner that it is compatible with.

Different test framework requires a different test runner to execute and often different test runner requires different .Net runtime. For instance, a .Net Core test project using XUnit can be executed by running dotnet test which internally invokes relevant test runner, but if it is a .Net Framework project then it requires to be run with a standalone xunit.console.exe of a specific version corresponding to which version of .Net Framework is targeted.

So it would be good to know which one is available on the Octopus server first, to do so, I created a temporary Step and ran it with an inline PowerShell script like following.

#This should cause error, if dotnet runtime is not installed, otherwise it should print out the version of .Net Core.dotnet --version#This script print out .Net Framework version (4.5 and later), the 394802 in the script below is the release DWORD, go here to find the full table of .Net framework version: https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/how-to-determine-which-versions-are-installed#version_tableGet-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full\' |  Get-ItemPropertyValue -Name Release | Foreach-Object { $_ -ge 394802 }

And it turned out .Net Core isn’t available but .Net Framework 4.6.1 is, so I need the xunit.console.exe of 4.6.1 to be available in the bin folder together with my test assemblies, followings shows how I did it.

Add xunit.console.runner package to the test project, after installed, I can then find the exe file in the global package store folder under $(USERPROFILE), (if you are using the old packages.config, you can find it inside the packages folder under the solution root folder.)
Then add the following to csproj file in order to output it to bin folder.

<None Include="path\to\xunit-console.exe"><CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory></None>

Apart from test runner, I also need a Powershell script that contains a script to execute the test runner once the package loaded in Octopus server. Following is how it looks like in Visual Studio.

.\xunit.console.exe .\sample.dll

The content of the script is literally that, or you can add more parameters to enrich your test, for example, adding Trait to enable filtering. As you can see the script above expects the exe is located at the same folder as the Powershell script, as well as the test assembly, which means it’s also required to be output to bin folder, to do that, I set the Properties of the file to Copy always.

Pack and publish test project in TeamCity

Once I have everything output to bin folder, then I created a nuspec file that instructs the pack command to pack the entire bin folder for publishing. You could exclude files that are not needed from packaging, for instance. all pdf files, to make it cleaner, I was a bit lazy just includes everything for simplicity.

To actually setup pack and publish step in TeamCity is pretty straightforward, both are well supported by TeamCity. I will not repeat here.

Download Package and run tests in Octopus

This isn’t a step right after publishing NuGet package, I’ve already had my Octopus project created as well as steps in Process for provisioning resources and deploying actual service/application I want to test against.

Then I added a new step by clicking Add Step and chose a built-in Octopus template to Run a Script and clicked Add

In the Step Details, selected Script file inside a package as script source, then chose the package feed to where my package was uploaded, the package ID, and the relative path to the script file that contains the script for executing test runner.

In my case, I also enabled the Octopus feature to transform the configuration file used by test so that integration test itself can be environment independent.