TeamCity deployment pipeline (part 2: TeamCity 8, Build once and UI tests)21 Apr 2014
The last post in this series covers a range of techniques for using TeamCity to build a continuous delivery deployment pipeline in a scalable way. This post builds on those techniques by providing some more options to consider given the new features in TeamCity 8 as well as what we've learnt since the original post.
This post is part of a blog series jointly written by myself and Matt Davies called Maintainable, large-scale continuous delivery with TeamCity:
- TeamCity deployment pipeline
- Deploying Web Applications
- MsDeploy (onprem and Azure Web Sites)
- OctopusDeploy (nuget)
- Git push (Windows Azure Web Sites)
- Deploying Windows Services
- Git push (Windows Azure Web Sites Web Jobs)
- Deploying Windows Azure Cloud Services
- How to choose your deployment technology
Following is a bunch of (mostly new) TeamCity features we didn't cover in the last post that we think are useful to know if you are looking after a large-scale TeamCity deployment.
One of the big improvements in Teamcity 8 is the addition of Projects Hierarchy, which allows you to nest projects. In a large-scale TeamCity deployment this is probably very useful:
- You can group related projects e.g. per-client, per-team, per-department etc.
- If you follow our advice in the last post about splitting up production deployments to a separate project (if you need to have different permissions) then you can group the prod and non-prod projects together
As well as a logical grouping, project hierarchy gives you the ability to share the following:
- Build configuration parameters and templates
- Clean-up rules
- VCS Roots
- Users and group roles
- Shared resources
Rob has written before about the importance of using pull requests for commercial software development. One of the really cool things in TeamCity is the ability to automatically build pull requests and then to report back their status via one of a number of plugins (if using GitHub, BitBucket, Stash). You can see more about how to do this in the excellent post by Mehdi Khalili.
Semantic Versioning is really useful for software libraries because it helps communicate the scope of a change to the users of that library. Jake Ginnivan has talked to us before about using semver for commercial projects to communicate the status of a build to the client e.g. major revision can be used for a large change / feature / rewrite, a minor revision can be used for a new feature and a patch can be used for bug fixes or small changes to an existing feature. If you want to use custom versioning with TeamCity then it actually provides you with the ability to dynamically change the build number to help accommodate this.
If you are using Git for your source control, then you are in luck because Jake Ginnivan has created a continuous delivery compatible way of using semver with TeamCity via the GitHubFlowVersion project (note: it's currently in the stages of being merged into the GitVersion project). It's important to note that you can no longer use the Assembly Info Patcher Build Feature when using GitHubFlowVersion, but it has the ability to patch the assemblies for you.
In the last post we discussed the disadvantage of not having the build number propagate to each of the consequential build configurations and provided a number of links to potential solutions. We have since discovered a way of sharing versions between build configurations that share a dependency. You can simply put in
<DEPENDENT_BUILD_CONFIG_ID> is the id of the build configuration you want to use the build number from (note: it must have a snapshot dependency for you to be able to reference it). TeamCity 8 makes this much easier since you can specify the build configuration ID and the auto-completion of dependent variable names works a lot better.
If you find you need multiple build templates and these templates all repeat two or more build configuration steps or a single complex build configuration step then you can extract the common step(s) into a meta-runner. For an example of a Meta Runner check out the TeamCity Meta-runner power pack and the recent post by Rob.
When you are administrating a large scale TeamCity deployment in a limited resource environment you may find you need to quickly monitor where your hard disk space is being allocated. One of the new features in TeamCity 8 that can help in this situation is the disk usage report. You should also keep in mind the clean-up configuration options available to you.
If you have a large TeamCity deployment then it's likely some of your builds may depend on limited external resources such as test databases, metered resources etc. If this applies to you then you may want to look into TeamCity 8's shared resources feature which allows you to limit the number of builds using these shared resources at any point in time.
If you find you have a lot of contention for your build agents you can take advantage of a new feature in TeamCity 8 which allows you to monitor all current builds queued and running on the server.
TeamCity simplifies using custom NuGet packages for your own libraries and products by providing a built-in NuGet server (introduced in TeamCity 7) which integrates with package artifacts generated by a build configuration. This feature is incredibly useful for large scale deployments where you have shared code between projects and you are using NuGet to manage your inter-project dependencies.
An important philosophy in Continuous Delivery is to make sure that your application is compiled once only and those same binaries are reused for every subsequent part of the deployment pipeline. This gives confidence that you are deploying the same code that has been tested and deployed earlier in the pipeline and form a part of the confidence that a proper deployment pipeline gives you.
One of the weaknesses in our last post was that the code could be rebuilt across build configurations if there is more than one agent (or MSBuild decided it wanted to rebuild). This was correctly pointed out in the comments by Marcin.
We have experimented with tweaking the pipeline to get build once and we have three main options we have explored:
- Ensure you have one agent only and trust MSBuild to not rebuild if the files haven't changed (aka the last post)
- Rather than using Web Deploy to perform the XDT transformation for each environment, create the MSDeploy package while building (using
/p:DeployOnBuild=true), put that package as an artifact, populate the package to deployment steps as an artifact dependency and then use the parameters functionality in MSDeploy to perform environment transformations at deploy time:
- Use OctopusDeploy, which supports deploy-time configuration changes (see the next post in the series)
One thing that wasn't mentioned in the previous post about TeamCity pipelines was running UI Tests. Since that post we have done a lot of work with integrating automated UI tests as part of our TeamCity pipelines. Providing guidance about running automated UI testing in TeamCity is a post in itself, but here are a few pointers to get you started:
- Consider what point in the pipeline you want to run your UI tests e.g. alongside your unit tests, after automatically deploying to your first environment after unit tests, after deploying to a test environment to provide more confidence before hitting prod, after deploying to a dedicated (static or on the fly) UI testing environment
- Remember that UI tests are generally slow so preferably you want to run it out of band of your first build configuration with unit tests
- If possible include a separate TeamCity agent that is dedicated for running UI tests so your continuous integration builds don't get delayed and you increase your immediate feedback loop
- Consider whether you want to run UI tests against all pull requests or not (on one hand you get a lot of confidence, on the other it takes a lot more time and requires isolated environments, which may be more complex
- Configure your agent machines so they are configured to be able to run the automated UI tests if you need interactive mode; note: if you are using PhantomJS then you might not need the agent to have interactive mode
- Make sure that you take screenshots whenever your UI tests fail and that the artifacts are dropped in a directory that is configured in your build configuration to pick up all files as artifacts
- Make sure you clear that directory before every build otherwise the screenshots will start stacking up from previous builds - TeamCity has an option for this