Refactoring CI Build Pipelines

Problem with CI Tooling

CI tools such as Team City are great. They provide a nice front end that can be used as control for all your builds. The likes of triggers, test results and build history are some of the stand out features.

In addition to these features there is a whole host of other functionality that is built in. Just try and add a new step, the dropdown will contain many different tools that can easily be configured.

My problem here is that configuring your build pipeline via these tools is asking for trouble. In fact I would go as far as to say that the use of these tools to control your build is an extremely poor practice that should be avoided.

  1. It ties you to that particular tool. While it is not hugely common to switch CI tools I have repeated this task several times over several companies. It does happen.

  2. Feedback is slowed down. If you wish to make a change you either need to commit or modify the pipeline. This either works or if it doesn't breaks the build for everyone else. If the key to success in DevOps is fast feedback, this style of work is not compatible. The use of a test CI pipeline is an overhead.

  3. Depending on the type of company you work for, you may or may not have full control over the CI environment. This further limits speed of feedback and experimentation.

  4. Source control is an issue with such tools. While it is possible to see and audit changes this is separate to your standard source control. If build 1.2.3 is produced it should contain all changes for that version, including build steps or deployment changes. I don't want these stored in an external tool or proprietary format.

Solution

To counter the issues above I have found a single script that can be run locally that your chosen CI tool runs is the best solution.

A dedicated build script has a number of benefits when compared to hosting this logic within your CI tool. Your chosen CI tool can then simply run this build script.

The acid test for this script is that it should be possible to run this anywhere. For example, running this against your personal Azure subscription should setup and run your application as if it was run against the real subscription. Your chosen CI tool can still provide benefits such as a nice UI, test formatting and triggers. It simply executes the script on your behalf.

If additional features of your CI tool are to be used, these should be complimentary rather than required. Ideally these are setup once in a base template that can be shared between projects of a similar type. If these additional steps do not execute it should not effect the output of the build.

Refactoring

A previous pipeline had a setup similar to the following, all configured in Team City. In other words, other than building and running tests locally within Visual Studio all the remaining build steps were only run on commit. The steps I took to refactor this are detailed.

  1. Build
  2. Run Tests
  3. Package Integration Tests
  4. Package Acceptance Tests
  5. Package Application
  6. Publish Application
  7. Publish Test Packages

The first step in this refactor is to add step zero that will run the build script before anything else.

  1. Run Build
  2. Build
  3. Run Tests
  4. Package Integration Tests
  5. Package Acceptance Tests
  6. Package Application
  7. Publish Application
  8. Publish Test Packages

Step zero initially will do nothing. Just execute the build script and return. If you can complete this step you are in a good place to start.

The next step will be to move step 1 into the build script. There are a number of ways to do this and validate the porting. You can check the Team City steps and use the same underlying command and parameters. To validate you can compare the output from the build script against the build output from Team City. In verbose mode the server will log the commands and parameters to stdout.

After successfully porting step 1 disable this step in Team City. This acts as your backup, if you wish to revert you can do so easily by just enabling the step.

The next part of this refactor is to actually test your builds. I suggest at least one deploy goes through and is deployed successfully for each step that is ported. While this is slower, it proves the porting is successful and will highlight any issues within a small scope. Slow and steady wins here.

  1. Run Build
  2. Build (Disabled)
  3. Run Tests
  4. Package Integration Tests
  5. Package Acceptance Tests
  6. Package Application
  7. Publish Application
  8. Publish Test Packages

Rinse repeat this process for each of the other steps.

Once all steps that you wish to port are disabled I suggest leaving a small period for the script to bed in. Once you're happy then the disabled steps can be deleted.

The ideal scenario here is that as much as the build is ported into the dedicated build script, with the only remaining steps either specific to your CI tool or base template specific. The true acid test here is that the script can be run anywhere as detailed previously.