This site uses cookies. Continue to use the site as normal if you are happy with this, or read more about cookies and how to manage them.

×

This site uses cookies. Continue to use the site as normal if you are happy with this, or read more about cookies and how to manage them.

×

Generating new Jenkins jobs from templates and Parameterised Builds

Build automation is a really important aspect of our work here at Black Pepper. If you can't automate it, you're going to end up spending more time than you'd like doing things by hand and likely also do do them wrong from time to time because we are only human.

We use Jenkins to run our builds in a continuously. This is great, but you still have to do a fair bit of configuration each time you set up a new job. If you have a fairly static set of builds this isn't a problem. We found ourselves in a situation where we had to create a lot of very similar builds quite regularly. Creating a new job by hand and manually changing the 10 or so tiny little things in each build is a pain and error prone. So...automate that too!

The steps

This is not really Jenkins for beginners, so I'll give an overview of the steps and then go through them in more detail.

  • Configure a single jenkins job that does what you want to duplicate
  • Refactor that job and replace all the bits that you want to change with placeholders
  • Use the Jenkins CLI to copy that job and replace your placeholders with new values
  • Create a job that runs your Jenkins CLI script to create new jobs for you
  • Run your job creating job with parameters to create as many new jobs like your old one as you like

Your template job

It will help if we have a concrete example of a job that we want to work with. In our case we are generating OpenStreetMap tiles for particular areas. We have a single GIT repo that contains folders with configuration of each set of map tiles we want to generate. The old sets rarely change and we don't want to go to the effort of regenerating all map tiles every time we commit to that repo. So we have a job to generate tiles for each single config folder.

config/
       tileset1/
               /data
               /mapnik
       tileset2/
               /data
               /mapnik
       tileset3/
               /data
               /mapnik

If something changes in config/tileset3/mapnik then I don't want to commit that and regenerate everything in tileset1 and tileset2. A bit of GIT config in our Jenkins job covers us nicely here as you can set the includes region in your GIT Jenkins job to something like "config/tileset3/.*"

Now consider that in any complex build there might be a large number of mentions of a specific variable like this or other bits of configuration. Also, what if it wasn't just one build that was needed when I added a new folder to that set of configs. What if I actually needed a job that published these tiles to preprod and another job that publishes them to prod, using the artifacts from the first job. Suddenly just adding a folder in git has meant I have to configure 3 new Jenkins jobs with dependencies between each other. Yikes.

So, I create my 3 jobs:

  • 1: Generate tileset1
  • 2: Publish tileset1 to prod
  • 3: Publish tileset1 to preprod

Refactor jobs ready to be templated

These jobs still have everything hard coded. Now take a copy of jobs 1 and 2 call them:

  • 1: Generate tiles TEMPLATE
  • 2: Publish tiles TEMPLATE

Then you can go in and replace any mentions of "tileset1" with @tileset@ in the configuration of that job. You'll also want to disable these jobs as you'll not be running these actual jobs.

In my config I then various bits of a jenkins config with @tileset@ in a field, such as the previously mentioned GIT includes region: "config/@tileset@/.*"

The Jenkins API and CLI

There is a little used feature of Jenkins called the Remote Access API. It is a HTTP API for automating Jenkins itself. You can drive that via your own code, or to make life even easier you can use the Jenkins provided Command Line Interface. This gives you a jar file that you can download and run commands like this:

java -jar jenkins-cli.jar -s http://myjenkins/ help

Which gives:

build
    Builds a job, and optionally waits until its completion.
  cancel-quiet-down
    Cancel the effect of the "quiet-down" command.
  clear-queue
    Clears the build queue
  connect-node
    Reconnect to a node
  copy-job
    Copies a job.
  create-job
    Creates a new job by reading stdin as a configuration XML file.
  delete-builds
    Deletes build record(s).
...
...
...
 wait-node-online
    Wait for a node to become online
  who-am-i
    Reports your credential and permissions

It's quite a list of commands which I've truncated because there are almost 40 of them!

Using the Jenkins CLI to copy and modify jobs

The bits of the CLI I'm intereseted in are:

  • get-job - So that I can download the XML definition of a job
  • create-job - So that I can reupload my altered XML definition of a job and create a new job
  • enable-job - So that I can enable my new jobs once they're created

I am at home using Apache Ant for my builds so that's what I'll use to coordinate this process. Some pseudo code for the process:

  • Run ant with -Dtileset=tileset3
  • Download jenkins-cli.jar
  • New generate tiles job
    • CLI get-job for "Generate tiles TEMPLATE"
    • Copy job definition XML to "generate-tiles-tileset3.xml"
    • Replace all occurances of @tileset@ with "tileset3"
    • CLI create-job called "Generate tiles tileset3" with XML from "generate-tiles-tileset3.xml"
  • New publish tiles jobs
    • CLI get-job for "Publish tiles TEMPLATE"
    • Copy job definition XML to "publish-tiles-tileset3-prod.xml"
    • Replace all occurances of @tileset@ with "tileset3"
    • Replace all occurances of @target@ with "prod"
    • CLI create-job called "Publish tiles tileset3 prod" with XML from "generate-tiles-tileset3-prod.xml"
    • Copy job definition XML to "publish-tiles-tileset3-preprod.xml"
    • Replace all occurances of @tileset@ with "tileset3"
    • Replace all occurances of @target@ with "preprod"
    • CLI create-job called "Publish tiles tileset3 preprod" with XML from "generate-tiles-tileset3-preprod.xml"
  • CLI enable-job called "Generate tiles tileset3"
  • CLI enable-job called "Publish tiles tileset3 prod"
  • CLI enable-job called "Publish tiles tileset3 preprod"

With a build script that does this, you should be able to run it by hand from your own machine and watch as 3 new builds appear in your Jenkins instance.

Create a job in Jenkins to create the new jobs

We're almost there, now we need to create a job in Jenkins for generating our new jobs with the script we just created. The easiest way to do this is to use the Parameterized Builds plugin in Jenkins. We set a parameter called "tileset" and run our Ant build step like this:

-Dtileset=${tileset}

${tileset} is how the Parameterized Build plugin inserts parameters. Now you just run this job and enter "tileset4" and magically 3 new builds appear for you in a few seconds.

Now, you might ask if I can run a build with the Parameterized Builds plugin then why didn't I just do that at the start and not bother with the CLI....good question.

A parameterized build is a single build with a single history and a single name. I wanted to have distinct builds that I can track and that can depend on other builds and see at a glance when they were last published or last created. It gives you a lot more flexibility in the complexity of your jobs if you can create jobs programmatically.

Conclusion

Jenkins is a really powerful tool for automating your processes. However, once you get beyond a few simple builds, it can be incredibly useful to understand the power behind the scenes of Jenkins to create complex jobs and relationships between jobs.

Build automation and continuous integration are just one of the many tools we use at Black Pepper to make us work more efficiently, reducing the repetitive manual work we do so we can spend more time solving the hard problems our customers have.