Why to Jenkins

Confession time: back when I was a consultant, I didn’t always use best practices. I know – that’s a loaded phrase anyway, but this wasn’t good. My name is Allister Banks, and I set up autopkg to run as root with a launchdaemon. In my defense, it was on an in-warranty Xserve, and I hadn’t seen Greg’s presentation at MacSysadmin.se 2013 where he showed that he had set it up with Jenkins… because that talk was still months in the future at that point.

When writing autopkg recipes I'm often glad that doesn't run as root. #macadmin pic.twitter.com/NWLIxuxarL

— Ipcress Rowley (@toofrag) December 1, 2014

Phew! Feels good, I feel lighter. Ok, now on to what I’d much rather you remember from this post.

Autopkg and Reposado both fit into the Jenkins model for me, as it’s something I want to run on a particular schedule, and Jenkins is purpose-built to send detailed notifications with minimal effort. I’ll follow up with a post about autopkg in the future, we’ll just cover Reposado for this post. (I’m also glossing over Jenkins setup, and how I’m using cascading to stage SUS updates before syncing them up to the production Reposado instance, which saves space and ‘push to prod’ time.) There are a few hoops to jump through when piecing a good workflow together, so I’ll detail the theory and execution behind each.

First, (with the exception of AD,) it’s common knowledge real servers run *nix, but we’re using macs as our build nodes for both autopkg and Reposado at present. For now I’m simply using the ssh connection agent Jenkins bundles to actually connect to the Macs and run commands. Setting that up is simple, but at present seems to require Java 6 on the Mac. If anyone has figured out how to get it to point to modern Java please do let me know.

For Reposado we want to know when something new was caught by a repo_sync, and what actions it took, like deprecating/replacing older versions. I came across a gist a while back which just grabs stdout and stderr for autopkg. I’ve modified it to be like so, in a script called repo_build in the workspace that Jenkins reaches out to on the build node and runs a few times daily:

<code>#!/bin/bash
TIMESTAMP=`date "+%Y-%m-%d_%H:%M"`
/path/to/reposado/code/repo_sync 2&gt;&amp;1 | tee /path/to/writable/logdir/repo_sync__$TIMESTAMP.log

RESULT=`grep Downloading /path/to/writable/logdir/repo_sync__$TIMESTAMP.log`
if [ -n "$RESULT" ]; then
    echo "SUS caught somethin'"
    exit 1
else
    echo "Nothing doin'."
    exit 0
fi
</code>

Email notifications are only triggered with the Jenkins bundled Mailer plug-in when it catches a failure, which is why we exit 1 when changes are detected. Superficially it barks that a build has ‘failed’, but the mailed output is really great to get a head-start on researching what needs to be done next. There are times that something in the catalog gets bumped without downloading or deprecating packages, but it’s infrequent enough that I don’t mind the false positives while we’re still getting used ot the flow of reposado’s general behavior. Folks are more than welcome to bikeshed the code in the comments!

Be on the lookout for part two, where we get to the main event with autopkg.