heya! this post is about how i added ccache to SuperCollider’s continuous integration (CI) setup, and the kind of benefit it had for our build times.

i’ll also be explaining how i managed to make it work with both of TravisCI’s linux and macOS environments, which required a somewhat tricky workaround when it came to using cmake, ccache, and xcode at the same time.

background

this part’s about how i got interested in this problem in the first place. if that’s not what you came here looking for, skip a bit farther down to the next section.

with the upcoming 3.9 release, i recently began looking into SuperCollider’s TravisCI integration. it’s a bit dusty at this point, since previous maintainers have either become active or stopped working on that part of the software. i’ve touched it a few times, and read over the TravisCI docs a bit when i was helping on another project (james bean’s dn-m). i’m no expert by any stretch of the imagination, but i think these types of development tools are incredibly important for a project of this size, so i took it upon myself to check it out.

an important thing to know, if you’re not familiar with TravisCI, is that it supports both linux and macOS environments, and our setup takes advantage of that fact, with two build configurations for linux and one for macOS.

i went line-by-line through our main .travis.yml file – the file that provides overall configuration and build scripts to travis – and through the scripts it referenced. it didn’t take long before i noticed a few points of confusion:

  • the language key wasn’t set to “cpp”, so our project was being recognized as Ruby. while this didn’t cause any major problems, it was a little unsatisfying and, according to the docs, affected the environment handed down to our builds. the only impact i could see was that our compiler key – set to gcc – was being completely ignored, which is fine; that’s already the default on linux builds, and xcode builds are more common among our macOS devs.
  • a few environment variables were being set that seemed to have no purpose. removing them from the script and retrying the build confirmed they were zombies. this was a sign to me that it really was a dusty, dusty config, and that i could probably do at least a few things to improve it.
  • the cache key was mapped to “bundler” and “apt”, but neither of these seemed to do anything of value. according to the docs, “apt” isn’t a valid key to put there, and our project doesn’t use Ruby at all, so caching bundler (the package manager for Ruby) was similarly pointless. this got me interested in caching, as i’ll discuss soon.

i also used travis’s little linter they designed for linting .travis.yml files. that revealed that someone had mistyped the name of a build stage: “after-deploy” instead of “after_deploy”! it was barely consequential, since all that bit did was echo a single line to the console. but i’m glad i ran it. overall, travis-lint seemed like a nice program, but its output produced a few false-positive warning messages, and according to its github page it has a number of known issues, so i would advise to not take it too seriously.

there are a couple other issues i noticed with our setup that i’d like to tackle in the future, but i focused on one that seemed achievable without too much work: making use of travis’s caching feature.

ccache + TravisCI

one of the features TravisCI advertises for C++ projects is its ability to cache the artefacts of ccache. i actually have to say that i didn’t know anything about ccache before i got to this point, but i’ll definitely be using it for supercollider from now on. in a nutshell, ccache sits at the mouth of your compiler invocations; when it detects that you’re compiling a translation unit that it’s seen before, with settings that it’s seen before, it just spits out the prerecorded “compilation matter”, which includes the target object file and compiler messages along with any warnings. that means you can cache builds even if you completely remove the build directory and start clean, something i frequently do while toying around with cmake configs.

for our two linux builds, using ccache in travis was super easy – all i had to do was add the value - ccache to the cache: section of our .travis.yml file. i instantly saw fantastic build time speedup, which i’ll discuss in a bit. for macOS, the solution was a bit more complicated. travis claims that you just need to add a few extra lines to your script to get ccache to work for macOS builds:

ccache is not installed on OS X environments but you can install it by adding

install:
  - brew install ccache
  - export PATH="/usr/local/opt/ccache/libexec:$PATH"

but as far as i could tell that’s simply not true. reading the build logs, it was clear that xcode was still being invoked, and looking at the build times, it was very clear that nothing was changing. our build time was still hovering around 15-16 minutes.

it seems that the issue is the way cmake handles its xcode generator. a typical solution to using ccache with cmake is to symlink ccache directly to the names of the compiler(s) you want to invoke, so that you’re always compiling through the sticky wrapper of ccache. with cmake’s xcode generator, the build commands directly reference the the compiler in its /Applications/XCode/… directory, and it’s unclear whether symlinking those binaries would be a good idea. i wasn’t interested in testing that theory just yet.

there are also options in cmake for doing this kind of thing. RULE_LAUNCH_COMPILE and CMAKE_C[XX]_COMPILER_LAUNCHER (new in v3.4) allow you to request a command with which to prefix the compiler invocation. this would be exactly what we want, but again, we’re put in a tight situation, because cmake only supports these options for Makefile and Ninja generators.

luckily, i came across a fantastic post by Craig Scott that explains a lovely, simple workaround for this problem. i won’t go into detail – you can read his post if you’d like a more thorough explanation of what’s wrong here, and how he found his workaround. the practical outcome is that by adding a few lines of configuration to the main CMakeLists.txt and a few configuration scripts in a seperate location, we can achieve the same exact result as you would see while attempting to use cmake + ccache with makefile generators.

in short time, i integrated his suggestions with our cmake configuration and tested it on my machine. once i saw that it was working (and fantastically reducing my fresh build time by more than half!), i pushed the changes to my github repository to see what the time difference would be for a travis build.

results

you can see the new .travis.yml, old build time, and new build time for yourself. here are the results summarized:

build environment old build time new build time time savings
linux, without building the Qt-based IDE 8’50” 2’27” 6’23”
linux, with Qt-based IDE 17’22” 3’40” 13’42”
macOS, with Qt-based IDE 14’08” 7’53” 6’15”
(total) 30’20” 14’00” 16’20”

this is a time savings of 53%. for the linux+IDE build, it’s 79%.

analysis

i was amazed to see these results: not only will we get faster build times, but it took barely any extra effort and will not affect our compilation logs in any way (to the best of my knowledge, and i looked them over pretty closely). i’m very happy with how this turned out, and i hope that my pull request to add it to our main config gets merged soon.

because i’m curious about these sorts of things, i estimated the power savings this change would offer. note: this is going to be very unscientific.

from this paper by Shea, Wang, and Liu, and this paper by Stoess, Lang, and Bellosa, and knowing nothing about what equipment travis actually uses for their builds, i estimated the average consumption of one VM at about 60 Watts. let’s consider that between merges and pull requests, supercollider averages about 2 changes per day, which translates to 6 builds per day. so, how much power would that save over the course of a year?

365 days * 6 builds/day * 16.33 minutes/build * 1/60 hours/minute * 60 Watts = 35.8 kWh

that’s a bit more than the cost it takes to power an American home for one day (using [this statistic][us-eia] from 2015, the number is about 29.6 kWh. it may not seem like a lot, but i think we have a responsibility as technologists to remain conscious of the cost of the tools we use, however slight it may seem. it all adds up.

conclusion

my music recommendation for this week is a great album by Contemporary Noise Quintet called Pig Inside the Gentleman. in particular, i’m really into the fourth track, Goodbye Monster. it’s brooding and focused, and it’s the kind of thing i could imagine playing in the late stages of a dinner party.