Under the Hood at Thinkful: Continuous Integration (CI) Rollout


At Thinkful, our belief in education through mentorship is reflected not only in the way we teach, but also in the fact that we’ve added five interns to our team this summer. Jason, one of our developer interns, is a sophomore computer science major who’s been programming since he was five (his first project was a robot that sorted gumballs by color.) He started last week and has already completed a project to improve developer workflow at Thinkful. Here’s his description of the process:

My first project at Thinkful was to develop a workflow for allowing developers to have confidence when deploying their code to production. We’re starting with integrating Continuous Integration (CI) into deployment.

After going through many, many different continuous integration platforms, the choice became clear that Codeship was the way to go. Some of the other choices were CircleCI, Tddium, Jenkins and Travis, but some major flaws in each allowed Codeship to come out on top.


: I spent three days working with both the CircleCI product and their two co-founders. I was attracted to CircleCI because it supports the full web stack and integration with github and Heroku. Following

Heroku’s best practices

, we store settings (database passwords, etc.) specific to each deployment (dev, staging, production) as environment variables. However, CircleCI requires this environment to be stored in a public file in the root of your repository. I tried “hacking” their system in a few ways, which included adding a private repository containing the environment as a submodule to our public repository, as well as cloning a private repository containing the information into our public one during the continuous integration process. Unfortunately, Circle’s backend prevented these hacks from working, and therefore, it was not a viable option.

Update (7/1/13): The dedicated team at CircleCI has since fixed this issue: there is now support for secure environment variables.  We now think they’re worth a second shot. 

Tddium: Tddium originally seemed like the frontrunner. It supports a Python test suite, integrates with any Git server, runs tests in parallel, sends build notifications to HipChat, GitHub, and email, and has a Heroku add-on to integrate with the deployment process. However, node.js support currently does not exist, which means we couldn’t test AngularJS. Tddium also has the same problem with storing public environment variables, like CircleCI. Customer service was very good, though, in helping me solve the problems I encountered.

Update (7/3/13): Node.js support in Tddium is now in beta. Also, private environment variables are now supported at either an account (meaning, across all repos) or within a single branch. Our environment requires private environment variables by repo, which Tddium has promised.

Jenkins: This was the only internally hosted solution I tried. While Jenkins is fully customizable, and has the ability to perform any tests that you want with installable plugins, the interface felt outdated, and the maintenance required to keep the server running with zero problems did not make sense to implement at this point. Though on the plus side, Jenkins supported multiple users, and the fact that it would be internally hosted provided for a safer location to store key environment variables.

Travis: I spent minimal time investigating Travis, as it was the only cloud hosted, fully supported, and free continuous integration system I came across. However, Travis only supports open source repositories, and upon realizing that the Thinkful codebase may not always be public, I moved on to the next product.

Codeship: With the suggestion of the Matchbook team, I took a look at Codeship. It immediately became clear that this was the best solution. With GitHub integration, Heroku deployment, the ability to use node.js and Python (and pretty much any other languages that can be installed on the command line), and the ability to have multiple accounts per repository, there are almost no problems that come to light right now. Furthermore, Codeship solves the problem that CircleCI and Tddium had: environment variables can be preloaded by the owner of a repository into their system. This way, these key usernames and passwords are never committed to a repository, but simply live in a secure area of codeship’s web app. There is one small bug in running Python tests, but their founder, Florian, already sent me a temporary solution, and I was told their documentation regarding this problem would be fixed by this week. Another cool feature is a continuously updated GIF of the build status of the repository that can be used to determine if a build was successful or not.