Continuous Deployment with Travis and CONDA

Continuous Integration platforms such as Travis and Azure are now ubiquitous in modern development environments. Using someone else’s computing resources automatically to test and deploy your software is certainly worth the headache of figuring out how these tools work. The following guide will push you a little further up the learning curve while using Python and CONDA.

The goal here is not just to get this working once, but how to create the lowest maintenance system possible. Note that this is not a getting started guide for either CONDA or Travis, the instructions below assume that you have an elementary knowledge of both systems. If not, here are some helpful links to get the ball rolling for Travis and CONDA respectively. What we will cover is:

• How to install CONDA
• How to install a Python package with a CONDA recipe
• How to upload a package to a CONDA

The first step is installing the Miniconda with the versions of Python we plan on using to test our package. If you are only familiar with Anaconda, Miniconda is the same package management system without the default set of preinstalled packages. This will keep our build times down. The setup below is boilerplate and can be shared between repositories without modification except for the matrix of Python versions

Installing Miniconda

Now that we have Miniconda installed we are ready to create a virtual environment with the necessary packages to execute our tests. The brute force approach looks something like this and I see it a lot in various repositories.

Not maintainable!

Now, this will work, but I see two major flaws that are quickly exposed during development cycles:

1. We have now created a second place in our repository where we need to manage our dependencies. If we need to change the environment at all we’ll need to remember to change both the CONDA recipe and this list contained in our Travis testing suite
2. Worse, we are not testing using our CONDA recipe. We won’t even notice that we are not specifying the correct packages until we try and use the CONDA package we are hoping to deploy here, meaning we will potentially deploy a CONDA package that will not work.

The solution here is to build and test with the environment specified in our CONDA recipe. I like to place my recipes in the same repository as my code, but if this is not a suitable option for your application it always possible to clone or submodule the recipe to ensure it is available on Travis. Here we assume that we have a package called pkg_name whose CONDA recipe lives in a top-level folder conda-recipe (real creative I know). This solution looks something like this:

The trick is we first build our package using the recipe, and then install it straight from the filesystem as if it was a channel hosted elsewhere. This is as close as possible to replicating the code that users will receive when they install the package from our repository. You may notice the additional dev-requirements.txt file used when we create the environment. This is where packages go that are only required to run the test suite, we don’t want to burden our users with extra dependencies, but we do want to be explicit to other developers about the environment they need to be in when running the test suite.

If switching from pip installing your package you may notice that this method takes longer to complete. In general, CONDA seems to be a little slower than pip, and the additional step of building our package before we test adds an overhead. However, if we are going to build and ship our package to our CONDA channel anyways this step has to be complicated, it is only a little bit obnoxious that it increases the time before we see whether or not the tests fail. I’d argue that the comfort knowing that the CONDA recipe you are deploying has been fully vetted is worth the extra minute or two, that’s part of the benefit of using someone else’s hardware to do all this work.

The actual invocation of your test suite is probably unique to your repository. The simplest execution may look something like this if you pytest.

I personally like to run my pytest suite with the verbose option, but you should select how the output should be structured based on your own personal taste.

Now, that we have executed the tests and are confident in the code we’ve created, we are ready to upload our package to our CONDA channel. Luckily, since we already built our package to test it, this step should be fairly short. We do have to watch out for a few ‘gotchas’ though. First off, we may only want to upload if we are on a specific branch. We also have to be careful that if someone forks this repository that their Travis builds won’t upload files. Below I’ve put an example of what this may look like:

Some environments variables that may be useful are:

• TRAVIS_PULL_REQUEST: (true/false) based on whether the Travis build was triggered by a pull request.
• TRAVIS_REPO_SLUG: The organization and repository name from which the Travis build was triggered
• TRAVIS_BRANCH: The name of the branch that triggered the Travis build
• TRAVIS_TAG: If the Travis build was triggered by a tag, it can be found in this variable. Otherwise, it will be empty.

You may notice that we utilize the an environment variable $CONDA_UPLOAD_TOKEN in the Travis file. This is a key that we generate on the Anaconda site that gives permission to upload new packages. Keep in mind that this token gives anyone the power to upload packages to your repository. Treat it with the same care you would a password.To generate the token:

1. Sign into https://anaconda.org and navigate to the organization where you wish to upload the package.
2. Click on your username in the uppermost corner and navigate to Settings -> Access. You will most likely be asked to re-authenticate at this point.
3. Use the form to create a new token. Give it a descriptive name and at least read and write access.

Your token will require both read and write privileges

4. Once created, the token will be visible in the table below. View it and copy it into your clipboard.

Now, you have a few options on how to add the environment variable as a token on Travis. The simplest is to navigate to the repository on the Travis website and find the “More options” menu in the upper right corner. From there, enter the “Settings” tab and use the prompt to add your token with the name matching the environment variable we wrote in the travis.yml file above. Make sure that the value is not set to display in the log output of your build.

Environment variable entry in Travis

Committing the changes to your Travis file to your repository should automatically trigger a build. You can check whether or not the package was properly uploaded at the conclusion of the build by looking in your list of builds on the Anaconda page, or use conda search from the command line. Note that if you are worried about another channel you have added to your Conda setup having a duplicate package you can always add the --override-channels option. The whole file can be found here.

And there you have it! Hopefully this is helpful. I couldn’t find a complete guide anywhere, so I figured I’d write out my process. Feel free to chime in with any feedback you may have.