17

As Travis-CI is evolving and extending its feature set it naturally becomes nicer and nicer to use. I recently read this article about "Speeding up the build". A build for the Django project I am working on takes ~25-30 minutes. Almost half of this time is spent on creating the virtualenv, i.e. installing the project's requirements. The other half of the time is used for the actual test run.

There are efforts on cutting down runtime for tests. Yet, I was wondering whether a bigger speed-up was up-for-grabs by caching or bundling the project's requirements. As for Plone there seem to be some options as it uses buildout. I was also looking at WAD. Of course, when caching the requirements they will need to get invalidated upon a requirements update.

Has anyone made any Travis build speed improvements for a (Django) project by cutting down setup time?

mamachanko
  • 886
  • 1
  • 7
  • 15

3 Answers3

19

Update This is now a first class feature of Travis: http://blog.travis-ci.com/2013-12-05-speed-up-your-builds-cache-your-dependencies/

I've just been playing around with this, and it looks like you can cache the virtualenv site-packages like this (update the path to your python version):

cache:
  directories:
    - /home/travis/virtualenv/python2.7/lib/python2.7/site-packages

There's a little issue that it doesn't cache the bin or the src directories. I tried caching the whole virtualenv directory, but I get strange errors for dependencies installed via git into the src directory.

You are still left with the problem of invalidating old requirements. if you remove something from the requirements, it will persist in the virtualenv so you either have to explicitly remove it with pip (pip remove foo) or wait until Travis create an API to invalidate the cache...

The other option is to use the --download-cache option for pip, then add that directory to the cache:

cache:
  directories:
    - $HOME/.pip-cache/

install:
  - pip install -r requirements.txt --download-cache $HOME/.pip-cache

This will make the downloads faster, but it will still have to compile and install all of the requirements!

John Oxley
  • 14,698
  • 18
  • 53
  • 78
Simon Hewitt
  • 291
  • 2
  • 5
  • It's officially a feature now including an invalidation API http://about.travis-ci.org/blog/2013-12-05-speed-up-your-builds-cache-your-dependencies/ Yet, they don't get specific for pip caching. – mamachanko Dec 06 '13 at 09:23
  • 2
    thank you! pip install with --download-cache works great for me; build time went from 15m to 3m :) – Tommaso Barbugli Feb 05 '14 at 16:01
  • 4
    You could also append "PIP_DOWNLOAD_CACHE=$HOME/.pip-cache" to your "env" lines in .travis.yml (instead of --download-cache on the pip command line) and this will be inherited by any tox-run pip commands as well. – Jason Antman Jan 06 '15 at 03:26
  • 1
    Instructions on caching pip dependencies can be found here: http://docs.travis-ci.com/user/caching/#pip-cache – James Brewer Apr 10 '15 at 10:38
  • 1
    Note this info is out of date now, Using it throws: `DEPRECATION: --download-cache has been deprecated and will be removed in the future. Pip now automatically uses and configures its cache.` – shacker May 23 '16 at 23:00
10

This has gotten even easier over the years. The latest way is:

cache: pip

That's it.

mlissner
  • 17,359
  • 18
  • 106
  • 169
  • the other answers are good if your language is different than the cache, for example using a ruby image but running python script (while getting python using addons>apt>packages) – Pixel May 21 '20 at 16:53
6

With pip 7:

cache:
  directories:
    - $HOME/.pip-cache/

install:
  - pip install --upgrade pip
  - pip install -r requirements.txt --cache-dir $HOME/.pip-cache
Tomasz Hemperek
  • 111
  • 2
  • 3