In this post, I’d like to outline how to create and manage virtual environments each with its own, independent set of dependencies with the following packages:

  • pip, A tool for installing and managing Python packages in the Python Package Index
  • virtualenv, A tool for creating isolated Python environments containing their own copy of python, pip, and other packages
  • virtualenvwrapper, A set of convenient enhancements to virtualenv

virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

The rationale behind virtual environments is to make available reproducible, isolated sets of dependencies/python packages without much configuration and without having to watch and manage the packages’ different versions.

We can have an isolated environment for each program, each containing a complete copy of everything needed to run that Python program, including the python binary itself. This allows you to effectively work on two different programs with two different sets of dependencies all on the same machine.

Isolated environments

Before we begin, type pip freeze to get the currently installed packages in your system. We will use this information for later comparison.

$ pip freeze
# An excerpt of my output
Cython==0.19.1
Flask==0.10.1
Jinja2==2.6
MDP==3.3
PIL==1.1.7
PyAudio==0.2.7
PySAL==1.5.0
PyYAML==3.10
Pygments==1.6
SQLAlchemy==0.8.1
Sphinx==1.1.3
Werkzeug==0.9.1
astropy==0.2.3
atom==0.2.3
binstar-client==0.1.0
biopython==1.61
bitarray==0.8.1
boto==2.9.6

The subset of virtualenvwrapper commands we will be using are:

mkvirtualenv #(create a new virtualenv)
rmvirtualenv #(remove an existing virtualenv)
workon #(show all env, switch current virtualenv)
deactivate #(deactivate virtualenv, which calls several hooks)

Having installed all the above packages (pip, virtualenv, virtualenvwrapper) in your system, let’s start by creating a new virtual environment:

$ mkvirtualenv myenv

The mkvirtualenv command creates does a couple of things:

  • Creates a new environment folder in your .virtualenv directory
  • Installs a new python binary into your new environment
  • Installs setuptools and pip into your new environment
  • Activates the environment

You should now see (myenv) on the left side of your shell prompt. This tells you that you currently have that virtual environment active.

So now, when we type pip freeze, we get:

wsgiref==0.1.2

Huh? wsgiref? Wait, shouldn’t a fresh virtual environment be completely free of packages? The reason wsgiref appears is because python 2.5+ standard library provides egg info to wsgireflib (and pip does not know if it stdlib or 3rd party package). Since this is the only package in our virtual environment, it’s safe to consider it as ‘fresh.’

Now, let’s install a package into our new virtual environment:

# installs requests, an HTTP library written in python
$ pip install requests
Downloading/unpacking requests
Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded
Running setup.py egg_info for package requests

Installing collected packages: requests
Running setup.py install for requests

Successfully installed requests
Cleaning up...

Let’s see what packages our virtual environment now have:

requests==1.2.3
wsgiref==0.1.2

You can install however many packages and versions of packages you’d like into your virtualenv without affecting any of your system’s packages.

What if we have multiple projects, each with its own set of dependencies? We are not limited to one virtualenv. We can type workon to see all of our virtual environments:

$ workon
django
myenv

We can switch to my other, pre-existing virtualenv 'django’ which has its own set of dependencies installed:

$ workon django
$ pip freeze
Django==1.5.1
Fabric==1.6.1
South==0.8.1
django-debug-toolbar==0.9.4
django-extensions==1.1.1
factory-boy==2.0.2
ipdb==0.7
ipython==0.13.2
paramiko==1.10.1
pycrypto==2.6
six==1.3.0
wsgiref==0.1.2

Now, let’s say that you’ve been working on your project with this virtual environment, and now you’d like to invite another developer to work on your project. How can she reproduce your virtual environment on her machine? Short answer: Requirements files.

Requirements files are plain text files that contain a list of packages to be installed and can be easily created by piping the output of pip freeze into a .txt file.

$ pip freeze > requirements.txt
# package name, version
MyPackage==3.0

Other developers can reproduce your virtual environment by creating a fresh virtual env and running a pip install with your requirements file:

$ pip install -r requirements.txt

Finally, you can deactivate your virtual environment and return to your system environment with the following command:

$ deactivate

To sum up, using pip, virtualenv, virtualenvwrapper, and requirements files helps you create isolated, reproducible environments with minimal effort.

You can read up more on advanced features of virtualenvwrapper, such as the postmkproject virtualenv hooks here.

Additional reading: