"Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.
This means it may not be possible for one Python installation to meet the requirements of every application. If application A needs version 1.0 of a particular module but application B needs version 2.0, then the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run.
The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages." - from the Python Tutorial
In this tutorial we will discuss about two major methods to create and manage virtual environments in Python.
To create a virtual environment, run the venv module as a script with the directory path. For example the following code uses the venv module, -m venv
, to create tutorial-env
in the current directory:
It also creates directories containing a copy of the Python interpreter, the standard library, and various supporting files. Once we’ve created a virtual environment, we can activate it by:
After activating the environment with the above command, we can use pip
to install required packages by:
For example, the following Bash script creates tutorial-env
if it does not exist, and install redis
and gitpython
after activation:
#!/bin/bash
if [ ! -d tutorial-env ]; then
python3 -m venv tutorial-env
fi
source tutorial-env/bin/activate
pip install redis gitpython
To uninstall packges use:
To keep a list of the packages (commonly called requirements.txt
) that are installed in the env, use:
To install packages from a requirements.txt
file use:
We can use deactivate
command to deactivate and env and use rm -rf <env_path>
to remove th env.
To create virtual environments, Conda could be the best environment management system. Miniconda is an open source package and environment management system that includes Conda. Conda quickly installs, runs and updates packages and their dependencies. To start using Conda, follow the instruction in here to install Miniconda (or Anaconda if you want to have most of the scientific packages) on your operating system.
When Miniconda is installed, use conda init <shell-name>
to initiate Conda and run conda config --set auto_activate_base false
to stop auto base activation. We can also use conda update conda
to upate Conda.
For adding Conda autocompletion, in bash terminal copy conda-bash-completion in /usr/share/bash-completion/completions/conda
. And for a macOS with zsh, copy conda-zsh-completion in ~/miniconda3/zsh-completion/_conda
and add the following to the ~/.zshrc
file:
To create a new environment, use conda create
command including names of the environment and required packages:
conda create --name <env_name> <package1> <package2=version> ...
# or
conda create --prefix <env_path> <package1> <package2=version> ...
We also can use --yes
flag to set up the environment without a question. We use --prefix <env_path>
to setup the environment in a certain path. Note that we can not use --prefix
and --name
at the same time.
To see list of environments, use the following:
To activate an environment type:
Note that, for Conda versions prior to 4.6 we need to use source
instead of conda
to activate an env. Now, use the following to install new packages in the activated env:
To uninstall packages use:
To see list of the installed packages within the env, use:
To keep a list of the packages (commonly called requirements.txt
) that are installed in the env, use:
We can deactivate the env by:
And remove the deactivated virtual environment by:
To remove cache files we can use:
We can regenerate an env from a requirements.txt
file:
Note that there is a short form for most of the Conda options that can be used instead of the long form. For instance, -n
, -p
, -c
, -y
, and -e
can be used instead of --name
, --prefix
, --channel
, --yes
, and --export
, respecively. You may find more information by using conda -h
or conda <command> -h
.
Whenever we use conda create
or conda install
without mentioning a channel name, Conda package manager search its default channels to install the packages. If you are looking for specific packages that are not in the default channels you have to mention them by using:
codna create --name <env_name> --channel <channel1> --channel <channel2> ... <package1> <package2> ...
For example the following creates new_env
and installs r-sf, shapely and bioconductor-biobase from r
, conda-forge
and bioconda
channels:
codna create --name new_env --channel r --channel conda-forge --channel bioconda r-sf shapely bioconductor-biobase
Ideally, you should create one environment per project and include all the required packages when you create the environment and try to use a single channel as much as possible. It is important using the same channels for updating the environment.
To find the required packages, we can visit anaconda.org and search for packages to find their full name and the corresponding channel. Another option is using conda search
command. Note that we need to search the right channel to find pakages that are not in the default channels. For example:
In a HPC cluster system, first we need to load miniconda3
module to be able to use conda
. We can use the following as a template for building a new environment:
module load miniconda3
if [ ! -d <env_path> ]; then
conda create --yes --prefix <env_path> <package1> <package2> ...
fi
source activate <env_path>
To keep Conda packages and caches somewhere except the home directory, we can update conda pathes by exporting new CONDA_PKGS_DIRS
and CONDA_ENVS_DIRS
or creating/updating ~/.condarc
file. Review here to learn more.