"Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.
This means it may not be possible for one Python installation to meet the requirements of every application. If application A needs version 1.0 of a particular module but application B needs version 2.0, then the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run.
The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages." - from the Python Tutorial
In this tutorial we will discuss about two major methods to create and manage virtual environments in Python.
To create a virtual environment, run the venv module as a script with the directory path. For example the following code uses the venv module, -m venv, to create tutorial-env in the current directory:
It also creates directories containing a copy of the Python interpreter, the standard library, and various supporting files. Once we’ve created a virtual environment, we can activate it by:
After activating the environment with the above command, we can use pip to install required packages by:
For example, the following Bash script creates tutorial-env if it does not exist, and install redis and gitpython after activation:
#!/bin/bash
if [ ! -d tutorial-env ]; then
python3 -m venv tutorial-env
fi
source tutorial-env/bin/activate
pip install redis gitpythonTo uninstall packges use:
To keep a list of the packages (commonly called requirements.txt) that are installed in the env, use:
To install packages from a requirements.txt file use:
We can use deactivate command to deactivate and env and use rm -rf <env_path> to remove th env.
To create virtual environments, Conda could be the best environment management system. Miniconda is an open source package and environment management system that includes Conda. Conda quickly installs, runs and updates packages and their dependencies. To start using Conda, follow the instruction in here to install Miniconda (or Anaconda if you want to have most of the scientific packages) on your operating system.
When Miniconda is installed, use conda init <shell-name> to initiate Conda and run conda config --set auto_activate_base false to stop auto base activation. We can also use conda update conda to upate Conda.
For adding Conda autocompletion, in bash terminal copy conda-bash-completion in /usr/share/bash-completion/completions/conda. And for a macOS with zsh, copy conda-zsh-completion in ~/miniconda3/zsh-completion/_conda and add the following to the ~/.zshrc file:
To create a new environment, use conda create command including names of the environment and required packages:
conda create --name <env_name> <package1> <package2=version> ...
# or
conda create --prefix <env_path> <package1> <package2=version> ...We also can use --yes flag to set up the environment without a question. We use --prefix <env_path> to setup the environment in a certain path. Note that we can not use --prefix and --name at the same time.
To see list of environments, use the following:
To activate an environment type:
Note that, for Conda versions prior to 4.6 we need to use source instead of conda to activate an env. Now, use the following to install new packages in the activated env:
To uninstall packages use:
To see list of the installed packages within the env, use:
To keep a list of the packages (commonly called requirements.txt) that are installed in the env, use:
We can deactivate the env by:
And remove the deactivated virtual environment by:
To remove cache files we can use:
We can regenerate an env from a requirements.txt file:
Note that there is a short form for most of the Conda options that can be used instead of the long form. For instance, -n, -p, -c, -y, and -e can be used instead of --name, --prefix, --channel, --yes, and --export, respecively. You may find more information by using conda -h or conda <command> -h.
Whenever we use conda create or conda install without mentioning a channel name, Conda package manager search its default channels to install the packages. If you are looking for specific packages that are not in the default channels you have to mention them by using:
codna create --name <env_name> --channel <channel1> --channel <channel2> ... <package1> <package2> ...For example the following creates new_env and installs r-sf, shapely and bioconductor-biobase from r, conda-forge and bioconda channels:
codna create --name new_env --channel r --channel conda-forge --channel bioconda r-sf shapely bioconductor-biobaseIdeally, you should create one environment per project and include all the required packages when you create the environment and try to use a single channel as much as possible. It is important using the same channels for updating the environment.
To find the required packages, we can visit anaconda.org and search for packages to find their full name and the corresponding channel. Another option is using conda search command. Note that we need to search the right channel to find pakages that are not in the default channels. For example:
In a HPC cluster system, first we need to load miniconda3 module to be able to use conda. We can use the following as a template for building a new environment:
module load miniconda3
if [ ! -d <env_path> ]; then
conda create --yes --prefix <env_path> <package1> <package2> ...
fi
source activate <env_path>To keep Conda packages and caches somewhere except the home directory, we can update conda pathes by exporting new CONDA_PKGS_DIRS and CONDA_ENVS_DIRS or creating/updating ~/.condarc file. Review here to learn more.