Moving Conda Environments with conda-pack

2 min read

A CKM Developers’ Blog

Why would you use conda-pack?

Sharing a Python project environment can be done with a spec list (for computers on the same OS) or an environment.yaml (if you are sharing across OS or platforms). But if you ever need to move a Python environment without internet access, conda-pack could be your answer. You might end up in this situation due to poor internet connection, or because of data security reasons and a client wanting to keep your working environment siloed.

Conda-pack is a command line tool allowing you to archive a conda environment, along with all the binaries of the packages installed on the environment. It is a great tool for reproducing a conda environment with limited or no internet access.

Conda-pack is specific to both the platform and OS, so ensure you set up your environment on the same platform and OS as you plan to use it on. It may be helpful to spin up a remote machine on AWS to achieve this.

Step 1 – Set Up

Create a conda environment as you usually would, along with the package requirements, e.g.

conda create -n my_env python=3.8
conda activate my_env
conda install pip
pip install -r requirements.txt

Remember to include the installation of Jupyter if you use Jupyter notebooks for your projects, and ipykernel in order to make the environment available in your Jupyter kernel list.

Step 2 – Pack

Install conda-pack. I recommend you do this from your root conda environment, as it will then be available in all your sub-environments as well:

conda install -c conda-forge conda-pack

Use conda-pack to package your environment by running:

conda pack -n my_env

This produces a .tar.gz file of the environment with the same name, my_env.tar.gz. Or you can specify the output name and file extension:

conda pack -n my_env -o output_name.zip

If you have created the environment on a remote machine and it is too big to upload to GitHub, try sharing via OneDrive or SharePoint.

Step 3 – Move

Assuming limited internet access on the target machine or client-side, you will need to get the client to drop the packaged environment into the server that you’ll be working on, along with a copy of Anaconda installer.

Once you have installed anaconda, you can extract your packaged environment into the anaconda3/envs/ folder.

Then it is simply a case of activating the environment with:

conda activate my-env

And if you need to access the environment from a Jupyter notebook kernel, run:

python -m ipykernel install --user --name=my_env

Conclusion – the conda-pack benefit

Having a good internet connection is often an assumption in modern working environments. But with concerns around security and information sharing becoming more prevalent, it is likely that if you work on client machines, you will encounter a situation in which internet access is not available.

Moving a codebase is a simple process, and now with conda-pack in your arsenal, moving your environment is straightforward as well.

We are always looking for Data Scientists and Developers to join our team! If interested, read more about us and apply here.

You may be interested in other technical blogs as part of this series:

Analyzing SQL queries with EXPLAIN ANALYZE and PEV2