Why and when to use a new anaconda environment? (One environment for everything? A new environment for every project?)

Question

I have a little bit of experience using Anaconda and am about to transition to using it much more, for all of my Python data work. Before embarking on this I have what feels like a simple question: "when should I use a new environment?"

I cannot find any good, practical advice on this on StackOverflow or elsewhere on the web.

I understand what environments are and their benefits and how, if I am working on a project that has a dependency on a specific version of a library that is different to e.g. the latest version of that library etc. etc. ... then virtual environments are the answer; but I am looking for some advice as to how to practically approach their use in my day-to-day work on different data projects.

Logically there appears to be (at least) two approaches:

1. Use one environment until you absolutely need a separate environment for a specific project
2. Use a new environment for every single project

I can see some pros and cons to each approach and am wondering if there is any best practice that can be shared.

If I should use just one environment until I need a second one, should I just use the default "root" environment and load all my required dependent libraries into that or is it best to start off with my own environment that is called something else?

An answer to this question "why create new environment for install" by @codeblooded gives me some hints as to how to use and approach conda environments and suggests a third way,

3. Create new environments on an as-needs basis, projects do not "live" inside environments but use environments at runtime, you will end up with as many different virtual environments as you need to run the projects that you regularly use on that machine, that may be just one environment or it may be more

Anyway, you can see that I am struggling to get my head around this, any help would be greatly appreciated. Thank you!

Answer 1

As a developer that works with data scientists I would strongly recommend creating an environment for each project. The benefit of python environments is that the encapsulate the requirements of a project from all other python projects.

In the case above, if you were to use Python36 for 8 different projects it would be very easy to accidentally upgrade a package or install a conflicting package that breaks other projects without you realising it.

In the work you do it might not be a big deal, but given how easy it is to create a separate environment for each project the benefits outway the small time cost.

I can tell you that if any of the developers I work with was found to be using a single python environment for multiple development projects they would be instructed to stop doing that immediately.

Answer 2

Ok, I think I worked this one out for myself. Seems kind of obvious now.

You do NOT need to create an environment for every project.

However if particular projects require particular versions of libraries, a particular version of Python etc then you can create a virtual environment to capture all of those dependencies.

To give an example,

• let's say you are working on a project that requires a library that has a dependency on a particular version of Python e.g. Python 3.6,
• and your base (root) environment is Python 3.7,
• you would create a new anaconda environment configured to use Python 3.6 (maybe call it "Python36")
• and you would install all the required libraries in that environment and you would use that environment when running that project.
• When you have another project that requires similar libraries, you may re-use your now existing Python 3.6 environment (named "Python36") to run this new project,
• you would not have to create a new Python 3.6 environment, in the same way that you would not have to install multiple instances of Python 3.6 in order to run multiple projects that required Python 3.6.