3

I use AWS SageMaker for ML software development. In SageMaker, there are several conda environments to choose from. I need to upgrade some packages in a conda environment that are pip installed. From my research, pip and conda are not compatible. So what is the best way to upgrade these pip-installed package?

As an example, the below image shows a conda_tensorflow_p36 environment and the keras package is pip installed. I want to upgrade the keras package to the current version. How do I do that?

Screenshot showing pip keras and conda keras are incompatible

charlesreid1
  • 4,360
  • 4
  • 30
  • 52
David293836
  • 1,165
  • 2
  • 18
  • 36
  • 1
    You can't use `conda` to upgrade `pip` packages. Have you tried simply using `pip`? i.e. `pip install keras --upgrade` – iz_ Jan 12 '20 at 23:48
  • Thanks. But how do I ensure the keras upgrade is compatible with other conda installed packages? – David293836 Jan 13 '20 at 00:02
  • This might be of use: https://www.anaconda.com/using-pip-in-a-conda-environment/. Not directly related to your question, but why was Keras of all things installed using Pip? – AMC Jan 13 '20 at 00:05
  • Thanks for the link. These are pre-installed packages on AWS SageMaker conda environment. I am wondering the same thing too. – David293836 Jan 13 '20 at 00:31
  • It might be worth trying to [enable Pip interoperability](https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/pip-interoperability.html). It's a newish feature (still considered experimental), but it could potentially enable Conda to deal with the upgrading of Pip installed packages *when newer versions are available through Conda*. – merv Jan 13 '20 at 06:31
  • SageMaker installs packages using a conda environment, not using pip - see https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html – charlesreid1 Jan 14 '20 at 00:04
  • To charlesreid1, that is true for adding new packages. In other words, when installing a new package, SageMaker prefers that one uses conda. However, the existing packages within the pre-defined conda environments use mixed combinations of conda and pip. Try to open a terminal and activate a virtual environment. Then do "conda list". – David293836 Jan 14 '20 at 20:15
  • To merv, that works for conda V4.6. SageMaker's conda is 4.5.12. It is too early to do that on SageMaker, IMO. It might be okay for experiments or research. But for deployment software, it is too risky. – David293836 Jan 14 '20 at 20:19

1 Answers1

1

You need to specify the name of the conda environment to use when upgrading, so change your conda upgrade keras command to:

conda upgrade -n conda_tensorflow_p36 keras

EDIT: Alternatively, the Install External Libraries and Kernels documentation page for SageMaker gives an example script that downloads/installs an entirely new version/instance of miniconda from the notebook. Then any packages you need (including keras) can be installed into that miniconda instance, independent of the versions provided by SageMaker.

charlesreid1
  • 4,360
  • 4
  • 30
  • 52
  • Thanks. But this does not work. I received similar error messages. – David293836 Jan 13 '20 at 23:49
  • Can you update/edit your question with the formatted output? – charlesreid1 Jan 13 '20 at 23:51
  • That link is about intalling external libraries and kernels. My issue is regarding the existing pip installed packages within a conda environment on SageMaker. – David293836 Jan 14 '20 at 23:25
  • You stated you wanted to upgrade your version of keras. The link's suggestion is to install an up-to-date version of keras as an _external_ library in an _external_ conda distribution. – charlesreid1 Jan 15 '20 at 18:14
  • Thanks. Keras is an example of pip installed package in AWS SageMaker. There are many packages like that in AWS SM. I supposed I should use pip to upgrade each package, as opposed to using conda to install/upgrade these pip installed packages. From what I read so far, using conda to install and upgrade packages that are pip installed would create problems. However, what kind of problems may arise if I use pip to upgrade these pip-installed packages in a conda environment? – David293836 Jan 16 '20 at 23:20
  • any solution here @David293836 ? i'm having the same issue with fuzzywuzzy – nerdlyfe Jul 16 '20 at 22:22
  • 1
    @ElChapo No solution here. I decided not to use SageMaker for this reason. I need to be able to manage packages. It is easier to do it without SageMaker. – David293836 Jul 17 '20 at 21:07
  • damn..brutal. i had to do a ton PYTHONPATH changes to get it to work...sad stuff, but agree. – nerdlyfe Jul 19 '20 at 19:11