Loading custom conda envs not working in SageMaker

Question

I have installed miniconda on my AWS SageMaker persistent EBS instance. Here is my starting script:

#!/bin/bash

set -e

# OVERVIEW
# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
# that these custom environments are available as kernels in Jupyter.
# 
# The on-start script uses the custom conda environment created in the on-create script and uses the ipykernel package
# to add that as a kernel in Jupyter.
#
# For another example, see:
# https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html#nbi-isolated-environment

sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
WORKING_DIR=/home/ec2-user/SageMaker/

for env in $WORKING_DIR/miniconda/envs/*; do
    BASENAME=$(basename "$env")
    source "$WORKING_DIR/miniconda/bin/activate"
    source activate "$BASENAME"
    pip install ipykernel boto3
    python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)"
done
# Optionally, uncomment these lines to disable SageMaker-provided Conda functionality.
# echo "c.EnvironmentKernelSpecManager.use_conda_directly = False" >> /home/ec2-user/.jupyter/jupyter_notebook_config.py
# rm /home/ec2-user/.condarc
EOF

echo "Restarting the Jupyter server.."
restart jupyter-server

I use this in order to load my custom envs. However, when I access the JupyterLab interface, even if I see that the activated kernel is the Custom one, the only version of python running on my notebook kernel is /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/python:

I also inspected the CloudWatch logs, and I see this error log: Could not find conda environment: [custom_env].

But, when I run the commands of the starting script within the JupyterLab terminal, conda succeeds in finding those envs. So the question is: what am I missing?

Thanks a lot.

Answer 1

Using !which python in a jupyter cell will always use the default system python.

But, if you selected your custom kernel in jupyter, the python used behind the scenes is the right one, you can verify it by comparing :

!python --version

!/home/ec2-user/SageMaker/miniconda/envs/<YOUR_CUSTOM_ENV_NAME> --version

Answer 2

Create a custom SageMaker Image with your kernel preloaded https://docs.aws.amazon.com/sagemaker/latest/dg/studio-byoi.html

Answer 3

I faced the same issue as this, and couldn't find a way out. Found a simple and straightforward workaround solution which I tried ( but with very minor modifications), where the kernel is not registered using python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)" command, but the Conda kernel is made to persist through the use of symlinks created in already existing anaconda3 environment.

Please refer this - https://medium.com/decathlontechnology/making-jupyter-kernels-remanent-in-aws-sagemaker-a130bc47eab7 and try it for yourself. Thanks.

Answer 4

I faced the same issue with not working custom kernel in the JupyterLab interface.

And I found this solution:

• first, in the SageMaker terminal, create custom conda environment (you can indicate python version as well) and install dependencies with these commands:

conda create -n custom_kernel_name python=3.6
source activate custom_kernel_name
pip install ipykernel

• install your kernel with ipykernel

python -m ipykernel install --user --name custom_kernel_name --display-name "custom_kernel_display_name"

• Then I found out, that something is wrong with kernel.json (wrong paths and launch commands), so you need to change it. Go to its location

cd /home/ec2-user/.local/share/jupyter/kernels/custom_kernel_name

• open kernel.json file, for example with nano

nano kernel.json

• change the content of kernel.json to this

{
  "argv": [
    "bash",
    "-c",
    "source \"/home/ec2-user/anaconda3/bin/activate\" \"/home/ec2-user/anaconda3/envs/custom_kernel_name\" && exec /home/ec2-user/anaconda3/envs/custom_kernel_name/bin/python -m ipykernel_launcher -f '{connection_file}' "
  ],
  "display_name": "custom_kernel_display_name",
  "language": "python",
  "metadata": {
    "debugger": true
  }
}

After this, you will be able to open Jupyter Notebook through Launcher (or File - New - Notebook) with your custom kernel. Use !python --version and !which python in this Notebook to be sure of using your custom kernel settings.

Answer 5

This is @Anastasiia Khil's answer, with a bit more abstraction and inline comments. They key part you were missing was updating kernel.json so the !jupyter cli was in your env.

#Set the internal & display names, pick your python
CKN=custom_kernel_name
CKNAME=$CKN
PYV=3.8
#create and activate the env
conda create -y -n $CKN python=$PYV
source activate $CKN
# Install ipykernel
pip install ipykernel
python -m ipykernel install --user --name $CKN --display-name $CKNAME
# Update kernel.json to match the others from SageMaker, which activate the env. 
cat >/home/ec2-user/.local/share/jupyter/kernels/$CKN/kernel.json <<EOL
{
  "argv": [
    "bash",
    "-c",
    "source \"/home/ec2-user/anaconda3/bin/activate\" \"/home/ec2-user/anaconda3/envs/$CKN\" && exec /home/ec2-user/anaconda3/envs/$CKN/bin/python -m ipykernel_launcher -f '{connection_file}' "
  ],
  "display_name": "$CKNAME",
  "language": "python",
  "metadata": {
    "debugger": true
  }
}
EOL

Answer 6

I had a similar issue. Sagemaker can't switch the Conda environment because of shell configuration issues. Here are the steps I followed, and it worked:

• Open Jupyter Lab
• From Launcher, open a terminal and run the following commands:

bash # switch to bash
conda create -n your_kernel_name python=3.9.7 # any kernel name and python version you want
conda activate your_kernel_name
pip install ipykernel
python -m ipykernel install --user --name your_kernel_name --display-name your_kernel_display_name

And you're done! The new your_kernel_display_name should be available now for any notebooks.