6

I recently follow the instructions to install Glue Interactive sessions with vscode, but I cannot find pyspark kernel (only can see "Glue spark"). I think I have installed everything below (btw I am on Windows)

pip3 install --upgrade jupyter boto3 aws-glue-sessions
pip3 show aws-glue-sessions
cd <site-packages location>\aws_glue_interactive_sessions
jupyter-kernelspec install glue_pyspark
jupyter-kernelspec install glue_spark

But I can run jupyter notebook on terminal to open jupyter notebook working with pyspark without a problem. I guess the issue only happens on working with vscode.

Anyone can give a suggestion what to do?

vscode screenshot:
vscode screenshot

Markus Meyer
  • 3,327
  • 10
  • 22
  • 35
Ryan
  • 71
  • 3

1 Answers1

2

According to this bug report this issue was fixed a while back VSCode Jupyter Issue 8286

However I'm still experiencing it as are others, as mentioned in the bug report setting the path the full path in the kernel spec file will solve the problem. The kernel spec location depends on where it was installed (user, system or virtual env)

Example using venv: /home/{user}/glue/venv/lib/python3.10/site-packages/aws_glue_interactive_sessions_kernel/glue_pyspark/kernel.json

Update it as follows:

{
  "argv": [
    "/home/{user}/glue/venv/bin/python",
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_pyspark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Glue PySpark",
  "language": "python"
}

then update the kernel spec: jupyter kernelspec install /home/{user}/glue/venv/lib/python3.10/site-packages/aws_glue_interactive_sessions_kernel/glue_pyspark

Dave
  • 36
  • 2
  • Thanks, Dave, I will have a try to see if its working. – Ryan Oct 25 '22 at 11:20
  • This worked as expected. For windows, the value would be something like: `C:\\Users\\{user}\\AppData\\Local\\Programs\\Python\\Python39\\python.exe` – human Apr 18 '23 at 00:43