1

I have a Window 10 VDI and have set-up AWS Glue Interactive session using this link. Python version set-up for this is Python 3.9.7

When I start the notebook from command prompt, I see the various kernels in the browser

enter image description here

I have VSCode ( updated to v1.77 ) and have installed the Python and Jupyter extensions as well and trying to set-up VSCode to use the Glue Interactive sessions using this .

In VSCode, I do not see Glue PySpark as kernel Option, though see Glue Spark. I have also added python path the kernel.json as described here

But I still do not see Glue PySpark as an option in VSCode. Am I missing any set-up ?

enter image description here

starball
  • 20,030
  • 7
  • 43
  • 238
adbdkb
  • 1,897
  • 6
  • 37
  • 66
  • Possibly useful: https://code.visualstudio.com/docs/datascience/jupyter-kernel-management#_jupyter-kernels – starball Apr 10 '23 at 17:00
  • Thanks. I now somehow got that to work. My next attempt is to use jupyter notebook server in VSCode. I will open another question ( or should I update the same question ?) – adbdkb Apr 10 '23 at 21:15
  • write your new question in a new question post, and write up the solution you found in an answer post here (see also [answer]) :) – starball Apr 10 '23 at 21:29
  • Thanks. The reason I did not write that as an answer, because I just uninstalled and reinstalled the Python and Jupyter extensions. Wasn't sure, if that was the actual reason, or other trial and errors I was doing fixed it. Wasn't anything specific that made it work. The github link that had in the question is definitely required though. Let me know, if I should still post about uninstall and reinstall part. Also, this is the new question. https://stackoverflow.com/questions/75981078/how-to-set-up-jupyter-server-in-vscode - Thanks – adbdkb Apr 10 '23 at 22:10
  • what github link are you referring to? – starball Apr 10 '23 at 22:15
  • I just realised I had put link to the SO question that had github link. This is the github link - https://github.com/microsoft/vscode-jupyter/issues/8286 - in this SO question - https://stackoverflow.com/questions/73994337/cannot-find-pyspark-kernel-aws-glue-interactive-sessions-with-vscode - which was on the same subject – adbdkb Apr 10 '23 at 23:26
  • This doesn't answer your questions, but I went the route of using a dockerized glue container over the interactive sessions. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html – db_brad Apr 12 '23 at 18:52

2 Answers2

0

I was able to resolve my issue by adding the below information found at the link Pyspark Kernel not showing

{
  "argv": [
    "D:\\venvs3\\.venv\\Scripts\\python.exe", <-- Full path to python from venv 
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_pyspark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "PySpark",
  "language": "python"
}

I also uninstalled and reinstalled the set-up a few times as trial and error, but don't know if that made any difference. The only reason I am mentioning it here is - during that process I changed different versions for packages and also installing some of them as --user as I was getting permissions issues otherwise.

That may or may not have anything to do with why it started working, but the above update to kernel.json is definitely needed.

adbdkb
  • 1,897
  • 6
  • 37
  • 66
0

Along with the above mentioned changed, I also had to change the language to "scala" which does not make sense but helped to get Glue PySpark in VSCode. "language": "scala"

IGT
  • 1