0

I am using an Apple M1 Pro Mac & trying to use a Docker container to developer AWS Glue Jobs locally and not use the AWS Console. I have been working through this blog post by AWS and I have pulled amazon/aws-glue-libs:glue_libs_3.0.0_image_01 from DockerHub and run the command docker run -itd -p 8888:8888 -p 4040:4040 -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/jupyter/jupyter_start.sh. The Docker container runs but gives the warning WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested, has no output logs and http://localhost:8888 does not work. If I run the command docker exec -it glue_jupyter bash it will hang for around 20 minutes and eventually give the following error and exit with the code 127:

Traceback (most recent call last):

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 87, in <module>

    automated_main(keystore_path=args.keystore, keystore_pw=args.keystorepw)

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 78, in automated_main

    generate_self_signed_ssl_cert(get_default_keystore_path(), keystore_pw, DEFAULT_KEY_ALIAS)

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 47, in get_default_keystore_path

    instance_dns = get_instance_dns()

  File "/home/glue_user/.certs/ssl_tls_setup.py", line 15, in get_instance_dns

    private_dns = instance_metadata['local-hostname']

  File "/usr/local/lib/python3.7/site-packages/boto/utils.py", line 275, in __getitem__

    return super(LazyLoadMetadata, self).__getitem__(key)

KeyError: 'local-hostname'

Error opening input file -nokeys

-nokeys: No such file or directory

Error opening input file -nokeys

-nokeys: No such file or directory

Error opening input file -nocerts

-nocerts: No such file or directory

starting org.apache.spark.deploy.history.HistoryServer, logging to /home/glue_user/spark/logs/spark-glue_user-org.apache.spark.deploy.history.HistoryServer-1-e08471474826.out

/usr/bin/bash: /home/jupyter/jupyter_start.sh: No such file or directory
samuel
  • 23
  • 5
  • You have to pass `--platform linux/amd64` to `docker run` to be able to run a amd64 container from your m1 mac, also check out this answer https://stackoverflow.com/a/70729178/1995266 to disable SSL and fix paths to `jupyter_start.sh` – Romibuzi Feb 18 '22 at 14:38

1 Answers1

1

Try this command

docker run -itd -p 8888:8888 -p 4040:4040 -e DISABLE_SSL="true" -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh

You need to disable SSL while running docker

And to run the Jupyter notebook use this address in your browser.

http://127.0.0.1:8888/lab/tree

Matthew Bakaitis
  • 11,600
  • 7
  • 43
  • 53
Anurag Singhal
  • 11
  • 3
  • getting an error docker run -itd -p 8888:8888 -p 4040:4040 -e DISABLE_SSL="true" -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. See 'docker run --help'. – NK7983 Apr 26 '22 at 23:51