-1

Ref - https://github.com/aws-samples/aws-glue-samples/tree/master/utilities/Spark_UI/ https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html

I am trying to access glue spark ui locally on my windows 10 - (docker desktop) using docker container.

I have followed following steps but I get error while starting container

$ docker build -t glue/sparkui:latest . 


$ LOG_DIR="s3a://path_to_eventlog/"
$ AWS_ACCESS_KEY_ID="ASIAxxxxxxxxxxxx"
$ AWS_SECRET_ACCESS_KEY="yyyyyyyyyyyyyyy"
$ SESSION_TOKEN="zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz"
$ docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID -Dspark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -Dspark.hadoop.fs.s3a.session.token=$SESSION_TOKEN -Dspark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"

D:\glue_spark_gui_docker_file>docker logs 05c595fc8c65
Error: Could not find or load main class $SPARK_HISTORY_OPTS
Sarde
  • 658
  • 1
  • 8
  • 19
  • can you run with -it instead of -itd as mentioned in https://aws.amazon.com/premiumsupport/knowledge-center/glue-spark-ui/ and look for the logs? – Prabhakar Reddy Sep 08 '20 at 07:40
  • D:\glue_spark_gui_docker_file>docker run -it -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.access.key=$AWS_ACCESS_KEY_ID -Dspark.hadoop.fs.s3a.secret.key=$AWS_SECRET_ACCESS_KEY -Dspark.hadoop.fs.s3a.session.token=$SESSION_TOKEN -Dspark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer" Error: Could not find or load main class $SPARK_HISTORY_OPTS – Sarde Sep 08 '20 at 12:42
  • 1
    same error with -it – Sarde Sep 08 '20 at 12:42
  • Any resulation to this @TusharS? – Andrey Bulezyuk Nov 04 '20 at 07:48
  • Hi @AndreyBulezyuk I followed this repo. https://github.com/moj-analytical-services/docker_spark_history_ui/blob/master/README.md I am able to see spark ui locally. let me know if you face any issue I will help you. – Sarde Nov 04 '20 at 10:38
  • I made it work locally and deployed to ECS. Had to replace "-e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS" with "-e SPARK_HISTORY_OPTS="-Dspark.hi..." – Andrey Bulezyuk Nov 05 '20 at 11:02
  • ok. are you using mac or windows ? – Sarde Nov 05 '20 at 14:43

1 Answers1

1

same error when trying docker on wsl with bash on windows so I solved this using windows cmd

1. Build image

C:\Users\Avinash\Spark_UI\glue-1_0-2_0> docker build -t glue/sparkui:latest .

2. set variable

set LOG_DIR="s3a://path_to_eventlog/"
set AWS_ACCESS_KEY_ID="XXXXYYYY"
set AWS_SECRET_ACCESS_KEY="SDFGHJHRFVBNMNBVC"

3. Run container remove $SPARK_HISTORY_OPTS and add %variable% instead of $variable

docker run -itd -e SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=%LOG_DIR% -Dspark.hadoop.fs.s3a.access.key=%AWS_ACCESS_KEY_ID% -Dspark.hadoop.fs.s3a.secret.key=%AWS_SECRET_ACCESS_KEY%" -p 18080:18080 --name sparkui glue/spark:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"
Avinash
  • 359
  • 3
  • 5