Where are the Spark logs on EMR?

Question

I'm not able to locate error logs or message's from println calls in Scala while running jobs on Spark in EMR.

Where can I access these?

I'm submitting the Spark job, written in Scala to EMR using script-runner.jar with arguments --deploy-mode set to cluster and --master set to yarn. It runs the job fine.

However I do not see my println statements in the Amazon EMR UI where it lists "stderr, stdoutetc. Furthermore if my job errors I don't see why it had an error. All I see is this in thestderr`:

15/05/27 20:24:44 INFO yarn.Client: Application report from ResourceManager: 
 application identifier: application_1432754139536_0002
 appId: 2
 clientToAMToken: null
 appDiagnostics: 
 appMasterHost: ip-10-185-87-217.ec2.internal
 appQueue: default
 appMasterRpcPort: 0
 appStartTime: 1432758272973
 yarnAppState: FINISHED
 distributedFinalState: FAILED
 appTrackingUrl: http://10.150.67.62:9046/proxy/application_1432754139536_0002/A
 appUser: hadoop

`

score 15 · Accepted Answer · answered May 29 '15 at 05:50

15

With the deploy mode of cluster on yarn the Spark driver and hence the user code executed will be within the Application Master container. It sounds like you had EMR debugging enabled on the cluster so logs should have also pushed to S3. In the S3 location look at task-attempts/<applicationid>/<firstcontainer>/*.

answered May 29 '15 at 05:50

ChristopherB

2,038
14
18

Yes, this is correct. Thank you. For other users knowledge, you can see this "Log URI" in the Amazon EMR Web UI for your cluster info/details. – Sean Bollin Jun 18 '15 at 17:26
In my case, I cannot found "task-attempts" when clicking "Log URI" in the EMR Web UI, was it renamed or moved? – Luis Leal Jul 01 '22 at 18:10
The path to the location for the logs has been changed. Please refer to answer by @Vsh - https://stackoverflow.com/a/62963889/6780081 – 7bStan Jan 10 '23 at 08:31

score 6 · Answer 2 · answered May 28 '15 at 03:54

6

If you SSH into the master node of your cluster then you should be able to find the stdout, stderr, syslog and controller logs under:

/mnt/var/log/hadoop/steps/<stepname>

answered May 28 '15 at 03:54

Anders Hammar

93
1

4

Those are the Step logs, which do not contain the Spark application logs (such as OP's println statements). – Azuaron May 30 '18 at 16:03
1

As stated in the above comment, this answer is incorrect. – kyle Sep 07 '18 at 16:56

score 6 · Answer 3 · answered Jul 18 '20 at 02:05

6

I also spent a lot of time figuring this out. Found logs in the following location: EMR UI Console -> Summary -> Log URI -> Containers -> application_xxx_xxx -> container_yyy_yy_yy -> stdout.gz.

answered Jul 18 '20 at 02:05

VSh

438
4
13

1

Best answer here! – Stempler Aug 30 '22 at 21:07

randal25 · Answer 4 · 2019-01-29T12:24:23.007

1

The event logs, the ones required for the spark-history-server can be found at :

hdfs:///var/log/spark/apps

edited Jan 29 '19 at 12:24

answered Jan 29 '19 at 12:16

randal25

1,290
13
10

score 0 · Answer 5 · answered May 28 '15 at 01:20

0

If you submit your job with emr-bootstrap you can specify the log directory as an s3 bucket with --log-uri

answered May 28 '15 at 01:20

Holden

7,392
1
27
33

thanks - I think this might be set when I create the cluster (not when submitting a job)? i'll try next time I create the cluster – Sean Bollin May 28 '15 at 03:52
The S3 logs are plain text, however in Spark History Server I can download JSON logs which are prefect for indexing in Elasticsearch (Download button under Event Log column). Where are these stored? – lfk May 21 '18 at 04:44

Where are the Spark logs on EMR?

5 Answers5