AWS Glue JOB: Command failed with error code 1

Question

We have python script for our glue job and the triggered runs for every one hour to convert the JSON S3 to parquet files and we are getting following issue..the following logs are taken from cloudwatch for the jobId :

CoarseGrainedExecutorBackend: Driver commanded a shutdown
18/06/25 08:54:03 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from ip-172-31-34-26.ec2.internal/172.31.34.26:36135 is closed
18/06/25 08:54:03 ERROR OneForOneBlockFetcher: Failed while starting block fetches
java.io.IOException: Connection from ip-172-31-34-26.ec2.internal/172.31.34.26:36135 closed
        at org.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146)
        at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
        at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
        at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1289)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:893)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:691)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)
18/06/25 08:54:03 INFO CoarseGrainedExecutorBackend: Driver from 172.31.47.44:45951 disconnected during shutdown
18/06/25 08:54:03 INFO CoarseGrainedExecutorBackend: Driver from 172.31.47.44:45951 disconnected during shutdown
18/06/25 08:54:03 INFO RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000 ms
18/06/25 08:54:03 INFO MemoryStore: MemoryStore cleared
18/06/25 08:54:03 INFO BlockManager: BlockManager stopped
18/06/25 08:54:03 INFO ShutdownHookManager: Shutdown hook called

Are you sure that's the only interesting log lines? I know that spark can be very verbose and it's sometimes difficult to understand what was the real problem. Looking at those log lines I'd imagine that there may be the cause of why the `Driver commanded a shutdown`. — botchniaque, Jun 25 '18 at 18:40
I did see in other blog fourms, that if the size of the file is bigger, and the job reaches the network bandwidth at shuffle stage, resulting in job failure as above. Maybe you can try to adjust memory configurations, or reduce the number of executors which helped in resolving this issue. You can follow suggestions from this https://stackoverflow.com/questions/49034126/aws-glue-executor-memory-limit and let me know. Also, please refer this : https://forums.databricks.com/questions/10872/error-transportresponsehandler-still-have-1-reques-1.html — Yuva, Jun 25 '18 at 19:30
I do suspect that it is because of huge data while conversion..can we delete files from S3 folder(not bucket) using any cloud formation script?..let me know if there are any samples for the same.. — rohith, Jun 26 '18 at 02:50
Hi I have added job parameter(to increase the memory) to the JOB but it was not increasing the memory, I do see the same in cloud watch logs also we tried keep Object Expiration in S3 to 45 to reduce the number of files still the issue is present.. — rohith, Jul 05 '18 at 11:20
is running beyond physical memory limits. Current usage: 5.6 GB of 5.5 GB physical memory used; 7.6 GB of 27.5 GB virtual memory used. Killing container. — rohith, Jul 05 '18 at 11:25

score 1 · Answer 1 · answered Nov 29 '18 at 12:58

1

Open Glue> Jobs > Edit your Job> Script libraries and job parameters (optional) > Job parameters near the bottom Set the following: key: --conf value: spark.yarn.executor.memoryOverhead=1024 spark.driver.memory=10g

answered Nov 29 '18 at 12:58

ashutosh singh

185
1
5

Why? Can you please elaborate on this? – Francesco Ambrosini Aug 30 '19 at 13:40

score -1 · Answer 2 · answered Jul 24 '18 at 04:34

There is no way to fix this issue,AWS Glue has so many enhancements that are to be done. As of now we split our folder into multiple sub folders and split our glue job to two to handle this scenario,and also the memory overhead was not being considered when we give our own script option.

score -1 · Answer 3 · answered Feb 21 '19 at 08:50

-1

You need to reduce the number of files that you are storing into the S3 bucket by accumulating the data into a single big file,glue is efficient on bigger files

answered Feb 21 '19 at 08:50

rohith

163
4
16

If you think this is the answer ok. But I would say that this can be a comment instead. – dpapadopoulos Feb 21 '19 at 09:14
yes thats the only way as of today..till aws comes up with more memory support. – rohith Feb 21 '19 at 09:18

AWS Glue JOB: Command failed with error code 1

3 Answers3