I have AWS GLUE Python application by default logs are available in Cloudwatch Log Group /aws-glue/jobs/output and /aws-glue/jobs/error for stdout and stderr respectively. I have explored AWS documentations and several websites for the process to redirect logs to a custom Cloudwatch Log Group like /dev/<app_name>/, but no luck. If any one has idea over this, please share the process.
Asked
Active
Viewed 5,341 times
3
vahdet
- 6,357
- 9
- 51
- 106
Tarun Mootala
- 219
- 3
- 8
-
Do you use a Python script or so, to perform the job? And also please tell more about what you've tried so that people would not come up with what you've already tried. – vahdet Mar 07 '19 at 14:04
-
Yes using a Python script. Unfortunately until now could not find any option to achieve this. I'm worrying even if this is possible. – Tarun Mootala Mar 07 '19 at 14:46
-
Couldn't find any direct option. Workaround used cloudwatch event (glue job state change) with Lambda target. Segregated the logs based on event and job name, saved to s3. – Tarun Mootala May 30 '19 at 11:16
1 Answers
2
for spark type glue jobs you can enable continuous logging but for a python shell type glue job continuous logging is not available , therefore you will not be able to create a custom log group for that, but there is a workaround for this. refer below for more info
How to use a CloudWatch custom log group with Python Shell Glue job?
Sithija Piyuman Thewa Hettige
- 1,768
- 20
- 26