I have a general glue job which will execute based on the file name that is landing in s3 bucket. So right now we are creating an event based trigger which will execute a workflow. (Event is whenever a file land in s3). Is there any way to get the file details like file name or s3 uri and then pass this info to the workflow as a parameter.
Asked
Active
Viewed 722 times
2
-
Use lambda as an intermidate step. – Marcin Jun 30 '22 at 05:57
-
ok.but how can I get the file details if my event will invoke a lambda function – Techno_Eagle Jun 30 '22 at 06:03
-
File details should be in the event payload. – Marcin Jun 30 '22 at 06:08
-
okay and then how can i use this fiile name as a parameter for my glue job? – Techno_Eagle Jun 30 '22 at 06:55
1 Answers
3
how can i use this fiile name as a parameter for my glue job?
The AWS docs explain how to pass arguments to a glue job and how to read them in the job itself. So basically your lambda would call start_job_run and pass Arguments (e.g. S3 object name) to your job. The job, when started, would read the arguments passed and perform actions based on them.
Marcin
- 215,873
- 14
- 235
- 294