I have created a Glue job using Glue studio Visual Editor. This job is getting data from a JDBC data source (Clickhouse Database) using a query. Doing some business transformations and then storing the data back to the database table using the same JDBC connection. In the source I want to parametrized the sql query so that I can change the dates with each run. I have defined the job parameters in the job details. Can any one help me how to access these params. As per the business requirement I have to develop the job using visual editor for easy maintenance purpose later on.
Asked
Active
Viewed 237 times
1 Answers
0
Unfortunately this is not possible by just using the UI currently.
If you can edit the code, you can use the getResolvedOptions function.
import sys
from awsglue.utils import getResolvedOptions
args = getResolvedOptions(sys.argv, ["enddate"])
print(args["enddate"])
Robert Kossendey
- 6,733
- 2
- 12
- 42
-
Like I have mentioned in my question I have developed the job using Visual editor. If I had chosen script mode, I would have used this function. Is it possible to access params if the job has been developed in visual mode. – Prateek Rastogi Oct 17 '22 at 11:41
-
The only option I found to write scripting lines is by using custom transform. My understanding is we can't use that functionality for this purpose – Prateek Rastogi Oct 17 '22 at 11:45
-
This is unfortunately not possible currently. Updated my answer accordingly. – Robert Kossendey Oct 17 '22 at 11:45
-
Thanks for confirmation – Prateek Rastogi Oct 17 '22 at 11:51