0

I need to build glue spark application to transform raw events and then execute REST API to push transformed data. I am using glue Dynamic Frame to transform raw events but not able to execute REST API call. Is there a way to execute REST API or boto3 API on dynamic frame.

PB22
  • 31
  • 4

1 Answers1

0

The Glue Dynamic Frame is a spark object that only lives inside the Glue job while it's running. It's not possible to expose that data during a job run. If you want to expose that data, you can write it out to S3, DynamoDB, RDS Database or Redshift. Then you can use API Gateway and Lambda to expose the data.

Coin Graham
  • 1,343
  • 3
  • 11
  • I need to parse json events and call AWS personalize API to push some of the json columns. Is it possible to run MAP function to execute API method with parsed events. – PB22 Apr 18 '22 at 16:16
  • Yes. If you're running PySpark glue jobs you can loop over the json and use the urllib package to make API requests. – Coin Graham Apr 19 '22 at 14:39
  • I was able to get this done with spark UDF and RDD map function – PB22 May 05 '22 at 21:18