I have a AWS Data Pipeline that runs a query on MySQL RDS DB and loads the result in Redshift, as described in AWS docs. Now Data Pipeline service is going away, so I need to migrate it out of this service. What would be an optimal implementation of this use case?
I have migrated the pipeline to AWS Step Functions, where the data transfer between MySQL, s3 and Redshift is done by AWS Lambdas. However, the performance of the MySQL queries seems to be worse than in AWS Data Pipelines (it consumes more Read IOPS) and for bigger queries the Lambda executions are timed out after 15 minutes.
I also checked AWS Glue but I can't seem to find a way to only export the output of a SQL query (I may use joins in the query, so table fullcopy or upsert won't help here).
