0

I am trying to write a nodeJS lambda function to query data from our database cluster and upload this to s3, we require this for further analysis. But my doubt is, if the data to be queried from the db is large (9GB), how does the lambda function handle this as the memory limit is 3008 MB ?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Sreerag
  • 1,381
  • 3
  • 11
  • 16
  • Is the database also hosted on AWS, or is it on-premises? – John Rotenstein Mar 06 '19 at 08:34
  • Yes, my database ( couchbase-cluster) is also hosted on AWS. Currently I am querying out the data on to an existing server and uploading to s3 using aws-cli. This is working out to be fairly easy with a few lines of shell script. But glad to know the options for a better and permanent solution. Will look into Amazon Athena as you suggested in your answer. Thanks. – Sreerag Mar 12 '19 at 11:38

2 Answers2

1

There is also a disk storage limit of 500MB.

Therefore, you would need to stream the result to Amazon S3 as it is coming in from the database.

You might also run into a time limit of 15 minutes for a Lambda function, depending upon how fast the database can query and transfer that quantity of information.

You might consider an alternative strategy, such as having the Lambda function call Amazon Athena to query the database. The results of an Athena query are automatically saved to Amazon S3, which would avoid the need to transfer the data.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
1

lambda have some limitations it term of run time and space . it's better to use crawler or job in amazon glue. it's the easy way of doing this. for that go to `

amazon glue>>job>>create job

and fill basic requirements like source and destination. and run job. there is no constrains for size and time limitation.

`

Primit
  • 825
  • 7
  • 13