2

Maybe it is duplicated of this question: Write to a specific folder in S3 bucket using AWS Kinesis Firehose. But maybe in the last year appeared new options.

I would like to know which is the best way to send incoming data streams to a specific path or prefix in an s3 bucket.

For instance, let's say that I have the following JSON object:

{"clientId":2,"event":"click","url":"http://www.url.com/foo"}

I would like to store it in a path like this: 

s3Bucket/{event}/{clientId} => s3Bucket/click/2

I'm considering these options:

  • Kinesis firehose: Maybe is it possible in some way to set a custom prefix like the example?
  • lambda function: Trigger a lambda function with the incoming streams and set the custom prefix reading the JSON object.
  • Other options ...

There's a better way to do that? Which of the above options is the best choice?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
JosepB
  • 2,205
  • 4
  • 20
  • 40
  • DO you want to store each JSON object in a separate Amazon S3 object, or do you want to combine them together into a smaller number of objects? I think the answers on the previous question are still valid. – John Rotenstein May 23 '19 at 08:16
  • I want to store the JSON objects in a separate Amazon S3 object. For instance, separate each object by `clientId`. – JosepB May 23 '19 at 08:29
  • If you want to store each message in a separate Amazon S3 object, then why not simply write the data direct to S3? Whatever source you have will need to know how to send the data to Kinesis/Lambda, so instead just get it to store directly in S3 instead, using your desired prefix. – John Rotenstein May 23 '19 at 08:33
  • @JohnRotenstein thanks. I'll take a look – JosepB May 23 '19 at 08:46
  • @JohnRotenstein what do you exotically mean? if i want to save in s3 with specific format i need to use kinesis stream and lambda to s3, i cant send from cloudwatch directly. or am i missing something? – inza Apr 06 '20 at 09:39

0 Answers0