0

I'm trying to create a batch processing started by a new file in AWS S3.

So the flow is:

1 - A new file is uploaded to AWS S3 Bucket
2 - SCDF detects
3 - SCDF launch the task (Spring Batch application)
4 - Spring Batch application process the file and stores in a DB.

Something similar to this, but with S3 Bucket: https://dataflow.spring.io/docs/recipes/batch/sftp-to-jdbc/

Maybe is a misunderstood of the concept, but in SFTP Source I could set Port, Host, User and Pass, but in S3 Source I doesn't have region and credentials properties.

Where do I set the AWS properties?

Guilherme Bernardi
  • 490
  • 1
  • 6
  • 18

2 Answers2

2

There's an Amazon AWS common options section in the README (see: old-app / new-app), which includes the common AWS-specific properties one can override.

You can pass them as inline properties in the stream definition or when deploying the stream by following the deployer properties convention.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
2

I provided a detailed example of this use case here (with Minio S3). This works out of the box with the latest release of stream applications, but will require some customization if you are using previous versions.

dturanski
  • 1,723
  • 1
  • 13
  • 8
  • Hi. Thank you for you attention. I was already following your use case, but I was having trouble with my bucket sync, if you could help me or give me some tip I appreciate a lot. I created another question [question](https://stackoverflow.com/questions/65646098/problem-synchronizing-bucket-to-local-directory-with-spring-cloud-dataflow-str). Thank you again. – Guilherme Bernardi Jan 09 '21 at 18:25