0

I'm trying to create a stream which needs to listen to a folder in S3 bucket.

I could create a stream listening to the root of the bucket using following properties and could trigger the process.

app.s3-in.s3.supplier.remote-dir=mybucket
app.s3-in.cloud.aws.credentials.accessKey=XXXXXXXXXX
app.s3-in.cloud.aws.credentials.secretKey=YYYYYYYYYYYYYYYYY
app.s3-in.cloud.aws.region.static=ZZZZZZZZ
app.s3-in.cloud.aws.stack.auto=false

But unable to listen to a folder in the bucket. I tried with following options:

app.s3-in.s3.supplier.remote-dir=mybucket/myfolder
app.s3-in.s3.supplier.remote-dir=mybucket/myfolder/
app.s3-in.s3.supplier.remote-dir=/mybucket/myfolder
app.s3-in.s3.supplier.remote-dir=/mybucket/myfolder/

I tried setting app.s3-in.path-style-access=true as well.

Venu Gopal
  • 15
  • 1
  • 7

1 Answers1

0

Try

app.s3-in.s3.supplier.remote-dir=mybucket
app.s3-in.s3.supplier.filename-pattern=myfolder/**

(or some variation of the file pattern).

onobc
  • 466
  • 1
  • 5
  • I tried with filename-pattern as well but result is not as expected – Venu Gopal Feb 16 '23 at 16:39
  • what was the result @VenuGopal? Same as w/ only specifying the `remote-dir`? Be sure that your pattern will match the value from s3 `file.getKey()`. Can you list one of the full file key here? Obfuscate any private info w/ `***`. – onobc Feb 18 '23 at 01:32
  • Hi @onobc, I tried with a simple stream to transfer files from one location to another location in S3. I have following files in my bucket. test1_input.xlsx,test2_input.xlsx,myinput/test3_input.xlsx,myinput/test4_input.xlsx and myoutput/ when I tried with app.s3-in.s3.supplier.filename-pattern=*.xlsx I see test1, test2 files moved to myoutput but when I tried with app.s3-in.s3.supplier.filename-pattern=myinput/*.xlsx ( and also tried with filename-pattern=myinput/**) I see test1, test2 files in myout folder along with a 0 Bytes file named myinput, but not file3 and file4 as expected. – Venu Gopal Feb 19 '23 at 14:19
  • When I tried dropping a file named test5_input.xlsx in root after deploying these 2 streams, test5 moved to myout folder in both the cases. But when I tried dropping the same file under myinput, the file could not be moved to myout folder which is expected with second stream. – Venu Gopal Feb 19 '23 at 14:28
  • I see following error message in log of S3 input component: org.springframework.messaging.MessagingException: Problem occurred while synchronizing 'mybucket' to local directory; nested exception is org.springframework.messaging.MessagingException: Failure occurred while copying 'mybucket/myinput/test3_input.xlsx' from the remote to the local directory; nested exception is java.io.FileNotFoundException: /var/folders/jt/wclvj13x6fv564hdjzvvmx9w0000gn/T/s3-supplier/myinput/test3_input.xlsx.tmp (Not a directory) – Venu Gopal Feb 19 '23 at 14:32
  • @VenuGopal what are the settings for the S3 sink (`bucket`, `key-expression`)? I would focus only on the S3 source first and then once thats working get the S3 sink working. It would simplify things if you instead use `S3 source + Log sink` for the pipeline. – onobc Feb 20 '23 at 19:35
  • Following are S3 sink properties app.s3-out-sink.cloud.aws.credentials.accessKey=XXXXXXX app.s3-out-sink.cloud.aws.credentials.secretKey=XXXX app.s3-out-sink.cloud.aws.region.static=XXXXX app.s3-out-sink.cloud.aws.stack.auto=false app.s3-out-sink.s3.consumer.bucket=mybucket app.s3-out-sink.s3.consumer.key-expression='myoutput/‘+headers.file_name I have tried with S3 source + log sink as well. I'll create a new one and post my findings here – Venu Gopal Feb 21 '23 at 04:09
  • I'm could see an object reference printed in log sink for each file – Venu Gopal Feb 21 '23 at 13:28
  • Hi @venu it turns out to be a bug in the S3 source app. Please follow https://github.com/spring-cloud/stream-applications/issues/431#issuecomment-1438963366 for fix and workaround. Once that ticket completes I will update this SO thread w/ a summary answer. – onobc Feb 22 '23 at 14:43
  • yes @onobc, I'm following this, it was created by me there. thanks for your time and guidance – Venu Gopal Feb 23 '23 at 16:55