0

I am trying to use lenses s3 source connector in aws msk(kafka). Download kafka-connect-aws-s3-kafka-3-1-4.0.0.zip as a plug-in, save it in S3 and register it. And the above plug-in was specified and the connector configuration was written as follows.

<Connector Configuration>
connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
key.converter.schemas.enable=false  
connect.s3.kcql=insert into my_topic select * from  my_bucket:dev/domain_name/year=2022/month=11/ STOREAS 'JSON'
tasks.max=2
connect.s3.aws.auth.mode=Default
value.converter.schemas.enable=false
connect.s3.aws.region= ap-northeast-2
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

The connector is normally created and data is read from S3 to the specified topic, but there are two problems here.

  1. As described in "connect.s3.kcql", data is imported based on /year=2022/month=11/, but other partitioned month and date data are also imported. It seems that the paths of "/year=" and "/month= " specified under /dev/domain_name(=PREFIX_NAME) are not recognized and all are imported. I wonder if there is a way.

(refer to my full s3 path: my_bucket/dev/domain_name/year=2022/month=11/hour=1/*.json )

  1. The json file exists more in the specified s3 path, but it is no longer imported into the topic. No errors occur and this is normal. When I look at the connector log, I keep getting "flushing 0 outstanding messages for offset commit" message.
jojo
  • 3
  • 3
  • The documentation says to replace the dollar sign values, not place them literally in the config. Also, S3 doesn't have "directories". It's all called the prefix, so try `your-bucket:/dev/domain_name/year=2022/month=11/` – OneCricketeer Nov 04 '22 at 13:45
  • @OneCricketeer Hi. I modified it because I thought the question was confusing. Of course I already made it up as you told me to, but it did not work – jojo Nov 04 '22 at 16:58
  • I see. Have you found any related issues at https://github.com/lensesio/stream-reactor/issues – OneCricketeer Nov 04 '22 at 23:45
  • @OneCricketeer thanks. thanks. I left an inquiry on the issue – jojo Nov 07 '22 at 13:07

0 Answers0