Can I map multiple buckets with multiple topics in a single Kafka-Connector S3 sink connector?

Question

Google didn't help me, so I want to ask you. I have a lot of kafka topics, and I want to store the messages of a particular topic in a particular S3 bucket.

Do I need to create an S3 sink connector for each bucket or can I configure all the stuff in a single S3 json file (map a bucket with a topic)?

Thanks for your help! :)

So one bucket per topic, or multiple topics into one bucket? — Robin Moffatt, Mar 19 '20 at 14:18

score 2 · Answer 1 · answered Mar 20 '20 at 12:00

2

The S3 bucket name is specified per-connector, so you will need to create one connector per bucket.

Note that you can specify multiple topics per connector (topics / topics.regex), so if you wanted multiple topics going to one bucket (but different paths), you could do this with a single connector. But for multiple buckets, you'll need multiple connectors.

answered Mar 20 '20 at 12:00

Robin Moffatt

30,382
3
65
92

Hey @robin , for multiple topics in single s3 bucket, do we just need to add topics.regex : (SAP|EQ).* or we need any other config as well ? – KanikaM May 12 '21 at 20:09
@KanikaM please post a new question. Thanks. – Robin Moffatt May 12 '21 at 21:40

Can I map multiple buckets with multiple topics in a single Kafka-Connector S3 sink connector?

1 Answers1