1

Is there any way we can tag the objects written in S3 buckets through the Kafka Connect S3 sink connector. I am reading messages from Kafka and writing the avro files in S3 bucket using S3 sink connector. When the files are written in S3 bucket I need to tag the files.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
user2531535
  • 11
  • 1
  • From what I've seen in the S3 Connect code, there's no tagging functionality... You'd have to use a Lambda to achieve something like this – OneCricketeer Jul 25 '19 at 05:00

1 Answers1

0

there is an API inside source code on the GitHub called addTags(), but it's now private and is not exposed to the connector client except this small config feature called S3_OBJECT_TAGGING_CONFIG which allows you to add start/end offsets as well as record count to s3 object.

 configDef.define(
      S3_OBJECT_TAGGING_CONFIG,
      Type.BOOLEAN,
      S3_OBJECT_TAGGING_DEFAULT,
      Importance.LOW,
      "Tag S3 objects with start and end offsets, as well as record count.",
      group,
      ++orderInGroup,
      Width.LONG,
      "S3 Object Tagging"
  );

If you want to add other/custom tags then answer is NO you cannot do it right now.

Useful feature would be to take the tags from the predefined part of an input document in Kafka but this is not available right now.

Eddie Jamsession
  • 1,006
  • 6
  • 24