Prevent camus from increasing the offset value

Question

Is there any way of preventing camus increase the offset value, and make it always read the previous value?

And also how to provide a custom name to the file that gets created instead of the general format i.e. topic.leaderId.partitionId.partitioner.encodePartition(context, key)?

Answer 1

File names of mapper's output get generated in generateFileName method of com.linkedin.camus.etl.Patitioner class.

You can implement a custom Partitioner and provide a logic in as

@Override public String generateFileName(JobContext context, String topic, String brokerId, int partitionId, int count, long offset, String encodedPartition) { return "MyCustomFileName-"+topic+partitionId; }