Kafka consumer does not read transactional messages

Question

My Kafka consumer does not read commited transactions from Kafka topic, when 'isolation.level': 'read_committed' is set. Customer, writen in Python, is using confluent-kafka module. Producer is Spring boot application. Confluent Kafka cluster has 6.0.0 version.

Please could someone suggest what is problem ? Am I hitting some bug in 6.0.0 version of Confluent distro ? Thank you.

Consumer code is :

conf = {'bootstrap.servers': "xxx:9092",
        'group.id': "inc.59803.test2",
        'auto.offset.reset': 'earliest',
        'isolation.level': 'read_committed',
        }

consumer = Consumer(conf)

consumer.subscribe(['owfe.history'])

print(consumer.get_watermark_offsets(TopicPartition('owfe.history',0)))

for i in range(200):
  msg = consumer.poll(timeout=5.0)
  if msg is not None:
    print(msg.topic(), msg.offset(), msg.key())
consumer.close()

When it is started, function get_watermark_offsets returns offsets (11551495, 11551495), and then reads nothing.

When I look into topic log using bin/kafka-dump-log it starts with lines :

Dumping data/broker/owfe.history-0/00000000000011551495.log
Starting offset: 11551495
baseOffset: 11551495 lastOffset: 11551495 count: 1 baseSequence: 4865 lastSequence: 4865 producerId: 112687 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: true isControl: false position: 0 CreateTime: 1667489390619 size: 1287 magic: 2 compresscodec: NONE crc: 2456153955 isvalid: true
| offset: 11551495 CreateTime: 1667489390619 keysize: 36 valuesize: 1010 sequence: 4865 headerKeys: [X-dynaTrace,__TypeId__] key: a91f7efc-18ba-481b-a142-2c90432379e4 payload: {...}
baseOffset: 11551496 lastOffset: 11551496 count: 1 baseSequence: 4866 lastSequence: 4866 producerId: 112687 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: true isControl: false position: 1287 CreateTime: 1667489390633 size: 1576 magic: 2 compresscodec: NONE crc: 1491220434 isvalid: true
| offset: 11551496 CreateTime: 1667489390633 keysize: 36 valuesize: 1298 sequence: 4866 headerKeys: [X-dynaTrace,__TypeId__] key: 45de15d9-31eb-4d94-ab6d-3021256e2f31 payload: {...}
baseOffset: 11551497 lastOffset: 11551497 count: 1 baseSequence: 4867 lastSequence: 4867 producerId: 112687 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: true isControl: false position: 2863 CreateTime: 1667489390635 size: 1590 magic: 2 compresscodec: NONE crc: 882194437 isvalid: true
| offset: 11551497 CreateTime: 1667489390635 keysize: 36 valuesize: 1312 sequence: 4867 headerKeys: [X-dynaTrace,__TypeId__] key: a129d115-ce35-4ef4-9338-921fc2d5be91 payload: {...}
baseOffset: 11551498 lastOffset: 11551498 count: 1 baseSequence: 4868 lastSequence: 4868 producerId: 112687 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: true isControl: false position: 4453 CreateTime: 1667489390637 size: 1550 magic: 2 compresscodec: NONE crc: 1381749723 isvalid: true
| offset: 11551498 CreateTime: 1667489390637 keysize: 36 valuesize: 1272 sequence: 4868 headerKeys: [X-dynaTrace,__TypeId__] key: 0161ec76-f82b-44f8-9d21-b5f076ca8b02 payload: {...}
baseOffset: 11551499 lastOffset: 11551502 count: 4 baseSequence: 4869 lastSequence: 4872 producerId: 112687 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: true isControl: false position: 6003 CreateTime: 1667489390641 size: 5922 magic: 2 compresscodec: NONE crc: 3323098775 isvalid: true
| offset: 11551499 CreateTime: 1667489390637 keysize: 36 valuesize: 1255 sequence: 4869 headerKeys: [X-dynaTrace,__TypeId__] key: 6e6b5735-0f24-4c8e-9da6-ddb6cd424d50 payload: {...}
| offset: 11551500 CreateTime: 1667489390639 keysize: 36 valuesize: 1307 sequence: 4870 headerKeys: [X-dynaTrace,__TypeId__] key: ac22dc87-2897-4e0e-891c-35911edeba79 payload: {...}
| offset: 11551501 CreateTime: 1667489390641 keysize: 36 valuesize: 1175 sequence: 4871 headerKeys: [X-dynaTrace,__TypeId__] key: 1d809e84-d41a-4195-aa18-9402a3b57b81 payload: {...}
| offset: 11551502 CreateTime: 1667489390641 keysize: 36 valuesize: 1255 sequence: 4872 headerKeys: [X-dynaTrace,__TypeId__] key: 1d496cee-0bb0-4f96-a059-c9811a8a1118 payload: {...}
baseOffset: 11551503 lastOffset: 11551503 count: 1 baseSequence: -1 lastSequence: -1 producerId: 112687 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: true isControl: true position: 11925 CreateTime: 1667489390707 size: 78 magic: 2 compresscodec: NONE crc: 651982539 isvalid: true
| offset: 11551503 CreateTime: 1667489390707 keysize: 4 valuesize: 6 sequence: -1 headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 0

If I understand correctly, at least messages with offsets 11551495 - 11551502 should be read, because they are commited correctly.

When I use same consumer on another topic it works fine.

Can you use [kafkacat](https://github.com/edenhill/kcat) to consume from the topic to validate that the messages are there? By default the isolation level of the tool is read_commit — ddss, Nov 12 '22 at 10:10
No, I can not consume messages using newest version of kafkacat : $ kcat -b xxx:9092 -t owfe.history -o beginning -c 2 -f "%p %o\n" % Auto-selecting Consumer mode (use -P or -C to override) % Reached end of topic owfe.history [0] at offset 11551495 But when I use older version of kafkacat (from 2019), where transactions are not supported, I am able consume messages : kafkacat.exe -b xxx:9092 -t owfe.history -o beginning -c 2 -f "%p %o\n" % Auto-selecting Consumer mode (use -P or -C to override) 0 11551495 0 11551496 — Robert Szomolanyi, Nov 12 '22 at 16:09
You can execute the kafkacat using the isolation level of read uncommitted. If you are able to see the messages you may have something wrong with your producer, the transaction it is not committed or it is aborted Example: ``` echo "isolation.level=read_uncommitted" > read_uncommitted.properties kafkacat -F read_uncommitted.properties ... ``` — ddss, Nov 14 '22 at 17:16
Yes, I can read messages if "isolation.level=read_uncommited" with any consumer. There is the dump from the log file in my original post, where it is clear, that producer 112687 (epoch 0) wrote messages with offsets 11551495 - 11551502, and also committed whole transaction ( in last line with offset 11551503). Therefore I think that producer commited messages correctly and they should be readable with "isolation.level=read_commited" ... — Robert Szomolanyi, Nov 15 '22 at 08:30

Kafka consumer does not read transactional messages

0 Answers0