i have 1 producer 1 consumer and 3 brokers in my set up. I am using python API for producer and consumer.
all kafka parameters are set to default meaning no compression should occur at any kafka stage.
when producer use "message = 'a' * message_size"
; the network TX BW at producer is very low than i anticipate(around 50% or less than calculated BW). when i use random (os.random) message i see network BW close to what i anticipate.
in both cases; when i look into message from broker side; it is read exactly as it was sent by producer.
these observation indicates that data is being compressed even-though kafka parameters are set for NO compression.
Do you know potential reason why and where compression is happening as message is sent from producer side?
I am adding code snippets as requested
producer = KafkaProducer(bootstrap_servers=args.kafka_broker, max_request_size=1000000000,
batch_size=args.batch_size, linger_ms=args.linger_ms)
for i in range(element_count):
byte_data = 'a' * send_size
data = {'data': byte_data, 'items': send_items, 'log': 1}
time.sleep()
element_name = hostname
json_aligned_data = json.dumps(data)
producer.send(produce_topic, key=element_name, value=json_aligned_data)
