Messages not getting flushed to kafka

Question

I have a topic with 5 partitions (A), and 5 consumers pulling messages from one partition each. Does some processing (takes about 30 seconds) and pushes to two topics with 1 partition each (Topic-B1, TopicB2). Topic-B1 is a log-compacted topic

kafka-python == 2.0.1

python3.8

producer = KafkaProducer(bootstrap_servers = cluster_ip, acks = -1, retries = 3)
#acks = -1 is similar to acks = all

I have added callback, errback while sending

def on_send_success(record_metadata):
    print(record_metadata.topic)
    print(record_metadata.partition)
    print(record_metadata.offset)

def on_send_error(excp):
    log.error('I am an errback', exc_info=excp)
    # handle exception

producer.send('my-topic', b'raw_bytes').add_callback(on_send_success).add_errback(on_send_error)
producer.flush()

Consumer Snippet-

while True:
    raw_messages = consumer.poll(timeout_ms=20000, max_records=2)
    for topic_partition, messages in raw_messages.items():
        for msg in messages:
            try:
                #process msg .....
                data1, data2 = some_func(msg.value)
                producer.send(Topic-B1, key = msg.key, value = data1).add_callback(on_send_success).add_errback(on_send_error)
                producer.flush()
                producer.send(Topic-B2, key = msg.key, value = data2).add_callback(on_send_success).add_errback(on_send_error)
                producer.flush()
                consumer.commit()
            except Exception as e:
                log.error('I am an errback', exc_info=str(e))

But a very few messages didn't get sent to Topic-B1, Topic-B2. I don't see any logs or errors in the consumers. How do i make sure that messages are sent for sure or how to raise the exception in case of failure.