KAFKA - how to re-balance the partitions to available brokers

Question

we have 3 kafka machines and 3 zookeeper's servers

kafka1 - 1001 ( broker ID )
kafka2 - 1002 ( broker ID )
kafka3 - 1003 ( broker ID )

we have issue is about to re-balance the partitions to available brokers , we can see ( down ) that some partitions under-replicated with only two brokers in isr. Instead with 3

what is the best way to re balance the kafka topic partitions in ISR

second - we can see that leader 1002 is missing , what is the solution for this?

remark - we have 23 topics ( the list down is partial )

[kafka@kafka01 bin]$ ./kafka-topics.sh -describe --zookeeper master:2181
Topic:lop_gt        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: lop_gt       Partition: 0    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 6    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 12   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 18   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 24   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 30   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 36   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 42   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 48   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
Topic:_schemas  PartitionCount:1        ReplicationFactor:3     Configs:cleanup.policy=compact
        Topic: _schemas Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
Topic:ambari_kafka_service_check        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ambari_kafka_service_check       Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002
Topic:jr_dfse    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: jr_dfse   Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 2    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 3    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: jr_dfse   Partition: 4    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 5    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 6    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 7    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 8    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 9    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:frte_nnc   PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: frte_nnc  Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 1    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: frte_nnc  Partition: 2    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: frte_nnc  Partition: 3    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 4    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: frte_nnc  Partition: 5    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:erw_plk    PartitionCount:100      ReplicationFactor:3     Configs:
        Topic: erw_plk   Partition: 0    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 6    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 12   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 18   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 24   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 30   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 36   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 42   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 48   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 50   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 51   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 52   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 53   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 54   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 55   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 56   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 57   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 58   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 59   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 60   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 61   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 62   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 63   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 64   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 65   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 66   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 67   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 68   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 69   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 70   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 71   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 72   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 73   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 74   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 75   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 76   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 77   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 78   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 79   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 80   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 81   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 82   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 83   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 84   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 85   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 86   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 87   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 88   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 89   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 90   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 91   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 92   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 93   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 94   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 95   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 96   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 97   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 98   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 99   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
Topic:loe_sd        PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: loe_sd       Partition: 0    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 1    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: loe_sd       Partition: 2    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: loe_sd       Partition: 3    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 4    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: loe_sd       Partition: 5    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002

still not , but what is your suggestion about what to search in the logs , do you mean thta we need to focus about network problems or else? — King David, Sep 29 '18 at 21:51
I not see something un-usual in the logs , can you please advice what are the reasons for this issue , so according to that , I will focus the searching — King David, Sep 29 '18 at 21:56
Well, you setup the cluster using Ambari and the servers should be all in the same LAN, so if you did have network issues, then there is just something wrong with how you prepared the servers to start with. One better explanation could be that some partitions are sitting on a failed hard drive Anyway, clearly one of your servers is not healthy and the logs or some other JMX metrics from the Kafka process should be indicating there is something wrong with it (hint: install and setup Grafana in HDP to monitor Kafka) — OneCricketeer, Sep 30 '18 at 01:38
we also checked the network and the disk that is mounted , and they are fine , so we need to search other things , see jakub answer he said to "more time to re-sync the data" do you know what is mean ? — King David, Sep 30 '18 at 06:30
Kafka has a process called a ReplicaFetcher. It can take time to sync the replicas, depending on network speed and other processes on your machines. Which is why it's not recommended to run Kafka on the same servers as Hadoop and other services. And just because a disk is mounted, doesn't mean it is healthy — OneCricketeer, Sep 30 '18 at 06:34
yes my kafka is on diff machine ( not with hadoop ) , second , is it possible to increase the time of the "re-sync" ? or what is your suggestion to understand why broker isn't in the leader list ? , about the disk - this is new disk so I am not sure this is disk problem ( disk with 2 Tbyte ) — King David, Sep 30 '18 at 06:40
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/181009/discussion-between-king-david-and-cricket-007). — King David, Sep 30 '18 at 06:41
I wouldn't recommend running that until the partitions are in-sync, no. In fact, the tool might not even let you run it while the `ISR != replication.factor` — OneCricketeer, Sep 30 '18 at 06:53
you say "I wouldn't recommend running that until the partitions are in-sync" . how do you know that partitions not in-sync , according to my info ? — King David, Sep 30 '18 at 06:59
The entire ISR section needs to consistently list 3 IDs for all topics with 3 replicas.. — OneCricketeer, Sep 30 '18 at 07:02
yes but this is the goal of the kafka-reassign-partitions , to consistently list 3 IDs — King David, Sep 30 '18 at 07:03
That's not what that script does, though. It actually moves partitions, not forces replicas to sync — OneCricketeer, Sep 30 '18 at 07:04
so if we want to summary the tests that should be done , ( according to what I wrote until now ) , what the verification or the tests that should be done on broker 1002 in order to understand what the problem here? — King David, Sep 30 '18 at 07:08
Sorry, no one can really answer that without physical access to the machine and any platforms monitoring it and tailing the logs... — OneCricketeer, Sep 30 '18 at 07:11

score 0 · Answer 1 · answered Sep 29 '18 at 20:59

You have two options how to influence the partition leadership. With the configuration option auto.leader.rebalance.enable set to true (which should be the default setting), Kafka will automatically try to move the leadership for each partitionto your preferred broker. Preferred broker is the first one in the list of replicas. It is executed as periodical check, so might not happen immediately.

Alternatively - if the automatic rebalance is turned off - you can reassign the replicas manually using the bin/kafka-preferred-replica-election.sh tool. For more info see the Kafka docs.

However, in your case it looks like your broker 1002 is either shortly after the restart or might need more time to re-sync the data, since the broker 1002 is not yet ISR for all partitions. If this is permanent state and you are 100% sure that your broker had enough time to sync all partitions etc., maybe there are some other problems with 1002. But that is hard to say without logs from the broker etc.

yes the default for auto.leader.rebalance.enable yes true , so what we need to verify about broker 1002 that not in leader ? , what I need to check , is it also can be because network problems? — King David, Sep 29 '18 at 21:07
second how to verify broker had enough time to sync all partitions ? — King David, Sep 29 '18 at 21:14
Well, you can always check the logs for errors. As for the time to sync the partitions, you need to know what the size of the partitions is. You can either check it with some file manager or command line by simply checking the directory sizes for the commit logs. Or you can use the utility `kafka-log-dirs.sh` to check the partitions of a topic and the lag. — Jakub, Sep 30 '18 at 14:15

KAFKA - how to re-balance the partitions to available brokers

1 Answers1