Alter client quotas using kafka-python

Question

I'm using kafka-python library to manage my kafka cluster. I want to add the quotas features (alter/describe), these features are not yet implented in kafka-python library but they are available in kafka protocol doc here.

AlterClientQuotas API (Key: 49):

Requests:

AlterClientQuotas Request (Version: 0) => [entries] validate_only 
entries => [entity] [ops] 
  entity => entity_type entity_name 
    entity_type => STRING
    entity_name => NULLABLE_STRING
 ops => key value remove 
    key => STRING
    value => FLOAT64
    remove => BOOLEAN
validate_only => BOOLEAN

Responses:

AlterClientQuotas Response (Version: 0) => throttle_time_ms [entries] 
 throttle_time_ms => INT32
 entries => error_code error_message [entity] 
   error_code => INT16
   error_message => NULLABLE_STRING
   entity => entity_type entity_name 
     entity_type => STRING
     entity_name => NULLABLE_STRING

To implement these features I'm adding the following code

kafka/protocol/admin.py

from kafka.protocol.api import Request, Response

class AlterClientQuotasResponse_v0(Response):
    API_KEY = 49
    API_VERSION = 0
    SCHEMA = Schema(
        ("throttle_time_ms", Int32),
        (
            "entries",
            Array(
                ("error_code", Int16),
                ("error_message", String),
                ("match", CompactString("utf-8")),
                (
                    "entity",
                    Array(
                        ("entity_type", String("utf-8")),
                        ("entity_name", String("utf-8")),
                    ),
                ),
            ),
        ),
    )

class AlterClientQuotasResponse_v1(Response):
    API_KEY = 49
    API_VERSION = 0
    SCHEMA = Schema(
        ("throttle_time_ms", Int32),
        (
            "entries",
            Array(
                ("error_code", Int16),
                ("error_message", String),
                ("match", CompactString("utf-8")),
                (
                    "entity",
                    Array(
                        ("entity_type", String("utf-8")),
                        ("entity_name", String("utf-8")),
                    ),
                ),
                ("tags", TaggedFields),
            ),
        ),
    )

class AlterClientQuotasRequest_v0(Request):
    API_KEY = 49
    API_VERSION = 0
    RESPONSE_TYPE = AlterClientQuotasResponse_v0
    SCHEMA = Schema(
        (
            "entries", Array(
                ("entity_type", String("utf-8")),
                ("entity_name", String("utf-8")),
            )
        ),
        (
            "ops", Array(
                ("key", String("utf-8")),
                ("value", Float64),
                ("remove", Boolean),
            )
        ),
        ("validate_only", Boolean)
    )

class AlterClientQuotasRequest_v1(Request):
    API_KEY = 49
    API_VERSION = 0
    RESPONSE_TYPE = AlterClientQuotasResponse_v1
    SCHEMA = Schema(
        (
            "entries", Array(
                ("entity_type", String("utf-8")),
                ("entity_name", String("utf-8")),
            )
        ),
        (
            "ops", Array(
                ("key", String("utf-8")),
                ("value", Float64),
                ("remove", Boolean),
            )
        ),
        ("validate_only", Boolean),
        ("tags", TaggedFields),
    )

AlterClientQuotasResponse = [
    AlterClientQuotasResponse_v0,
    AlterClientQuotasResponse_v1
]
AlterClientQuotasRequest = [
    AlterClientQuotasRequest_v0,
    AlterClientQuotasRequest_v1
]

kafka/admin/client.py

request = AlterClientQuotasRequest[0](
  entries=[("user", "<...>.producer"),],
  ops=[("producer_byte_rate", 1024, False),],
  validate_only=True,
  #tags={}
)
print("--> Quota Request :", request)
future = self._send_request_to_node(self._client.least_loaded_node(), request)
self._wait_for_futures([future])
print("-->>> FUTURE Value", future.value)

Running the code below gets the follwing errors :

client output

File /<...>/py310/lib/python3.10/site-packages/kafka/admin/client.py:1342, in KafkaAdminClient._wait_for_futures(self, futures) 1339 self._client.poll(future=future) 1341 if future.failed(): -> 1342 raise future.exception
```
KafkaConnectionError: KafkaConnectionError: socket disconnected
```
server output

Feb 03 11:24:25 broker-05: org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: ALTER_CLIENT_QUOTAS, apiVersion: 0, connectionId: , listenerName: ListenerName(SECURED), principal: User: Feb 03 11:24:25 broker-05 Caused by: java.lang.RuntimeException: Tried to allocate a collection of size 292211, but there are only 71 bytes remaining. Feb 03 11:24:25 broker-05 at org.apache.kafka.common.message.AlterClientQuotasRequestData$EntryData.read(AlterClientQuotasRequestData.java:347) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.message.AlterClientQuotasRequestData$EntryData.(AlterClientQuotasRequestData.java:300) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.message.AlterClientQuotasRequestData.read(AlterClientQuotasRequestData.java:125) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.message.AlterClientQuotasRequestData.(AlterClientQuotasRequestData.java:73) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.requests.AlterClientQuotasRequest.parse(AlterClientQuotasRequest.java:142) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.requests.AbstractRequest.doParseRequest(AbstractRequest.java:269) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.requests.AbstractRequest.parseRequest(AbstractRequest.java:165) Feb 03 11:24:25 broker-05 at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:95) Feb 03 11:24:25 broker-05 at kafka.network.RequestChannel$Request.(RequestChannel.scala:101) Feb 03 11:24:25 broker-05 at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1096) Feb 03 11:24:25 broker-05 at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608) Feb 03 11:24:25 broker-05 at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1074) Feb 03 11:24:25 broker-05 at kafka.network.Processor.run(SocketServer.scala:960) Feb 03 11:24:25 broker-05 at java.base/java.lang.Thread.run(Thread.java:829)

Has anyone been able to implement this ? help please ?

Thank you !

Alter client quotas using kafka-python

0 Answers0