2

I am trying to find size of kafka SinkRecord in bytes because our Sink Connector buffers data for some time.

I have looked at various stackoverflow answers to find the right way to find approximate size in bytes.

In Java, what is the best way to determine the size of an object?

But, I am seeing various results from different approaches.

final String JSON_WITHOUT_SCHEMA = "{\"userid\": \"User_1\"}";
JsonConverter converter = new JsonConverter();
HashMap<String, String> converterConfig = new HashMap<String, String>();
converterConfig.put("schemas.enable", "false");
converter.configure(converterConfig, false);
SchemaAndValue noSchemaInputValue = converter.toConnectData("topic", JSON_WITHOUT_SCHEMA.getBytes(StandardCharsets.UTF_8));
    SinkRecord noSchemaRecordValue =
            new SinkRecord(
                    "topic",
                    0,
                    Schema.STRING_SCHEMA,
                    "test",
                    noSchemaInputValue.schema(),
                    noSchemaInputValue.value(),
                    0);
System.out.println("Sink record size in bytes from instrumentation:" + InstrumentationAgent.getObjectSize(noSchemaRecordValue));


System.out.println("Sink record size in bytes from jol-core:" + GraphLayout.parseInstance(noSchemaRecordValue).totalSize());

Sink record size in bytes from instrumentation:56
Sink record size in bytes from jol-core:792

I also tried out this code and it returns 1168 bytes.

I have used this code for instrumentation API:

Question:

  • Is the instrumentation API returning bytes after deep visit of object?
  • Does kafka have any API to determine size of record in bytes?
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
JayPatel
  • 121
  • 1
  • 6

0 Answers0