7

@kafka users:

I have been trying to understand python client for kafka, including PyPy client as well. Following was a good benchmarking i read and realized some similar results:

http://mrafayaleem.com/2016/03/31/apache-kafka-producer-benchmarks/

I am extremely confused whether Java has a massive advantage over python, as the libraries are written using Java and kafka. So my question is, does the native implementation of Kafka in Java helps with performance tremendously when Java is used, or PyPy/Python works equally better?

Being a python programmer, I am not at all comfortable with java, and hence confused.

Omkar
  • 2,274
  • 6
  • 21
  • 34

1 Answers1

13

Apache Kafka defines a language neutral wire protocol (see https://kafka.apache.org/protocol) and so clients can be written in any programming language and do not need to be based on the Java client that ships with the core Kafka distribution. For example, the c/c++ librdkafka library is a very high performance non-Java client. There are a number of python Kafka clients including one that is based on librdkafka. Benchmark results and other comparison information for the various Kafka Python clients is available here http://activisiongamescience.github.io/2016/06/15/Kafka-Client-Benchmarking/

Hans Jespersen
  • 8,024
  • 1
  • 24
  • 31
  • thanks a lot. Just one quick question, there seem to be a lot of courses which teach kafka and other apache stack in Java, that actually made me assume that native language implementation helps performance. So, according to your answer, all you are saying that kafka-python and other clients are simply communicators and have less to do with actual backend engine? – Omkar Jun 02 '17 at 10:58
  • 1
    The Java client ships with Kafka and all the other clients are shipped separately. That's why most online materials focus on Java. It has nothing to do with performance. – Hans Jespersen Jun 02 '17 at 20:49