1

Please help me, I need to do the performance testing for the pulsar service with the message rate above 1000000, it there any testing tool can do it?

Olaf Kock
  • 46,930
  • 8
  • 59
  • 90
hezonglin
  • 19
  • 3

2 Answers2

1

Pulsar includes a performance testing tool called pulsar-perf. Given a sufficiently large test server, you can generate high traffic rates using that.

For more information on pulsar-perf, go here

Chris Bartholomew
  • 1,000
  • 6
  • 7
  • but when i use the pulsar perf by the command : pulsar-perf produce topic1 -n 1 -t 10 -r 1000000, the max message rate is just about 100000 msg/s, how can i incress the message ? – hezonglin Jul 03 '21 at 01:09
0

You're probably going to need to take a distributed approach to hit that kind of throughput because you're probably hitting hardware limits that are preventing you from reaching that rate. It simply might not be possible to get that rate if you're running the entire test on a local machine.

If I were trying to run a test at that rate, I'd use pulsar-perf like Chris suggested, but I'd setup a cluster (e.g. in GKE) and have multiple producer applications (on different VMs or containers) running pulsar-perf in parallel, all producing to a partitioned topic. From there, if you want to test the throughput of a non-partitioned topic, you can have a Pulsar function that consumes from the partitioned topic and writes to a non-partitioned topic.

You can also use NiFi or Spark to produce with high throughput. So, those can be good options as well.

devinbost
  • 4,658
  • 2
  • 44
  • 57