vert.x eventloop is slow consuming from eventbus

Question

We are polling messages from kafka (using Executor Thread) and put it on vert.x eventbus. eventually the verticles(non-workers) consuming those messages from the eventbus.

we measured that the time it takes to each message to be consumed from vertx eventbus after it was sent to it can reach to 4 seconds on peaks.

for some reason occasionally it takes to the eventbus long time to consume messages but not showing thread block error therfore we cant tell what delaying it

weird thing that we dont see any event loop block warnings. what else can we do? cpu/ram are all good. only metric that can actually show something is that:

but it doesnt tell us anything (no idea how eventbus handlers can be associated with eventloop latency)

what shall we check to understand what makes our eventloop consumption from vert.x eventbus to be slow?

It's hard to guide you. For example we don't even know where does this metric come from. I would advise to post a message to the Vert.x users forum with some more context about your application. — tsegismont, Dec 21 '18 at 09:12
@tsegismont regardless this metric. Have you ever had a situation where you slow consuming from eventbus? — rayman, Dec 21 '18 at 10:46
any idea why the eventloop slowly consuming from the eventbus? — rayman, Dec 23 '18 at 21:30

score 0 · Answer 1 · answered Dec 29 '18 at 07:46

0

You should try to measure the amount of data you're sending through the eventbus in a timeline and the amount of data that are received on each verticle through the same eventbus. This can highlight a bottleneck issue.

Maybe you should take a look to the jmx metrics provided by vertx-dropwizard-metrics project. That's can be usefull to be able to compute this timelines.

You also might be interested by taking a look to a Java APM like glowroot for example to have a better understanding of what's wrong.

answered Dec 29 '18 at 07:46

Idriss Neumann

3,760
2
23
32

we are measuring it all using the dropwizard. we measure the time it took a message to be consumed and also measured the average latency of each eventloop request. we cant find anything extreme. – rayman Dec 30 '18 at 09:58
@rayman Maybe you should also analyze the amount of data in memory during the periods when messages took too much time to be consumed. You should also take a look to a Java APM like glowroot to have a better understanding of what's wrong. – Idriss Neumann Dec 31 '18 at 08:16
how java APM would help here? what should I look after? how could it be related to slow eventloop consuming? – rayman Jan 01 '19 at 09:29
@rayman slow eventloop can be related to many reasons: number of waiting threads, size of the thread pool, amount of data in memory, etc. A java APM is a way (among others) to analyze that kind of things and find what's wrong more easily. – Idriss Neumann Jan 01 '19 at 10:28

vert.x eventloop is slow consuming from eventbus

1 Answers1