4

I have a problem in my Storm setup and it looks like there's some discrepancy between the number of executors I set for the topology, and the number of actual bolt processes I see running on one of the servers in that topology.

When setting the number of executors per bolt I use the setBolt method of TopologyBuilder. The number of executors per the UI is correct (total of 105), and when drilling down to the number of executors per server I see that every server in my topology should hold 7-9 executors. This is all good and well however, when sshing to the server and using htop I see that there is one parent process with at least 30 child processes running for that bolt type.

A few notes:

  • I am using a very old version of Storm (0.9.3) that unfortunately I can't upgrade.
  • I'm running a Storm instance that is running python processes (don't know how relevant that is)

I think I'm missing something on the relation between the number of Storm processes and number of bolts/executors I'm configuring or, how to read htop properly. In any case, I would love to get some explanation.

I found this answer, saying that htop shows threads as processes but I still don't think that answers my question.

Thank you

Guy Grin
  • 1,968
  • 2
  • 17
  • 38
  • Are you using repartitions or aggregations on parallel streams? In that case, the bolts will be replicated, but I would not assume that this is displayed in the UI... – moosehead42 Jul 01 '21 at 14:16
  • Hi @moosehead42, this is pretty dated, and we've made modifications from then on, but yes, we did have aggregations. I did not think it would be relevant though... why would the bolts be replicated? We assumed that the reason was that htop also shows kernel level threads (all python threads are such) – Guy Grin Jul 04 '21 at 07:26

0 Answers0