I am wondering what is the best practice for rebalancing a Trident topology? Storm Trident topology seems to set the number of tasks according to the parallelism hints of the stream. When i run the rebalance command I cannot increase the number of executors since executors must be be <= # of tasks. I am using Storm 9.0.1. Any leads will be appreciated.
Asked
Active
Viewed 232 times
1
-
Have you tried to increase the parallelism? Also, it depends on the number of workers available, if all are exhausted, guess you need more hardware (or steal from an other topology). – noMAD Feb 21 '14 at 23:38
-
yes, running the rebalance with -e fails to increase the executors (decrease works BTW). I assume this is because of the tasks boundary. Don't think it is related to # of workers, I am trying to increase the # of executors. – ybensimhon Feb 21 '14 at 23:41
-
I told you about workers cause you were trying to run the `rebalance` command, which would only change the number of workers. If you want to change the executors (threads each worker runs) you need to try changing the parallelism. Also try a different type of grouping (depending on your needs) – noMAD Feb 21 '14 at 23:49
1 Answers
1
The number of tasks always has to be greater-than-or-equal-to the number of executors. So use large parallelism_hint values and then you will be able to re-balance the number of executors to match.
user2601228
- 41
- 4