How to set resource requirements of components for a Storm topology running in Heron?

Question

Recently, I want to migrate a Storm Topology to Heron. But there are some problems with it.
In Heron topology, I can set the resources of components and containers, such as:

conf.setComponentRam("spout", ByteAmount.fromMegabytes(512));
conf.setComponentRam("split", ByteAmount.fromMegabytes(512));
conf.setComponentRam("count", ByteAmount.fromMegabytes(512));
conf.setContainerDiskRequested(ByteAmount.fromGigabytes(3));
conf.setContainerRamRequested(ByteAmount.fromGigabytes(3));
conf.setContainerCpuRequested(2);

But In Storm topology, I can't set these resources requirements. So when I try to running a Storm topology in Heron using RoundRobin Packing, it will use the default resource settings as following:

DEFAULT_DISK_PADDING_PER_CONTAINER = ByteAmount.fromGigabytes(12);
DEFAULT_CPU_PADDING_PER_CONTAINER = 1;
MIN_RAM_PER_INSTANCE = ByteAmount.fromMegabytes(192);
DEFAULT_RAM_PADDING_PER_CONTAINER = ByteAmount.fromGigabytes(2);

But the problem is: the worker node in my Heron cluster does not has so many RAM or Disk resources. So when submitted the Storm topology, the state of this topology will be pending in Aurora, because of the Insufficient: disk or ram.

I am not very familiar with Storm, can I set the resource requirements of the components in the Storm topology? If not, in addition to increasing the resource capacity of the working nodes in the cluster, is there any other way to solve this problem? Any help is very grateful.

score 1 · Accepted Answer · answered Oct 22 '18 at 18:57

First, let me try to understand your question: You are asking about how to set cpu/memory/disk limits for each component in your topology when using the RoundRobinPacking resource packing algorithm.

Second, if I understand your question correctly. Then the answer is just the first part of your question. You can set those limits with the conf object provided by heron framework. Some things to keep in mind are:

You can set the container resources only. In this case, the spouts/bolts in one container will share these resources equally.
You can set the resources for each type of bolt/spout. In this case, the total resources used for each container is the sum of those spouts/bolts packed into that container.
You can combine the above two. In this case, the unspecified spouts/bolts will share the resources remaining.

Third, Heron's underlying architecture is very different from Storm. I don't think Storm offers such fine-grained level of resource control. And Storm doesn't have these configs.

Thanks for your answer. I know how to config the resources of component and container in Heron with the conf object in topology. **But the question is: when I migrated the topology of Storm to run on Heron, there was a problem that the resources could not be allocated due to insufficient resources. Can I set the resource requirements for the Strom topology in Heron?** I think the answer should be **NO**. So in addition to improving the resources of the nodes in the Heron cluster, is there any other solutions? — Yitian Zhang, Oct 28 '18 at 13:39
If the total resource cap is exceeded, then adding new resources into the cluster is the only way to solve this problem. — Neng, Nov 07 '18 at 21:31

How to set resource requirements of components for a Storm topology running in Heron?

1 Answers1