FLINK : java.io.IOException: Insufficient number of network buffers

Question

I am trying to use flink for data-enrichment on multiple streams of data.

Here I have some data in account_stream and status_stream. I want to add that data to all other streams coming from multiple different sources. all the streams have one field common in their data: "account_id".

This is the approach i took.

account_stream.connect(status_stream)
                     .flat_map(EnrichmentFunction())
                     .filter(lambda x: x['name'] != "-" and x['date'] != "0000-00-00 00:00:00")
                     .key_by(lambda row: row['account_id'])
                     .connect(stream1)
                     .flat_map(function_2())
                     .filter(lambda x: x!="2")
                     .key_by(lambda row: row['account_id'])
                     .connect(stream2)
                     .flat_map(function_2())
                     .key_by(lambda row: row['account_id'])
                     .connect(stream3)
                     .flat_map(function_3())
                     .key_by(lambda row: row['account_id'])
                     .connect(stream4)
                     .flat_map(function_4())
                     .key_by(lambda row: row['account_id'])
                     .connect(stream5)
                     .flat_map(function_5())
                     .key_by(lambda row: row['account_id'])
                     .connect(stream6)
                     .flat_map(function_6())
                     .key_by(lambda row: row['account_id'])
                     .connect(stream7)
                     .flat_map(function_7())
                     .key_by(lambda row: row['account_id'])
                     .connect(stream_8)
                     .flat_map(function_8())
                     .map(lambda a: str(a),Types.STRING())
                     .add_sink(kafka_producer)

I am saving necessary data in state and appending that to all streams using flat_map function. And at the end adding one kafka sink to send all streams enriched with state.

Now once I execute this, I am getting this error:''java.io.IOException: Insufficient number of network buffers: required 17, but only 8 available. The total number of network buffers is currently set to 2048 of 32768 bytes each.''

I tried changing taskmanager.memory.network.fraction to 0.5 , taskmanager.memory.network.max to 15 gb and taskmanager.memory.process.size to 10 gb in flink config file. But it still gave same error. Do I have to do something other than just saving it to see the changes reflect in flink job? or problem is something else?

Also let me know if this approach is not efficient for the task and if there's something else I should try?

I am using single 32gb ram, 8-core server to run this in python with pyflink library, with kafka and elastic running on same server.

Thank you.

What version of Flink are you using? The right way to handle this has evolved over time. — David Anderson, Oct 11 '21 at 17:00
Note that it doesn't make sense to set taskmanager.memory.network.max to be greater than taskmanager.memory.process.size. — David Anderson, Oct 11 '21 at 17:03
@DavidAnderson I am using version 1.13. And I see, it appears that this is not the optimal way to approach this. Maybe can you point me in right direction then? — Mandir Vaibhav, Oct 12 '21 at 03:39

score 0 · Answer 1 · answered Oct 12 '21 at 11:05

You can refer to the Set up TaskManager Memory page of the official documentation for how to configure the network memory for TaskManager. There are several things that need to be taken care of:

taskmanager.memory.network.fraction of total flink memory to be used as network memory. If the derived size is less/greater than the configured min/max size, the min/max size will be used.
The size of network memory cannot exceeds the size of total process memory.
You can find the current max/min value of network memory at the beginning of TaskManager's log. Check it to see whether your configuration works or not.

If you can upgrade your Flink to 1.14, you can try the newest feature: Fine-Grained Resource Management. With this feature the network memory will be automatically configured as the amount each TaskManager requires. However, to use this feature you need to set SlotSharingGroups for each operator and config CPU and memory resources for them. For more details, please refer to the official documentation.

Thanks for the reply, I will look at that v1.14 feature. But I still don't know how much process memory and how much network parameters will be needed for my configuration. Is there any way to know how much memory should I allot? And then any other way to make this scalable? In case I want to add more datastreams? — Mandir Vaibhav, Oct 13 '21 at 04:09

FLINK : java.io.IOException: Insufficient number of network buffers

1 Answers1