0

Im getting following error while changing the APM configuration in apm-server.yml -

'output.elasticsearch.bulk_max_size': has to be a number between [100, 5000] inclusive

I am changing the config to increase the size of APM queue so that it can handle more than 30K events. Below is the config change that I did -

output.elasticsearch:
    worker: 4
    bulk_max_size: 10000
queue.mem.events: 40000

I cannot understand the reason of this error. ref - https://www.elastic.co/guide/en/apm/server/master/tune-apm-server.html#adjust-queue-size

Note - The issue is partially resolved by the answer given by Opster Elasticsearch Ninja's . But - The issue is not fully resolved yet. But I'm getting this error a lot less (2 times last week) after his suggestion to increase node and keeping bulk_max_size as 5000 only. But I'm still looking into some quantitative answers to this issue. Something like X node, W worker, B bulk_max_size, and S queue.mem.events will provide us the capacity to have a maximum N number of events per minute or at a time. For example - one of the last errors was when events in 1 minute reached around 90-100K, so I'm still researching.

Rahul
  • 863
  • 6
  • 23
  • The issue is not fully resolved it. But I'm getting this error a lot less (2 times last week) after your suggestion to increase node and keeping bulk_max_size as 5000 only. Marking answer as accepted. But I'm still looking into some quantitative answer to for this issue. Something like X node with W worker with B bulk_max_size and S queue.mem.events will provide us the capacity to have maximum N number of events per minute or at a time. For example - one of the last error was when events in 1 minute reached around 90-100K, so Im still researching. But thanks @OpsterElasticsearchNinja – Rahul Aug 04 '20 at 12:15

1 Answers1

2

Error message is very clear that you can't have it beyond 5k while you are giving value to 10K, can you try limiting in the range and see it works?

Please refer official bulk_max_size doc, which also mentions

setting bulk_max_size to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

Drawback of setting it to very large value

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Amit
  • 30,756
  • 6
  • 57
  • 88
  • Yeah Opster. I wanted to know the reason of 5K limit. because I know for sure that it can be set more than 5K but something else is blocking it to be set in my case. https://www.elastic.co/guide/en/apm/server/master/tune-apm-server.html#adjust-queue-size In this example also they ask to try 5120. Thanks anyway. – Rahul Jul 15 '20 at 06:36
  • @Rahul updated my answer with the reasoning of limiting – Amit Jul 15 '20 at 06:37
  • 1
    that is not a reason, that is a recommendation I think. In my case the elasticsearch is well configured and properly tuned so it can handle the request. APM CPU usage are around 20-30%. – Rahul Jul 15 '20 at 06:42
  • @Rahul also which version you are using, this looks like a mistake on their documentation part, you can look at this value in their source code? – Amit Jul 15 '20 at 06:44
  • 7.8. Anyways lets suppose that the limit is 5K then how do one handle 30-35K requests. APM starts giving error. – Rahul Jul 15 '20 at 06:48
  • @Rahul then you need to add more nodes and scale it horizontally or throttle it from your application, in short – Amit Jul 15 '20 at 07:00
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/217870/discussion-between-opster-elasticsearch-ninja-and-rahul). – Amit Jul 15 '20 at 07:04