How to change queue.mem.events in Kibana ECK operator APM server?

Question

i've been getting this 503 queue full errors in kibana APM server.

{"request_id": "1854b2e8-2930-4895-ac96-a213e1253215", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.36.0.128", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/10.17.0", "response_code": 503, "error": "queue is full"}

Looking into the documentation they suggest to change queue.mem.events and output.elasticsearch.bulk_max_size besides horizontally or vertically scaling the server. The problem is i havent find the way to tune this variables in the ECK ApmServer operator yaml:

Tried this:

---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server
  namespace: apm
spec:
  version: 7.5.1
  count: 12
  elasticsearchRef:
    name: "elasticsearch"
  config:
    queue.mem.events: 8198 
    output.elasticsearch.bulk_max_size: 8198
    apm-server:
      rum.enabled: true
      ilm.enabled: true
      rum.event_rate.limit: 1000
      rum.event_rate.lru_size: 1000
      rum.allow_origins: ['']
      rum.library_pattern: "node_modules|bower_components|~"
      rum.exclude_from_grouping: "^/webpack"
      rum.source_mapping.enabled: true
      rum.source_mapping.cache.expiration: 5m
      rum.source_mapping.index_pattern: "apm--sourcemap*"
  http:
    service:
      spec:
        type: LoadBalancer
    tls:
      selfSignedCertificate:
        disabled: true

Also, added a few nodes to the ElasticSearch cluster, but still, the variable doesn't seem to be set correctly because a few minutes later the APM queue gets full again and the errors reappear.

So, how can i change these variables in the ApmServer Kiubernetetes CRD?

How to change queue.mem.events in Kibana ECK operator APM server?

0 Answers0