0

As of a few weeks ago we added filebeat, metricbeat and apm to our dotnet core application ran on our kubernetes cluster. It works all nice and recently we discovered filebeat and metricbeat are able to write a different index upon several rules.

We wanted to do the same for APM, however searching the documentation we can't find any option to set the name of the index to write to.

Is this even possible, and if yes how is it configured?

I also tried finding the current name apm-* within the codebase but couldn't find any matches upon configuring it.

The problem which we'd like to fix is that every space in kibana gets to see the apm metrics of every application. Certain applications shouldn't be within this space so therefore i thought a new apm-application-* index would do the trick...

Edit

Since it shouldn't be configured on the agent but instead in the cloud service console. I'm having troubles to 'user-override' the settings to my likings.

The rules i want to have:

  1. When an application does not live inside the kubernetes namespace default OR kube-system write to an index called apm-7.8.0-application-type-2020-07
  2. All other applications in other namespaces should remain in the default indices

I see you can add output.elasticsearch.indices to make this happen: Array of index selector rules supporting conditionals and formatted string.

I tried this by copying the same i had for metricbeat and updated it to use the apm syntax and came to the following 'user-override'

output.elasticsearch.indices:
- index: 'apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}'
  when:
    not:
      or:
        - equals:
            kubernetes.namespace: default
        - equals:
            kubernetes.namespace: kube-system

but when i use this setup it tells me:

Your changes cannot be applied
'output.elasticsearch.indices.when': is not allowed

Set output.elasticsearch.indices.0.index to apm-%{[observer.version]}-%{[kubernetes.labels.app]}-%{[processor.event]}-%{+yyyy.MM}
Set output.elasticsearch.indices.0.when.not.or.0.equals.kubernetes.namespace to default
Set output.elasticsearch.indices.0.when.not.or.1.equals.kubernetes.namespace to kube-system

Then i updated the example but came to the same conclusion as it was not valid either..

Baklap4
  • 3,914
  • 2
  • 29
  • 56
  • 1
    Are your APM Server outputing directly to elasticsearch? The index name is defined in the [APM Server](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#index-option-es). – leandrojmp Jul 27 '20 at 14:04
  • Yes, @leandrojmp is right, the .NET Agent itself does not control the index name, this is done in the APM Server and the linked docs explains it. ps: I work on the .NET Agent. – gregkalapos Jul 27 '20 at 14:08
  • Ahh i see! We are using the elastic cloud solution. So we haven't set up the configuration of APM-server itself. Is there anyway to update this configuration? – Baklap4 Jul 27 '20 at 14:44
  • Updated the question with a more specific output i'd like – Baklap4 Jul 28 '20 at 09:52

1 Answers1

1

In your ES Cloud console, you need to Edit the cluster configuration, scroll to the APM section and then click "User override settings". In there you can override the target index by adding the following property:

output.elasticsearch.index: "apm-application-%{[observer.version]}-{type}-%{+yyyy.MM.dd}"

Note that if you change this setting, you also need to modify the corresponding index template to match the new index name.

Val
  • 207,596
  • 13
  • 358
  • 360
  • With this solution i can also add the `When` condition rules? I'd like my applications to use a specific template: `apm-%{[observer.version]}-application-%{+yyyy.MM}` while system components (living in a different kubernetes namespace) use the system default `apm-%{[observer.version]}-%{+yyyy.MM.dd}` – Baklap4 Jul 27 '20 at 15:05
  • I've updated my question with the more specific output i'd like :) – Baklap4 Jul 28 '20 at 09:51
  • @Baklap4 found any solution for this ? We have one APM server & want to have multiple applications (apm agents) send the metrics to the same APM server. And we want each application to output the data to different ElasticSearch index. – Kalmesh Sam Feb 16 '21 at 14:22
  • @KalmeshSam there's currently no solution to this. An internal ticket is opened at elastic's side. They should come back to me once it's settled, yet i didn't hear anything as of now. The new Cloud Console UI looks promising though – Baklap4 Feb 16 '21 at 14:27
  • do we have any solution to this. We are adopting APM now and really looking to understand how separate indexes can be used for different application using same APM server – Manoj K Sardana Aug 22 '21 at 12:15