1

I am trying to export opentelemetry metrics to open search.

My configurations are as mentioned below

 metrics-pipeline:
  source:
    otel_metrics_source::
  processor:
    - otel_metrics_raw_processor:
  sink:
    - opensearch:
        hosts: ["https://<domain-name>:443"]
        insecure: true
        username: "username"
        password: "password

I was going through one of the data prepper issue and came to know that metrics support is included recently.

https://github.com/opensearch-project/data-prepper/issues/242

I am not able to find proper documentation on this.

In the data prepper pod, getting the below exception

    com.amazon.dataprepper.model.plugin.NoPluginFoundException: Unable to find a plugin named 'otel_metrics_source:'. Please ensure that plugin is annotated with appropriate values.
    at com.amazon.dataprepper.plugin.DefaultPluginFactory.lambda$getPluginClass$2(DefaultPluginFactory.java:111) ~[data-prepper.jar:1.5.1]
    at java.util.Optional.orElseThrow(Optional.java:401) ~[?:?]
    at com.amazon.dataprepper.plugin.DefaultPluginFactory.getPluginClass(DefaultPluginFactory.java:111) ~[data-prepper.jar:1.5.1]
    at com.amazon.dataprepper.plugin.DefaultPluginFactory.loadPlugin(DefaultPluginFactory.java:62) ~[data-prepper.jar:1.5.1]

Appreciate any inputs on this.

Muqthar Ali
  • 83
  • 10

2 Answers2

2

Currently, there is no section on the opentelemetry metrics support in the general documentation of DataPrepper. You can find documentation within the respective plugin directories:

There is also a blog post on the OpenTelemetry metrics ingestion with DataPrepper in the OpenSearch blogs. It contains a configuration example.

Karsten Schnitter
  • 183
  • 4
  • Thanks for the update. I am using same configurations but getting the below exception. com.amazon.dataprepper.model.plugin.NoPluginFoundException: Unable to find a plugin named 'otel_metrics_source:' Added this exception details in question. – Muqthar Ali Sep 07 '22 at 11:57
  • Your error message looks, as if the plugin was not included in the jar. OTel Metrics support is included since version 1.4.0. The error message indicates you use version 1.5.1. This should include the metrics source. You got a double :: at the end of the line with otel_metrics_source. Maybe this leads to a problem? – Karsten Schnitter Sep 07 '22 at 13:12
  • As per this documentation https://opensearch.org/blog/technical-post/2022/05/opentelemetry-metrics-for-data-prepper/ double colon :: is required. Let me check by removing one colon : – Muqthar Ali Sep 07 '22 at 16:59
  • Let me know, if this worked. I think, I have made a mistake in the blog post. I will correct it depending on your findings. – Karsten Schnitter Sep 07 '22 at 19:17
  • If I remove one colon : , getting the below exception Caused by: java.lang.IllegalArgumentException: ssl is enabled, sslKeyCertChainFile can not be empty or null at com.amazon.dataprepper.plugins.source.otelmetrics.OTelMetricsSourceConfig.validateSSLArgument(OTelMetricsSourceConfig.java:108) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.source.otelmetrics.OTelMetricsSourceConfig.validateSSLCertificateFiles(OTelMetricsSourceConfig.java:113) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.source.otelmetrics.OTelMetricsSourceConfig. – Muqthar Ali Sep 08 '22 at 06:33
  • I have added ssl:false for otel_metrics_source: then ssl issue is fixed but getting new exception Caused by: java.lang.IllegalStateException: Missing required properties:indexAlias at com.amazon.dataprepper.plugins.sink.opensearch.index.IndexConfiguration.(IndexConfiguration.java:68) ~[data-prepper.jar:1.5.1] – Muqthar Ali Sep 08 '22 at 06:49
  • Did you switch off ssl in the data-prepper-config.yaml as shown in https://github.com/opensearch-project/data-prepper/blob/main/docs/simple_pipelines.md#simple-configuration? Otherwise you will have to configure and provide the certificate to be used. – Karsten Schnitter Sep 08 '22 at 11:04
  • Yes, I switched off ssl in config yaml. Now there is no exception during data prepper start up. Can you please help me on how to check the data metrics in open search? Please point me to any documentation if you have. – Muqthar Ali Sep 08 '22 at 11:28
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247878/discussion-between-muqthar-ali-and-karsten-schnitter). – Muqthar Ali Sep 08 '22 at 11:30
  • The documentation has been updated to reflect the solution quoted in the answer by @Mugthar Ali. – Karsten Schnitter Sep 18 '22 at 18:46
2

Just remove the extra colon at the end of otel_metrics_source , set ssl flag to false and add the index in opensearch section.

Thank you @Karsten Schnitter for your help.

Updated configurations

    metrics-pipeline:
         source:
            otel_metrics_source:
              ssl: false
         processor:
            - otel_metrics_raw_processor:
         sink:
            - opensearch:
                 hosts: ["https://<domain-name>:443"]
                 insecure: true
                 username: "username"
                 password: "password
                 index: metrics-otel-v1-%{yyyy.MM.dd}
Muqthar Ali
  • 83
  • 10