Services empty in Opensearch Trace Analytics

Question

I'm using Amazon OpenSearch with the engine OpenSearch 1.2.

I was working on setting up APM with the following details

• Service 1 - Tomcat Application running on an EC2 server that accesses an RDS database. The server is behind a load balancer with a sub-domain mapped to it. I added a file - setenv.sh file in the tomcat/bin folder with the following content

#!/bin/sh
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:<PATH_TO_JAVA_AGENT>"
export OTEL_METRICS_EXPORTER=none
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<OTEL_COLLECTOR_SERVER_IP>:4317
export OTEL_RESOURCE_ATTRIBUTES=service.name=<SERVICE_NAME>
export OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING=<RDS_HOST_ENDPOINT>=Database-Service

• OTEL Java Agent for collecting traces from the application
• OTEL Collector and Data Prepper running on another server with the following configuration

docker-compose.yml

version: "3.7"
services:
  data-prepper:
    restart: unless-stopped
    image: opensearchproject/data-prepper:1
    volumes:
      - ./pipelines.yaml:/usr/share/data-prepper/pipelines.yaml
      - ./data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
    networks:
      - apm_net

  otel-collector:
    restart: unless-stopped
    image: otel/opentelemetry-collector:0.55.0
    command: [ "--config=/etc/otel-collector-config.yml" ]
    volumes:
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
    ports:
      - "4317:4317"
    depends_on:
      - data-prepper
    networks:
      - apm_net

data-prepper-config.yaml

ssl: false

otel-collector-config.yml

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  otlp/data-prepper:
    endpoint: http://data-prepper:21890
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/data-prepper]

pipelines.yaml

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - otel_trace_raw_prepper:
  sink:
    - opensearch:
        hosts:
          [
            <AWS OPENSEARCH HOST>,
          ]
        # IAM signing
        aws_sigv4: true
        aws_region: <AWS_REGION>
        index_type: "trace-analytics-raw"
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
    - service_map_stateful:
  sink:
    - opensearch:
        hosts:
          [
            <AWS OPENSEARCH HOST>,
          ]
        # IAM signing
        aws_sigv4: true
        aws_region: <AWS_REGION>
        index_type: "trace-analytics-service-map"

The data-prepper is getting authenticated via Fine access based control with all_access role and I'm able to see the otel resources like indexes, index templates generated when running it.

On running the above setup, I'm able to see traces from the application in the Trace Analytics Dashboard of OpenSearch, and upon clicking on the individual traces, I'm able to see a pie chart with one service. I also don't see any errors in the otel-collector as well as in data-prepper. Also, in the logs of data prepper, I see records being sent to otel service map.

However, the services tab of Trace Analytics remains empty and the otel service map index also remains empty.

I have been unable to figure out the reason behind this even after going through the documentation and any help is appreciated!