Have a opentelementry lambda extension that is running waiting for events. Have the collector to export out to Elasticsearch. I am seeing the service map of the services that lambda is hitting, but not seeing trace data. I am following the documentation in these links. Think a step is missing, but not able to find documentation on exactly how to do this.
https://aws-otel.github.io/docs/getting-started/lambda/lambda-js
I added the OpenTelemetry collector as a lambda extension. Pass in the ES endpoint and token as environment variables. I set the configuration in the collector config yml. I then added instrumentation in the lambda code to export trace data to collector. This is my collector config:
receivers:
otlp:
protocols:
http:
grpc:
exporters:
logging:
loglevel: debug # can be: info, error, debug, warning (default info)
otlp/elastic:
endpoint: "${ELASTIC_OTLP_ENDPOINT}"
headers:
# Elastic APM Server secret token
Authorization: "Bearer ${ELASTIC_OTLP_TOKEN}"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [logging, otlp/elastic]
metrics:
receivers: [otlp]
exporters: [logging, otlp/elastic]
logs:
receivers: [otlp]
exporters: [logging, otlp/elastic]
I can see in log that the collector is starting up and says its listening. The lambda is connected to api gateway endpoint. The ES endpoint and token passed in as environment variables. The lambda adds a record to dynamodb. When api request goes through it looks like it is exporting some spans to ES. One span has status code STATUS_CODE_OK. Another span has status code STATUS_CODE_UNSENT. Besides that not seeing any obvious error info. I can see the connection from lambda to dynamodb in the ElasticSearch Service Map, but not seeing any trace data on the trace page. Something is not getting exported.
Seeing this error from collector:
{
"kind": "exporter",
"name": "otlp/elastic",
"error": "max elapsed time expired failed to push trace data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded",
"dropped_items": 2
}
