I am pretty new to opentelemetry. I am trying to create a simple setup and get the traces linked. Here is my setup: api_gateway -> producer_lambda -> sqs -> consumer_lambda. I will triggered producer_lambda using api_gateway. In producer_lambda I put a message into sqs, then take the message out in my consumer_lambda. I am using python and arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-python-amd64-ver-1-16-0:2.
I saw on this page saying that there will be "links" field to connect all these traces. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/instrumentation/aws-lambda/#sqs However, with this setup, I am getting 2 traces without any "links" field to connect them together. Am I missing something ? Any help will be appreciated.
producer_lambda code:
import json
import boto3
import os
def lambda_handler(event, context):
client = boto3.client('sqs')
message = client.send_message(
QueueUrl='https://sqs.us-west-2.amazonaws.com/account_number/shuwpan-sqs-between-f1-f2',
MessageBody=("Hello from producer!")
)
otel_env = os.getenv('OTEL_TRACES_SAMPLER')
xray_env = os.getenv('_X_AMZN_TRACE_ID')
print("-----------------", otel_env, " ", xray_env)
return {
'statusCode': 200,
'body': json.dumps('Hello from producer!')
}
consumer_lambda code:
import json
import os
def lambda_handler(event, context):
for record in event['Records']:
payload = record["body"]
print(str(payload))
otel_env = os.getenv('OTEL_TRACES_SAMPLER')
xray_env = os.getenv('_X_AMZN_TRACE_ID')
print("-----------------", otel_env, " ", xray_env)
return {
'statusCode': 200,
'body': json.dumps(str(payload))
}
traces I got, print out with logging exporter:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: 2023-04-17T22:32:12.327Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 3}
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: 2023-04-17T22:32:12.328Z info ResourceSpans #0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Resource SchemaURL:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Resource attributes:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.sdk.language: Str(python)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.sdk.name: Str(opentelemetry)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.sdk.version: Str(1.16.0)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> cloud.region: Str(us-west-2)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> cloud.provider: Str(aws)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.name: Str(shuwpan-lambda-demo-producer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.version: Str($LATEST)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.instance: Str(2023/04/17/[$LATEST]1f5ffcd3e5274523accc6f3572481b06)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> service.name: Str(shuwpan-lambda-demo-producer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.auto.version: Str(0.37b0)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ScopeSpans #0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ScopeSpans SchemaURL:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: InstrumentationScope opentelemetry.instrumentation.botocore 0.37b0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Span #0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Trace ID : 643dc8ea07eb15672e1b9db40d2171ad
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Parent ID : 6ecbbf87e36ad5c0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ID : 3d46aac3d2017ce2
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Name : SQS.SendMessage
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Kind : Client
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Start time : 2023-04-17 22:32:10.831843613 +0000 UTC
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: End time : 2023-04-17 22:32:11.024289366 +0000 UTC
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Status code : Unset
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Status message :
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Attributes:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> rpc.system: Str(aws-api)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> rpc.service: Str(SQS)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> rpc.method: Str(SendMessage)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> aws.region: Str(us-west-2)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> aws.queue_url: Str(https://sqs.us-west-2.amazonaws.com/account_number/shuwpan-sqs-between-f1-f2)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> messaging.system: Str(aws.sqs)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> messaging.url: Str(https://sqs.us-west-2.amazonaws.com/account_number/shuwpan-sqs-between-f1-f2)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> messaging.destination: Str(shuwpan-sqs-between-f1-f2)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> aws.request_id: Str(8faa93e6-1e2e-5a52-8ca7-cf51d5044418)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> retry_attempts: Int(0)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.status_code: Int(200)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> messaging.message_id: Str(519bf217-1e0d-49b4-9017-a23597005caa)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ScopeSpans #1
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ScopeSpans SchemaURL:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: InstrumentationScope opentelemetry.instrumentation.aws_lambda 0.37b0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Span #0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Trace ID : 643dc8ea07eb15672e1b9db40d2171ad
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Parent ID : 2ca98be35330b231
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ID : 6ecbbf87e36ad5c0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Name : lambda_function.lambda_handler
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Kind : Server
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Start time : 2023-04-17 22:32:10.77153787 +0000 UTC
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: End time : 2023-04-17 22:32:11.031541907 +0000 UTC
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Status code : Unset
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Status message :
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Attributes:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.id: Str(arn:aws:lambda:us-west-2:account_number:function:shuwpan-lambda-demo-producer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.execution: Str(1298c1b3-d92e-4117-9af6-2b41de77f4f6)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.trigger: Str(http)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.method: Str(GET)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.route: Str(/shuwpan-lambda-demo-producer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.user_agent: Str(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.scheme: Str(https)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> net.host.name: Str(3tms7s0yyk.execute-api.us-west-2.amazonaws.com)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.target: Str(/shuwpan-lambda-demo-producer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> http.status_code: Int(200)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ResourceSpans #1
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Resource SchemaURL:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Resource attributes:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.sdk.language: Str(python)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.sdk.name: Str(opentelemetry)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.sdk.version: Str(1.16.0)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> cloud.region: Str(us-west-2)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> cloud.provider: Str(aws)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.name: Str(shuwpan-lambda-demo-consumer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.version: Str($LATEST)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.instance: Str(2023/04/17/[$LATEST]487a43e5cfda4e2c9483f8ae3e9a389b)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> service.name: Str(shuwpan-lambda-demo-consumer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> telemetry.auto.version: Str(0.37b0)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ScopeSpans #0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ScopeSpans SchemaURL:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: InstrumentationScope opentelemetry.instrumentation.aws_lambda 0.37b0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Span #0
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Trace ID : 643dc8ebcb6a6e5aadf52f1b9d0035dc
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Parent ID : dcd6f05e40fca1d4
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: ID : a32d9178b3bdcdfc
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Name : lambda_function.lambda_handler
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Kind : Consumer
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Start time : 2023-04-17 22:32:11.164329872 +0000 UTC
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: End time : 2023-04-17 22:32:11.223801355 +0000 UTC
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Status code : Unset
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Status message :
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: Attributes:
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.id: Str(arn:aws:lambda:us-west-2:account_number:function:shuwpan-lambda-demo-consumer)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: -> faas.execution: Str(6d3a9450-065f-59ed-a571-c8e7cd76d360)
Apr 17 22:32:12 ip-10-0-0-9 appdotelcol[411]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
