3

Using the Serverless framework, I have functions that aren’t attached to an API Gateway Endpoint, such as:

  • Cognito Triggers
  • S3 Event
  • DynamoDB Stream
  • SQS Events

I am also using XRAY tracing, which I have set as tracing: true in my serverless.yml file. It seems that these functions are not being traced, the debug message I receive is:

Ignoring flush on subsegment 20dcd559aa2ab487. Associated segment is marked as not sampled.

Is there any way to have these functions added, either via serverless or cloudformation?

Thanks in advance.

Matt Rowles
  • 7,721
  • 18
  • 55
  • 88

2 Answers2

2

To enable X-Ray tracing for all your Service’s Lambda functions you just need to set the corresponding tracing configuration on the provider level:

provider:
  tracing:
    lambda: true

If you want to setup tracing on a per-function level you can use the tracing config in your function definition:

functions:
  myFunction:
    handler: index.handler
    tracing: true

Setting tracing to true translates to the Active tracing configuration. You can overwrite this behavior by providing the desired configuration as a string:

functions:
  myFunction:
    handler: index.handler
    tracing: PassThrough

Also note that you can mix the provider- and function-level configurations. All functions will inherit the provider-level configuration which can then be overwritten on an individual function basis:

service:
  name: my-tracing-service

provider:
  name: aws
 stage: dev
  runtime: nodejs8.10
  tracing:
    lambda: true

functions:
  myFunc1: # this function will inherit the provider-level tracing configuration
    handler: index.func1
  myFunc2:
    handler: handler.func2
    tracing: PassThrough # here we're overwriting the provider-level configuration

It's recommended to setup X-Ray tracing for Lambda with the aforementioned tracing configuration since doing so will ensure that the X-Ray setup is managed by the Serverless Framework core via CloudFormation.

You can get more granular and specify which resources you want traced as well:

Open your serverless.yml and add a tracing config inside the provider section:

provider:
  ...
  tracing:
    apiGateway: true
    lambda: true

IMPORTANT: Due to CloudFormation limitations it's not possible to enable AWS X-Ray Tracing on existing deployments which don’t use tracing right now.

Please remove the old Serverless Deployments and re-deploy your lambdas with tracing enabled if you want to use AWS X-Ray Tracing for Lambda.

Lastly, don't forget to have the right IAM permission policies configured:

provider:
  ...
  iamRoleStatements:
    - Effect: Allow
      Action:
        ...
        - xray:PutTraceSegments
        - xray:PutTelemetryRecords
      Resource: "*"

To enable X-Ray tracing for other AWS services invoked by AWS Lambda, you MUST Install the AWS X-Ray SDK. In your project directory, run:

$ npm install -s aws-xray-sdk Update your Lambda code and wrap AWS SDK with the X-Ray SDK. Change:

const AWS = require('aws-sdk');

To:

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));

As of Release Serverless v140

lopezdp
  • 1,538
  • 3
  • 21
  • 38
  • 1
    Thanks! Sorry, I should have mentioned that I have `tracing: true` set in Serverless, it does not pick up for anything other than API Gateway attached functions. – Matt Rowles Dec 15 '20 at 19:57
  • 1
    I have tried all of the above btw and it still does not have an automatic segment for S3 events etc. e.g., `const segment = getSegment();` does not find a segment and `const segment = new Segment('some-func');` does not have a trace id – Matt Rowles Dec 15 '20 at 20:00
  • 1
    I have to add a few new services and will play around with this a bit to see what I uncover and ill come back to it and let you know what I find. – lopezdp Dec 16 '20 at 01:13
  • 1
    Can you also try using --> - xray:* also and let me know... Make sure to delete your old deployment bucket and your cloud formation stacks before redeploying it. I know this setting isn't great but as a test Id like to see what you get then work backwards from IAM. – lopezdp Dec 16 '20 at 01:15
  • I cannot delete the stack as it is in production, so will have to wear the lack of functionality if that is the case! But I will try some of the other things. Thanks for your help, will get back in touch shortly. – Matt Rowles Dec 16 '20 at 03:07
1

At the moment Lambda doesn't support continuing traces from triggers other than REST APIs or direct invocation

The upstream service can be an instrumented web application or another Lambda function. Your service can invoke the function directly with an instrumented AWS SDK client, or by calling an API Gateway API with an instrumented HTTP client.

In every other case it will create its own, new Trace ID and use that instead.

You can work around this yourself by creating a new AWS-Xray segment inside the Lambda Function and using the incoming TraceID from the SQS message. This will result in two Segments for your lambda invocation. One which Lambda itself creates, and one which you create to extend the existing trace. Whether that's acceptable or not for your use case is something you'll have to decide for yourself!

If you're working with Python you can do it with aws-xray-lambda-segment-shim.
If you're working with NodeJS you can follow this guide on dev.to.
If you're working with .NET there are some examples on this GitHub issue.

Sam Martin
  • 1,238
  • 10
  • 19