Wrong aws request signature caused by opentelemetry https plugin

Question

When using the @opentelemetry/plugin-https and the aws-sdk together in a NodeJS application, the opentelemetry plugin adds the traceparent header to each AWS request. This works fine if there is no need for retries in the aws-sdk. When the aws-sdk retries a request the following errors can occur:

• InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
• SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.

The first AWS request contains the following headers:

• traceparent: '00-32c9b7adee1da37fad593ee38e9e479b-875169606368a166-01'
• Authorization: 'AWS4-HMAC-SHA256 Credential=<credential>, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-target, Signature=<signature>'

Note that the SignedHeaders doesn't include traceparent.

The retried request contains the following headers:

• traceparent: '00-c573e391a455a207469ffa4fb75b3cab-6f20c315628cfcc0-01'
• Authorization: AWS4-HMAC-SHA256 Credential=<credential>, SignedHeaders=host;traceparent;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-target, Signature=<signature>

Note that the SignedHeaders does include traceparent.

Before the retry request is sent, the @opentelemetry/plugin-https sets new traceparent header and this makes the signature of the AWS request invalid.

Here is a code which reproduces the issue (you may need to run the script a few times before hitting the rate limit which causes the retries):

const opentelemetry = require("@opentelemetry/api");
const { NodeTracerProvider } = require("@opentelemetry/node");
const { SimpleSpanProcessor } = require("@opentelemetry/tracing");
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");

const provider = new NodeTracerProvider({
    plugins: {
        https: {
            enabled: true,
            path: "@opentelemetry/plugin-https"
        }
    }
});

const exporter = new JaegerExporter({ serviceName: "test" });

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

provider.register();

const AWS = require("aws-sdk");

const main = async () => {
    const cwl = new AWS.CloudWatchLogs({ region: "us-east-1" });

    const promises = new Array(100).fill(true).map(() => new Promise((resolve, reject) => {
        cwl.describeLogGroups(function (err, data) {
            if (err) {
                console.log(err.name);
                console.log("Got error:", err.message);
                console.log("ERROR Request Authorization:");
                console.log(this.request.httpRequest.headers.Authorization);
                console.log("ERROR Request traceparent:");
                console.log(this.request.httpRequest.headers.traceparent);
                console.log("Retry count:", this.retryCount);

                reject(err);
                return;
            }

            resolve(data);
        });
    }));

    const result = await Promise.all(promises);

    console.log(result.length);
};

main().catch(console.error);

Possible solutions:

1. Ignore all calls to aws in the @opentelemetry/plugin-https.
   • Ignoring the calls to aws will lead to loosing all spans for aws requests.
2. Add the traceparent header to the unsignableHeaders in the aws-sdk - AWS.Signers.V4.prototype.unsignableHeaders.push("traceparent");
   • Changing the prototype seems like a hack and also doesn't handle the case if another node module uses different version of the aws-sdk.

Is there another solution which could allow me to keep the spans for aws requests and guarantees that the signature of all aws requests will be correct?

Update (16.12.2020):

The issue seems to be fixed in the aws sdk v3

The following code throws the correct error (ThrottlingException):

const opentelemetry = require("@opentelemetry/api");
const { NodeTracerProvider } = require("@opentelemetry/node");
const { SimpleSpanProcessor } = require("@opentelemetry/tracing");
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");
const { CloudWatchLogs } = require("@aws-sdk/client-cloudwatch-logs");

const provider = new NodeTracerProvider({
    plugins: {
        https: {
            enabled: true,
            path: "@opentelemetry/plugin-https"
        }
    }
});

const exporter = new JaegerExporter({ serviceName: "test" });

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

provider.register();

const main = async () => {
    const cwl = new CloudWatchLogs({ region: "us-east-1" });

    const promises = new Array(100).fill(true).map(() => new Promise((resolve, reject) => {
        cwl.describeLogGroups({ limit: 50 })
            .then(resolve)
            .catch((err) => {
                console.log(err.name);
                console.log("Got error:", err.message);

                reject(err);
            });
    }));

    const result = await Promise.all(promises);

    console.log(result.length);
};

main().catch(console.error);

Answer 1

someone prepared a NPM package for it as well

https://www.npmjs.com/package/otel-aws-sdk-retry-patch

https://github.com/NativeChat/otel-aws-sdk-retry-patch