Problem:
I'm building a composition API resource where I in the inbound scope have defined some send-request policy's. For observability we use Application Insights, and I've activated this in APIM. The logging in APIM is configured on a global level with 100% sampling, verbose, and Legacy protocol to not miss anything. The goal I want to achieve is to have my all services traced, and visible through Application Insights so that I can utilize the transaction diagnostics under the Failures and Performance tabs. All downstream services also utilize their own Application Insights and are .Net C# based.
My expected results were that APIM would use the same Request-Id for all dependent services called by my send-request policies which would result in a complete trace in the transaction diagnostic view.
The actual result is a trace graph where only the incoming request is logged, and not the called services.
Dependencies are correctly correlated for the requests made in the backend scope however.
I have managed a workaround and all my debugging efforts point me to that APIM is not appending the Request-Id header to the send-request policies in the inbound section.
Workaround:
Some people on the internet has suggested that manually adding a set-header, preferably to the global policy like this solves it:
<set-header name="Request-Id" exists-action="skip">
<value>@(context.RequestId.ToString())</value>
</set-header>
The above combined with either mode="copy" or a set-header policy in the send-requestpolicies do work.
However, this is only true if we force the client to send the Request-Id header in the original request, which is very fragile. We can't let our internal tracing be dependent on API callers using it correctly.
If the caller instead does not send it, the above policy will try to resolve the Request-Id value used by Application insights for the initial request and backend request in a policy, but the actual result is that the value will differ from the one shown as Operation Id in Application Insights used for correlation, which breaks the trace and no dependencies are shown.
Just to be clear, the built in trace available in blob storage after setting the Ocp-Apim-Trace: true header is not what I'm after, that already works.
I feel like this is very basic functionality and that I must be doing something wrong. Any help is greatly appriciated!
Below is a sample configuration that doesn't trace as it should. Global policy is default with just CORS added:
<policies>
<inbound>
<cors allow-credentials="true">
<allowed-origins>
<origin>https://api.website.com</origin>
</allowed-origins>
<allowed-methods preflight-result-max-age="300">
<method>*</method>
</allowed-methods>
<allowed-headers>
<header>*</header>
</allowed-headers>
<expose-headers>
<header>*</header>
</expose-headers>
</cors>
</inbound>
<backend>
<forward-request />
</backend>
<outbound />
<on-error />
<policies>
<inbound>
<base />
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
<policies>
<inbound>
<base />
<!-- No requests done in this inbound scope get the `Request-Id` header and consequently no traces -->
<send-request mode="copy" response-variable-name="service1" timeout="10" ignore-error="false">
<set-url>myservice</set-url>
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override">
<value>application/json</value>
</set-header>
<set-body>"hello": "world"</set-body>
</send-request>
</inbound>
<backend>
<base />
<!-- the forwarded request in this backend block always get the correct `Request-Id` header, without any configuration at all. -->
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
Edit 1: Similar problem but different correlation protocol: How to Get/Set Traceparent header in APIM inbound Policy if client dose not sent?
Edit 2: Corrected misstake in example
