I'm working on an API that queries several APIs. My API is instrumented with opentelemetry. I also have a httpx client that is also instrumented. My API queries the other APIs with this httpx client.
For most of the APIs I'm talking to, the httpx instrumentation works and I'm seeing the traces/spans in Cloud Trace. For 2 APIs, I don't see any span. This is most likely because these APIs are behind a load balancer and/or Apigee. I'm not in control of the LB or of Apigee. One of those removes the x-cloud-trace-context from the responses header, which I think prevents the instrumentation from being sent to Cloud Trace.
I would like to inject the trace ID anyway. Here is what I tried:
from opentelemetry import trace
async_client = AsyncClient()
HTTPXClientInstrumentor().instrument_client(async_client, request_hook=request_hook, response_hook=response_hook)
async def request_hook(span, request):
# trace_id = format(trace.get_current_span().get_span_context().trace_id, "032x")
trace_id = str(trace.get_current_span().get_span_context().trace_id)
if "x-cloud-trace-context" not in request.headers:
request.headers["x-cloud-trace-context"] = f"{trace_id};o=1"
request.headers["X-Cloud-Trace-Context"] = f"{trace_id};o=1"
span.trace_id = trace_id
request.headers["traceparent"] = trace_id
return span, request
async def response_hook(span, request, response):
# trace_id = format(trace.get_current_span().get_span_context().trace_id, "032x")
trace_id = str(trace.get_current_span().get_span_context().trace_id)
response.headers.pop("x-request-id")
response.headers["x-cloud-trace-context"] = f"{trace_id};o=1"
response.headers["X-Cloud-Trace-Context"] = f"{trace_id};o=1"
return span, request, response
Whatever I try, it looks like Cloud Trace isn't picking up these HTTP requests. Would you know why?
