I'm trying to use OpenTelemetry to trace the microservices I'm working on. The services build up some long running gRPC stream connections with each other and send payload messages in the stream. The gRPC stream's lifespan is as long as the whole microservice and we put the request body into the stream for processing (as the requests are asynchronous and caller doesn't need to get response).
Though the OpenTelemetry provides a set of gRPC stream interceptors, the interceptors only recognize the whole stream connection as a single span. Because each stream message comes as a request with tracing context, I want to the stream message to be able to pass the context around each microservice, and let the other microservice to link to the previous span, so that I can get more useful traces.
currently what I'm doing is add a field and put span ID into the message. But I think this is not a elegant solution as the tracing facility intrudes the business logic. Another solution I'm considering is to put the tracing context into protobuf's unknown fields, but this is some sort of a undefined behavior and those fields might get discarded by the receiving side. I'm wondering what is the best practice in such situation?
