3

We have a distributed application following microservice Architecture. In one of our microservice we are following producer-consumer pattern.

The producer receives requests, persists it to database, pushes the request into a BlockingQueue and sends the response back to the client. The consumer running on a separate thread is listening to the blocking queue. The moment it gets the request object it performs specific operations on it.

The request received by the producer is persisted to the database asynchronously using CompleteableFutures.

The problem here is how to forward TraceId to the methods processing the requestObject inside consumer thread. Since the consumer thread might process these objects much later after the response is sent to the consumer.

Also how to forward the traceId across Asynchronous calls?

Thanks

Apurv
  • 315
  • 5
  • 14

1 Answers1

0

That's an interesting question. I think that what you can do is to persist the request together with its headers. Then on the consumer side you can use the SpanExtractor interface in a similar way as we do here - https://github.com/spring-cloud/spring-cloud-sleuth/blob/v1.3.0.RELEASE/spring-cloud-sleuth-core/src/main/java/org/springframework/cloud/sleuth/instrument/web/TraceFilter.java#L351 (Span parent = spanExtractor().joinTrace(new HttpServletRequestTextMap(request));). That means that from the HttpServletRequest we're extracting values to build a span. Then, once you've retrieved the Span, you can just use Tracer#continueSpan(Span) method before processing, and then Tracer#detach(Span) in the finally block. E.g. 

Span parent = spanExtractor().joinTrace(new HttpServletRequestTextMap(request));
try {
   tracer.continueSpan(parent);
   // do whatever you need
} catch(Exception e) {
  tracer.addTag("error", doSthWithTheExceptionMsg(e));
} finally {
  tracer.detach(parent); 
}
Marcin Grzejszczak
  • 10,624
  • 1
  • 16
  • 32
  • 1
    We'll try doing this as well, but what we're doing right now is, persisting the traceId, then in the consumer thread we make a call to :- `Span parent = Span.builder().traceId(Span.hexToId(traceId)).build(); tracer.createSpan("ConsumerSpan", parent);` Do you see any drawbacks with this approach? – Apurv Jan 08 '18 at 03:48
  • moreover the consumer again spawns multiple threads as all the calls inside consumer are async, what would be the right approach to deal with this? – Apurv Jan 08 '18 at 03:54
  • It looks ok in my opinion. Do you observe any problems? – Marcin Grzejszczak Jan 08 '18 at 07:40
  • None so far, Thanks :) – Apurv Jan 08 '18 at 09:38
  • Hi Marcin, Going by this approach, everytime I am making a http request using restTemplate, a new TraceId is generated rather than forwarding the same TraceId also I am not getting x-b3-traceId as a response Header. – Apurv Jan 30 '18 at 10:32
  • Is your RestTemplate a bean? Is it generated on the client side or the server side? If it's not a bean then it will be generated on the server side since the client side will not be instrumented. – Marcin Grzejszczak Jan 30 '18 at 10:34
  • yes it's a bean and it's generated at the server side. – Apurv Jan 30 '18 at 11:52
  • I guess it will be difficult to help you without a sample – Marcin Grzejszczak Jan 30 '18 at 11:52
  • 1
    Hi, Fixed the issue by passing the traceableExecutorPool to CompletableFuture.supplyAsync. Since restTemplate was getting executed in a new thread, which had no traceId associated with it. A new TraceId was being generated for each call to restTemplate.exchange. `CompletableFuture.supplyAsync(() -> restTemplate.exchange(requestEntity, String.class), traceableExecutorService)` Somebody might find this useful :) – Apurv Jan 31 '18 at 04:36
  • 1
    This example is available in the docs afair. The one with wrapping of a callable via Traceable ES. Anyways I'm happy that things are working for you! Can we mark this as answered? – Marcin Grzejszczak Jan 31 '18 at 06:44