3

I'm trying to set up a local k8s cluster and on minikube with installed istio and I have an issue with enabling distributed tracing with Jaeger. I have 3 microservices A -> B -> C. I am propagating the all the headers that are needed:

{"x-request-id", "x-b3-traceid", "x-b3-spanid", "x-b3-parentspanid", "x-b3-sampled", "x-b3-flags", "x-ot-span-context"}

But on Jaeger interface, I can only see the request to the service A and I cannot see the request going to service B.

I have logged the headers that are sent in the request. Headers from service A:

Header - x-request-id: c2804368-2ff0-9d90-a2aa-972537968924
Header - x-b3-traceid: 3a2400b40bbe5ed8
Header - x-b3-spanid: 3a2400b40bbe5ed8
Header - x-b3-parentspanid: 
Header - x-b3-sampled: 1
Header - x-b3-flags: 
Header - x-ot-span-context:

Headers from service B:

Header - x-request-id: c2804368-2ff0-9d90-a2aa-972537968924
Header - x-b3-traceid: 3a2400b40bbe5ed8
Header - x-b3-spanid: 3a2400b40bbe5ed8
Header - x-b3-parentspanid:
Header - x-b3-sampled: 1
Header - x-b3-flags:
Header - x-ot-span-context:

So the x-request-id, x-b3-traceid, x-b3-sampled, and x-b3-spanid mathces. There are some headers that aren't set. Also, I'm accessing service A via k8s Service IP of type LoadBalancer, not via ingress. Don't know if this could be the issue.

UPD: I have setup istio gateway so now I'm accessing service A via istio gateway. However the result is the same, I can see the trace for gateway->A but no any further tracing

Sergii Bishyr
  • 8,331
  • 6
  • 40
  • 69
  • 2
    I've found that some frameworks give you empty string when you look for a specific header that does not exist so you may end up propogating a non-existent header to the next request with empty value. I solved a similar issue by only propogating tracing headers with non-empty values. – Utku Turunç Aug 28 '18 at 15:08
  • @UtkuTurunç Wow, looks like this is the issue! Please make it as an answer so I can accept it! :) – Sergii Bishyr Aug 29 '18 at 09:31
  • 1
    glad it worked :) answer is done – Utku Turunç Aug 31 '18 at 08:36
  • 1
    I'm facing the same issue. How did you solve it? In my case, `B` receives the same *value* of `spanid` as `A` sends it. I think the `spanid` should be different at `B`, as by that time a new span should have been created at the sidecar (envoy proxy), right? – Nawaz Mar 15 '19 at 18:20
  • @Nawaz please see the answer by UtkuTurunç . In short: filter the tracing headers that have empty values in your application – Sergii Bishyr Mar 15 '19 at 18:22
  • @SergiiBishyr: I'm not sending empty headers. I'm *propagating* only `x-request-id`, `x-b3-traceid` , `x-b3-spanid` and `x-b3-sampled` because those are what `A` receives in its headers. And all of these are non-empty. – Nawaz Mar 15 '19 at 18:24
  • @Nawaz are you sure that the values of these headers aren’t empty? – Sergii Bishyr Mar 15 '19 at 18:25
  • Yes. the service `A` and `B` return them as *responses* (in their corresponding *body*). So I can see them in the output. – Nawaz Mar 15 '19 at 18:26
  • I'm using `Minikube` BTW. – Nawaz Mar 15 '19 at 18:30
  • @Nawaz there are no additional requirements for minikube. Maybe you can share the code for forwarding the headers? – Sergii Bishyr Mar 15 '19 at 20:36
  • @SergiiBishyr: Please see the gists here: https://gist.github.com/snawaz/0fc8ff3f5e964f146d8abf9cf344aea6 – Nawaz Mar 16 '19 at 05:56
  • My issue is similar to [this](https://groups.google.com/forum/#!topic/envoy-users/5L-mvO6_Q1Y) and [this posted in different group](https://groups.google.com/forum/#!topic/istio-users/p_HE1FdCBWk) – Nawaz Mar 16 '19 at 09:12
  • @Nawaz why don't you froward headers `x-b3-parentspanid `, `x-b3-parentspanid ` and `x-b3-flags`? – Sergii Bishyr Mar 19 '19 at 11:16
  • @SergiiBishyr: they're *empty* .. actually they don't even exist in the headers. – Nawaz Mar 19 '19 at 16:38

1 Answers1

3

Some web frameworks return empty string if a non-existent header is queried. I have seen this in Spring Boot and KoaJS.

If any of the tracing headers is not sent by Istio, this header logic causes us to send empty string for those non-existent headers which breaks tracing.

My suggestion is after getting the values for headers filter out the ones with empty string as their values and propogate the remaining ones.

Utku Turunç
  • 334
  • 1
  • 7
  • 2
    Thank you for this answer. Worked perfectly for Istio 1.0.6 and Jaeger. I wasted two days trying to solve this issue in Go. – Gary Stafford Mar 19 '19 at 01:52