Can we trace external API calls with Istio behind proxy via Kiali?

Question

We have a Nodejs based microservices running in our on-prem kubernetes v1.19 with Istio v1.8.0. What I would like to achieve is trace or display the external API calls in Kiali where we have Jaeger clients for each microservices and able to trace internal traffics.

But so far I could not able to trace any external API calls hits from any microservices.The only thing that I can see the traffic for proxy in Kiali's graph overview.

We have a cooperate proxy, and each container have env proxies set for both http_proxy, https_proxy.Any external service accessible via a cooperate proxy thus traffics should go through the our cooperate proxy first. We have a secured gateway with TLS and we do not have egressgateway where only have istio-ingressgateway.

So is there anyway to trace external traffics likewise the internal traffics inside cluster?If yes what might be the missing thing?

   $ kubectl get pods -n dev
    NAME                                     READY   STATUS    RESTARTS   AGE
    api-dev-74896ff4f9-slxt5                 3/3     Running   0          7h1m
    auth-dev-98f77d487-qt5zd                 3/3     Running   0          3d5h
    backend-dev-bb7765464-b7bpr              2/2     Running   0          7d3h
    mp-dev-86d6b8b978-slqp7                  3/3     Running   0          5d9h
    ui-dev-d5667946b-sdvlc                   2/2     Running   0          5d4h

Here are the ServiceEntries and VirtualServices that I created where I would like to use the retry feature as well the calls for proxy and externalAPI

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: company-proxy
  namespace: dev
spec:
  hosts:
  - foo-proxy.net
  ports:
  - number: PORT
    name: tcp
    protocol: TCP
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: proxy
  namespace: dev
spec:
  hosts:
    - "foo-proxy.net"
  http:
    - name: "company-proxy"
      match:
        - uri:
            prefix: "/"
      route:
        - destination:
            host: "foo-proxy.com"
      timeout: 90s
      retries:
        retryOn: "5xx"
        attempts: 3
        perTryTimeout: 30s
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: foo-example.com
  namespace: dev
spec:
  hosts:
    - "foo-example.com"
  ports:
    - number: 80
      name: http
      protocol: HTTP
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: foo-example.com
  namespace: dev
spec:
  hosts:
    - "foo-example.com"
  http:
    - name: "developer-api"
      match:
        - uri:
            prefix: "/"
      route:
        - destination:
            host: "foo-example.com"
      timeout: 90s
      retries:
        retryOn: "5xx"
        attempts: 3
        perTryTimeout: 30s

@suren with envoy / istio, some parts are automatic (all inbound/outbound HTTP requests), some parts are still up to you (propagating context within your application to "link" the generated span together) — Joel, Feb 18 '21 at 22:55
@suren, we have a Jaeger client endpoint as env variable for each microservices in our code not kiali, where with Istio we can trace all the traffic between microservices without adding extra things to our application's code..But this is not valid for external calls which I suppose it can. I believe as Joel mentioned we need a egress gateway — semural, Feb 19 '21 at 09:19
@suren,they communicate over http in k8s through envoy proxy via mTLS after we set service mesh with Istio.Not sure really what you need further — semural, Feb 19 '21 at 09:25
@suren well I did not add any additional things for tracing or Jaeger cause as I saw it already provides what I want with Istio for internal communications.. So as far as I understand from you, we can manage external API calls without touching istio with Jaeger, right ? And able to trace them in Kiali as well then — semural, Feb 19 '21 at 09:30
@suren, thanks for your answer.. I ll try to do that first with egressgateway and serviceEntry if it ll not work, then no any other choice to do this without adding extra code as you mentioned — semural, Feb 19 '21 at 09:51
@suren "Re: What parts?" => exactly what I wrote before, passing context throughout your app, we are actually saying the same thing — Joel, Feb 19 '21 at 14:01
(but I don't think OP question is about how to propagate headers. If I understand correctly, it's about seeing traces to mesh-external services) — Joel, Feb 19 '21 at 14:03

score 1 · Answer 1 · answered Feb 17 '21 at 20:55

I am not sure why Istio doesn't automatically trace your calls to external APIs. Perhaps it requires an egress gateway to be used, I'm not sure. Note also that Istio creates traces for http(s) traffic, not TCP.

However, this is something you can still do programmatically. You can use any of the Jaeger client libraries to augment"the traces already created by Envoy by appending your own spans.

To do so, you need first to extract the trace context from the HTTP headers of the incoming request (assuming that your external API calls are consecutive to an incoming request), and then create a new span as child of that previous span context. A good idea would be to use OpenTracing semantic conventions when you tag your new span. Tools like Kiali will be able to leverage some information if it follows this convention.

I've found this blog post that explains how to do it with the nodejs jaeger client: https://rhonabwy.com/2019/01/06/adding-tracing-with-jaeger-to-an-express-application/

Can we trace external API calls with Istio behind proxy via Kiali?

1 Answers1