6

I want to be able to capture (log) (at least some of) envoy's HTTP headers on my istio service mesh.

I have gone through envoy's docs, and in the log levels' section, it does not mention any header-specific information.

Currently, my istio-proxy log is like this (this is from a stern output):

mysvc-69c46fbc75-d9v8j istio-proxy {"bytes_sent":"124","upstream_cluster":"inbound|80|http|mysvc.default.svc.cluster.local","downstream_remote_address":"10.11.11.1:0","authority":"some.url.com","path":"/health?source=dd_cluster_agent","protocol":"HTTP/1.1","upstream_service_time":"1","upstream_local_address":"127.0.0.1:40406","duration":"2","upstream_transport_failure_reason":"-","route_name":"default","downstream_local_address":"10.11.32.32:20000","user_agent":"Datadog Agent/7.24.0","response_code":"200","response_flags":"-","start_time":"2021-01-17T18:54:57.449Z","method":"GET","request_id":"61ae63c7-aa10-911b-9562-939kdhd49ddhj","upstream_host":"127.0.0.1:20000","x_forwarded_for":"10.16.32.1","requested_server_name":"outbound_.80_.mysvc_.faros.default.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}

Is there a way to log http headers? (ideally some of them, to keep the logging cost under control)

edit1 following advice in the comments, I checked my istio-operator resource and I see that access logging seems to be enabled

    meshConfig:
      accessLogEncoding: JSON
      accessLogFile: /dev/stdout

edit2 I have also tried the following:

curl -i -H "Custom-Header: application/json" https://my.url.net

but in the logs of the istio-ingressgateway I don't see my custom header

istio-ingressgateway-58f69d8696-rmpwn istio-proxy {"user_agent":"curl/7.64.1","response_code":"200","response_flags":"-","start_time":"2021-01-18T19:02:48.645Z","method":"GET","request_id":"8e32c93c-484d-9c56-9489-8c5392793d97","upstream_host":"10.16.32.55:20000","x_forwarded_for":"10.16.32.1","requested_server_name":"my.url.net","bytes_received":"0","istio_policy_status":"-","bytes_sent":"124","upstream_cluster":"outbound|80||mysvc.default.svc.cluster.local","downstream_remote_address":"10.16.32.1:52804","authority":"my.url.net","path":"/","protocol":"HTTP/2","upstream_service_time":"9","upstream_local_address":"10.16.32.17:49826","duration":"10","upstream_transport_failure_reason":"-","route_name":"-","downstream_local_address":"10.16.32.17:8443"}
pkaramol
  • 16,451
  • 43
  • 149
  • 324
  • Have you tried with [envoy access logging](https://istio.io/latest/docs/tasks/observability/logs/access-log/)? Also you should be able to check the http headers with either [jaeger/zipkin](https://istio.io/latest/faq/distributed-tracing/) or some [lua script](https://github.com/kenju/service-mesh-patterns/tree/c22ae3850c2b34515c88a91459542af872d55e24/envoy-lua-filters). – Jakub Jan 18 '21 at 15:50
  • 1
    thanks for the fdeedback Jakub, however I just checked and it seems to be enabled; see my update on the question – pkaramol Jan 18 '21 at 18:56

1 Answers1

6

I think I've succesfully made a reproduction of your issue and I was able to print MY_CUSTOM_HEADER in the ingress gateway logs.

There is a part of my istio ingress gateway logs.

[2021-01-20T08:26:18.587Z] pkarambol GET /productpage HTTP/1.1 200

I've used below curl command:

curl -v -H "MY_CUSTOM_HEADER: pkarambol" xx.xxx.xx.xxx/productpage

To make that happen you have to change the default format of the logs. As mentioned in the documentation, you can use meshConfig.accessLogFormat to change that.

There is an example of Istio Operator I've used.

%REQ(MY_CUSTOM_HEADER)% is a part responsible for displaying the custom header.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogFormat: "[%START_TIME%] %REQ(MY_CUSTOM_HEADER)% %REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% %REQ(X-FORWARDED-FOR)% %REQ(USER-AGENT)%
%REQ(X-REQUEST-ID)% %REQ(:AUTHORITY)% %UPSTREAM_HOST%\n"
Jakub
  • 8,189
  • 1
  • 17
  • 31
  • Thanks `Jakub` I had come to the same conclusion; I was stuck however by the fact that a) I see in my `istio-proxy` logs some fields not existing in the so called default format, e.g. `istio_policy_status: "-"`; so I was trying to find a way to append to the existing log structure and not override it; I can't seem to find where `istio` adds filed that do not exist in the `defailt format` – pkaramol Jan 20 '21 at 17:29
  • 1
    actually since this is a good answer you provided, I will accept it and follow up with my new inquiry on a new SO question (which I will post here fyi) – pkaramol Jan 20 '21 at 17:40
  • https://stackoverflow.com/questions/65815085/istio-somehow-overriding-default-access-logging-format-of-envoy fyi – pkaramol Jan 21 '21 at 07:43