How to use Traefik and Envoy in the same project to handle grpc-web?

Question

I've got a service up and running on traefik with LetsEncrypt at grpc.mydomain.com. However, traefik doesn't support routing grpc-web request due to some issue with CORS (https://github.com/containous/traefik/issues/4210). Envoy appears to be an alternative to traefik which works with grpc-web, but I don't want to go about reconfiguring everything.

If I put envoy at envoy.mydomain.com then it actually hits traefik first and traefik can't route the grpc-web requests to envoy. So this doesn't work.

If I put envoy outside of traefik (mydomain.com:9091) then envoy doesn't have the TLS support that traefik has.

Do I need to switch everything to envoy? Is there an alternative I haven't considered? Any guidance welcome :)

Current Traefik Setup:

  traefik:
    image: traefik:v2.0.0
    container_name: traefik
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.grpc.address=:8090
      - --providers.docker
      - --api
      - --serversTransport.rootCAs=/certs/grpc.cert
      # Lets Encrypt Resolvers
      - --certificatesresolvers.leresolver.acme.email=${EMAIL}
      - --certificatesresolvers.leresolver.acme.storage=/etc/acme/cert.json
      - --certificatesresolvers.leresolver.acme.tlschallenge=${TLS_CHALLENGE}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/acme/:/etc/acme/
      - ./secrets/grpc.cert:/certs/grpc.cert
    # Dynamic Configuration
    labels:
      # Dashboard
      - "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAIN}`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.tls.certresolver=leresolver"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.middlewares=authtraefik"

      # https://docs.traefik.io/middlewares/basicauth/
      # password generated from `echo $(htpasswd -nb admin $PASSWORD) | sed -e s/\\$/\\$\\$/g`
      - "traefik.http.middlewares.authtraefik.basicauth.users=admin:$$apr1$$6VzI3S0N$$29FC82dYEbjFN9tPSfWLX1"

      # global redirect to https
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"

      # middleware redirect
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    ports:
      - 80:80
      - 443:443
      - 8090:8090
    networks:
      - internal
      - proxied

  grpc_server:
    image: ${GRPC_IMAGE}
    container_name: grpc_server
    volumes:
      - /tmp/keyset.json:/tmp/keyset.json
      - ./secrets/:/secrets/
    working_dir: /app/__main__/
    labels:
      - "traefik.http.routers.combined_server.rule=Host(`grpc.${DOMAIN}`)"
      - "traefik.http.routers.combined_server.entrypoints=grpc"
      - "traefik.http.routers.combined_server.tls=true"
      - "traefik.http.routers.combined_server.tls.certresolver=leresolver"
      # http
      - "traefik.http.services.grpc-svc.loadbalancer.server.scheme=h2c"
      - "traefik.http.services.grpc-svc.loadbalancer.server.port=8090"
    expose:
      - 8090
    networks:
      - internal
      - proxied

I also tried setting these to fix the CORS error but got nowhere.

      - "traefik.http.middlewares.testheader.headers.accesscontrolallowmethods=GET,PUT,DELETE,POST,OPTIONS"
      - "traefik.http.middlewares.testheader.headers.accesscontrolallowheaders=keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout"
      - "traefik.http.middlewares.testheader.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.testheader.headers.addvaryheader=true"
      - "traefik.http.middlewares.testheader.headers.alloworigin=*"

score 4 · Answer 1 · answered Apr 18 '20 at 13:49

A way to approach this issue is by using 2 different URLs that are both being handled by traefik at first. One URL is being used for "direct grpc" (grpc.mydomain.com), the other one for grpc-web (let's call it grpc-web.mydomain.com). Traefik does TLS termination for both.

The grpc.mydomain.com traffic is directly passed to the container running the grpc_server. The grpc-web.mydomain.com traffic is passed to envoy which acts as a grpc-web-proxy and then passes the traffic to the grpc_server.

So as you are using docker-compose, you would need to add an envoy service to your docker-compose.yml:

---
version: '3'
services:
  traefik:
    # traefik configuration from your question
    # ...
   grpc-server:
    # grpc_server configuration from your question
    # ...
  envoy:
    image: envoyproxy/envoy:v1.14.1
    restart: unless-stopped
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
    labels:
      - traefik.enable=true
      - traefik.http.routers.envoy.rule=Host(`grpc-web.mydomain.com`)
      - traefik.http.services.envoy.loadbalancer.server.port=8080
      - traefik.http.routers.envoy.tls=true
      - traefik.http.routers.envoy.tls.certresolver=leresolver

The envoy.yaml configuration (mounted in the volumes section above) looks like this:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/" }
                          route:
                            cluster: grpc_service
                            max_grpc_timeout: 0s
                      cors:
                        allow_origin_string_match:
                          - prefix: "*"
                        allow_methods: GET, PUT, DELETE, POST, OPTIONS
                        allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                        max_age: "1728000"
                        expose_headers: custom-header-1,grpc-status,grpc-message
                http_filters:
                  - name: envoy.grpc_web
                  - name: envoy.cors
                  - name: envoy.router
  clusters:
    - name: grpc_service
      connect_timeout: 0.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      hosts: [{ socket_address: { address: grpc-server, port_value: 8090 }}]

This is a pretty basic grpc-web config for envoy. The important part to notice is that we set address: grpc-server, port_value: 8090 in the configuration of the "grpc_service" cluster configuration to the service name from the docker-compose.yml and to the port your grpc-server is listening on. Please note I renamed your service from grpc_server to grpc-server as the underscore is not a valid charater in hostnames.

On the client side, use:

"grpc-web.mydomain.com" in your javascript (grpc-web) code.
"grpc.mydomain.com" when writing a client in another language (like golang for example).

I created a working example, which can be found under: https://github.com/rbicker/greeter

dmaixner · Answer 2 · 2020-06-21T09:32:36.340

If you want to get rid of deprecated warnings in envoy, you can update envoy.yaml from this answer with those three changes:

replace:

            - name: envoy.http_connection_manager
              config:

with:

            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager

replace

                  - name: envoy.grpc_web
                  - name: envoy.cors
                  - name: envoy.router

with

                  - name: envoy.filters.http.grpc_web
                  - name: envoy.filters.http.cors
                  - name: envoy.filters.http.router

replace

hosts: [{ socket_address: { address: grpc-server, port_value: 8090 }}]

with

      load_assignment:
        cluster_name: cluster_0
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: grpc-server
                      port_value: 8090

How to use Traefik and Envoy in the same project to handle grpc-web?

2 Answers2