Traefik poor upload perfomance

Question

Recently I moved to traefik as my reverse proxy of choice. But noticed that upload speed to my synology NAS decreased dramatically while using traefik with tls enabled. I did a little of investigation and installed librespeed container to do some speed tests. The results surprised me. Plain http (directly to container over VPN) 150/300, and while using traefik (over public IP) the best it can do was 100/20. VM configuration is 16 CPUs (hardware AES encryption supported / AMD Epyc 7281) and 32 gigs of ram with 10Gb net. Is it the right perfomance I should expect from traefik? Upload speed decreased more than 10 times. Maybe it is configuration issue?

services:
  traefik:
    image: traefik:v2.9.6
    container_name: traefik
    restart: unless-stopped
    networks:
      - outbound
      - internal
    command:
      - "--serversTransport.insecureSkipVerify=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker=true"
      - "--providers.docker.watch"
      - "--providers.docker.network=outbound"
      - "--providers.docker.swarmMode=false"
      - "--entrypoints.http.address=:80"
      - "--entrypoints.https.address=:443"
      - "--entryPoints.traefik.address=:8888"
      - "--entrypoints.http.http.redirections.entryPoint.to=https"
      - "--entrypoints.http.http.redirections.entryPoint.scheme=https"
      - "--providers.file.directory=/rules"
      - "--providers.file.watch=true"
      - "--api.insecure=true"
      - "--accessLog=true"
      - "--accessLog.filePath=/traefik.log"
      - "--accessLog.bufferingSize=100"
      - "--accessLog.filters.statusCodes=400-499"
      - "--metrics"
      - "--metrics.prometheus.buckets=0.1,0.3,1.2,5.0"
      #- "--log.level=DEBUG"
      - "--certificatesResolvers.myresolver.acme.caServer=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.myresolver.acme.storage=acme.json"
      - "--certificatesResolvers.myresolver.acme.httpChallenge.entryPoint=http"
      - "--certificatesResolvers.myresolver.acme.tlsChallenge=true"
      - "--certificatesResolvers.myresolver.acme.email=asd@asd.me"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./traefik/acme.json:/acme.json
      - ./traefik/traefik.log:/traefik.log
      - ./traefik/rules:/rules
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "80:80"
      - "443:443"
      - "8888:8888"
  librespeed:
    image: adolfintel/speedtest
    container_name: librespeed
    environment:
      - MODE=standalone
    networks:
      - outbound
    ports:
      - 8080:80
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.librespeed.rule=Host(`s.mydomain.com`)"
      - "traefik.http.services.librespeed.loadbalancer.server.port=80"
      - "traefik.http.routers.librespeed.entrypoints=https,http"
      - "traefik.http.routers.librespeed.tls=true"
      - "traefik.http.routers.librespeed.tls.certresolver=myresolver"

Maybe up to 2x times speed decrese.

So, regarding CPU usage: server is almost empty. Load average is 0.01 When uploading it spikes up to 5% on one of the traefik workers. — ritualmind, Jan 13 '23 at 06:45
Tried removing logs and disabling all other routers. It affected upload performance by 5% from ~26mbps to ~28mbps. One of workers spikes up to 20% CPU utilization. Thats' it. System is pretty empty. I just don't understand where to look next. — ritualmind, Jan 13 '23 at 13:21

score -1 · Answer 1 · answered Jan 13 '23 at 06:37

There could be a few reasons why you are experiencing a decrease in upload speed when using Traefik as your reverse proxy with TLS enabled.

One potential reason is that the overhead of the encryption and decryption process is causing a bottleneck in your system. The CPU usage of your VM may be high when running Traefik, which can cause a decrease in performance.

Another potential reason could be that the configuration of your Traefik container is not optimized for performance. For example, there might be some misconfigured settings that are causing high CPU usage, or there might be some settings that are not properly utilizing the resources available on your system.

You could try some of the following steps to help improve the performance of your Traefik container:

Increase the number of worker threads in Traefik by adding the --global.sendTimeout=6h and --global.readTimeout=6h to the command. Increase the number of worker processes in Traefik by adding the --workers=16 to the command. To check if the problem is related to the encryption process, you could try disabling the encryption to see if that improves the performance. Finally, you could try disabling the access log, which could help to reduce the CPU usage

There is no such command as "--workers=16" in traefik 2.9.6. As well as --global.readTimeout and --global.sendTimeout. https://doc.traefik.io/traefik/reference/static-configuration/cli/ — ritualmind, Jan 13 '23 at 18:29

Traefik poor upload perfomance

1 Answers1