Gitlab-runner Interactive Web Terminal behind Apache Proxy

Question

Summary

I'm currently trying to setup the nice Web-Terminal feature on my self-hosted Gitlab-Instance. But I can't get it running properly. Also debugging is a rather difficult task.

Environment description/Steps to reproduce

Gitlab version: GitLab Community Edition 13.9.0 Gitlab runner on the same machine as Gitlab: Runs in a docker-container with multiple runners configured. Output of docker ps (port 8093 is forwarded correctly):

CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS        PORTS                                                                                                                                                                                                                               NAMES
CONTAINERID    gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   15 hours ago   Up 12 hours   0.0.0.0:8093->8093/tcp                                                                                                                                                                                                              gitlab-runner2

The config.toml is placed in /srv/gitlab-runner/config/config.toml (from the gitlab-docs and I added the following configuration (from the gitlab-docs):

[session_server]
  listen_address = "[::]:8093"
  advertise_address = "localhost:8093"
  session_timeout = 1800

The official docs state that the advertise_address is not needed necessarily and it falls back to the listen_address. Nevertheless I read in several issues here on that topic, that this needs to be the external IP/hostname, where the runners are accessible. In my case they do not need any external IP as they are running on the same machine. Nevertheless I also tried it with my static server-ip and the domain of my selfhosted gitlab without success.

Gitlab is running behind an Apache-Proxy. The Gitlab-docs do not offer a lot information about how the proxy should be configured for the interactive terminals.

My current Apache-config for gitlab looks like this (gitlab is running on port 7777):

<VirtualHost *:443>
    ServerName git.example.com

    ProxyPreserveHost On
    ProxyRequests     Off
    RewriteEngine on
    SSLProxyEngine On
    AllowEncodedSlashes NoDecode

    <Location />
        ProxyPass http://localhost:7777/ nocanon
        ProxyPassReverse https://git.example.com/
        ProxyPassReverse http://localhost:7777/
        Require all granted
    </Location>

    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule .* "wss:/localhost:7777/$1" [P,L]

    ProxyPass "*/terminal.ws" "wss://localhost:7777/"
    <Location */terminal.ws>
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteCond %{HTTP:Connection} upgrade [NC]
        RewriteRule ^/?(.*) "wss://localhost:7777/$1" [P,L]
     </Location>
</VirtualHost>

<VirtualHost *:443>
    ServerName pages.example.com
    ServerAlias *.pages.example.com

    ProxyPreserveHost On
    ProxyPass "/" "http://127.0.0.1:8090/"
    ProxyPassReverse "/" "http://127.0.0.1:8090/

    RewriteEngine on
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
    RewriteRule .* http://127.0.0.1:8090%{REQUEST_URI} [P,QSA]
</VirtualHost>

Here are my relevant changes in the gitlab.rb-file:

nginx['listen_addresses'] = ['0.0.0.0', "[::]"]
nginx['listen_port'] = 7777
nginx['listen_https'] = false
puma['port'] = 8081
external_url 'https://git.example.com'
pages_external_url "http://pages.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['listen_proxy'] = "localhost:8090"

gitlab_pages['redirect_http'] = false
pages_nginx['enable'] = false

I also tried adding this to the gitlab.rb but I think this is the default configuration anyway:

nginx['proxy_set_headers'] = {
  "Host" => "$http_host_with_default",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
}

This is my .gitlab/.gitlab-webide.yml of the project, which I want to use:

terminal:
  image:
    name: python:latest
  script: sleep 60

I also tried to disable my firewall (ufw) and allowed connections on port 8093 without success.

Actual behavior

Gitlab pages are working fine, also the configured runners are working fine in pipelines. But when I click either the "Debug" option for running jobs or the Web-Terminal, a connection error occurs:

terminal.js:47 WebSocket connection to 'wss://git.example.com/user/project/-/jobs/JOB_ID/terminal.ws' failed: Error during WebSocket handshake: Unexpected response code: 404

But the Job for the Webterminal is actually running (the sleep 60 line runs, after this the pipeline succeeded). I can view the log of the job at https://git.example.com/user/project/-/jobs/JOB_ID.

After searching a lot, I now don't know how to proceed - especially because I have no good debug-strategy.

Edit

I tried to connect with the server with websocat wss://0.0.0.0:8094 which returns

websocat: WebSocket SSL error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915: (self signed certificate)
websocat: error running

When I ignore the self signed cert with websocat -t - ws-c:sh-c:'socat - ssl:0.0.0.0:8093,verify=0' --ws-c-uri=wss://echo.websocket.org (some workaround for websocat) it returns the same error as in the WebIDE:

websocat: WebSocketError: Received unexpected status code (404 Not Found)
websocat: error running

The log of the gitlab-runner indicates that the server is listening - then I wonder, why I get a 404 response when I'm trying to connect. docker logs gitlab-runner returns:

All workers stopped. Can exit now                   builds=0
Runtime platform                                    arch=amd64 os=linux pid=6 revision=775dd39d version=13.8.0
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
Running in system-mode.                            
                                                   
Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
Session server listening                            address=[::]:8093 builds=0

The request header of terminal.ws looks like this:

Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: no-cache
Connection: Upgrade
Cookie: event_filter=all; sidebar_collapsed=false; diff_whitespace=0; collapsed_gutter=true; hide_auto_devops_implicitly_enabled_banner_10=false; hide_auto_devops_implicitly_enabled_banner_28=false; diff_view=inline; hide_auto_devops_implicitly_enabled_banner_9=false; pipeline_schedules_callout_dismissed=true; _ga=*; _fbp=*; _gid=*; __stripe_mid=*; known_sign_in=*; _gitlab_session=*
Host: git.example.com
Origin: https://git.example.com 
Pragma: no-cache
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: *
Sec-WebSocket-Protocol: terminal.gitlab.com
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36 OPR/74.0.3911.107

Expected behavior

The terminal should be visible in the browser or any outputs in any log-file.

Used GitLab Runner version

Version:      13.8.0
Git revision: 775dd39d
Git branch:   13-8-stable
GO version:   go1.13.8
Built:        2021-01-20T13:32:47+0000
OS/Arch:      linux/amd64