1

As the title says, I'm trying to serve Gitlab through an nginx reverse proxy, with both programs being run in separate docker containers connected through a docker network. A picture as an example:

Linux Host
 ____________________________
|                            |
| Docker                     |
|  __________________________|
| |                          |
| | Docker network (test-net)|
| |  ________________________|
| | |                        |
| | | nginx        gitlab    | Only nginx has a port bound to the host (443).
| | | |   |        |   |     | TLS is terminated at nginx as well.
| | | |   |   -->  |   |     | in my test, I have nginx running as localhost.
| | | |___|        |___|     | To access gitlab, hit https://localhost/git/
| | |________________________|
| |__________________________|
|____________________________|

nginx runs with this docker command:

docker run -dit --network=test-net --name=nginx -p 443:443 -v "$PWD/conf":/etc/nginx:ro nginx:alpine && docker logs -f nginx

nginx.conf

<Removed unnecessary config from here, very basic setup>
http {
    keepalive_timeout 65;
    server {
        listen 443 ssl;
        server_name localhost;
        ssl_certificate localhost.crt;
        ssl_certificate_key localhost.key;
        ssl_protocols   TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!MD5;
        location /git/ {
            proxy_pass http://test/;
        }
    }
}

gitlab.rb

<only relevant parts added here>
external_url 'https://localhost'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
 "Host" => "$http_host_with_default",
 "X-Real-IP" => "$remote_addr",
 "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
 "X-Forwarded-Proto" => "http",
 "Upgrade" => "$http_upgrade",
 "X-Forwarded-Ssl" => "on",
 "Connection" => "$connection_upgrade"
}
nginx['custom_error_pages'] = {
  '404' => {
    'title' => '404',
    'header' => 'You\'ve been hit by !! You\'ve been struck by ! A false URL.',
    'message' => 'Double check that URL! Is it correct?'
  }
}

docker-compose.yml for gitlab:

version: '3.7'
services:
  gitlab:
    image: 'internal-docker-repo:1234/gitlab/gitlab-ce:11.8.3-ce.0'
    restart: always
    hostname: 'test'
    container_name: test
    volumes:
      - './config:/etc/gitlab:rw'
    networks:
      - net
networks:
  net:
    external: true
    name: test-net

Internally (to docker networks) nginx is known as nginx and gitlab is known as test. I have confirmed I can ping each container from inside the other, using their container names.

As it is now, it almost works. When I go to https://localhost/git/ on my linux host I get a 404 error page from gitlab, but no login screen.

the 404 screen. Custom, so I know gitlab is running and picked up the configuration

I'm obviously missing something but I'm not sure what it is. It's hard for me to tell if it's an NGinx configuration issue or a Gitlab configuration issue.

Log output when I hit https://localhost/git/

nginx log output:

172.19.0.1 - - [07/Jan/2020:21:28:35 +0000] "GET /git/ HTTP/1.1" 404 2289 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"      

gitlab log output:

test      | ==> /var/log/gitlab/nginx/gitlab_access.log <==
test      | 172.19.0.3 - - [07/Jan/2020:21:28:35 +0000] "GET / HTTP/1.0" 404 2289 "" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
test      | 
test      | ==> /var/log/gitlab/gitlab-workhorse/current <==
test      | 2020-01-07_21:28:35.10649 test 127.0.0.1:0 - - [2020/01/07:21:28:35 +0000] "GET / HTTP/1.1" 404 3108 "" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" 0.001
test      | 
Scrambo
  • 11
  • 1
  • 5
  • I think it is somewhat related to this: https://serverfault.com/questions/799012/dynmap-through-mod-proxy-on-a-external-webserver - Try to inspect the page and check where the errors occur, and then remap it accordingly. Otherwhise, just use a subdomain for gitlab, you wont have that issue then. – Orphans Jan 14 '20 at 14:38

4 Answers4

1

I think at least one of the problems (if not the most important) is the fact that gitlab expects https://localhost as the URL in your browser. Unfortunately, you're not giving it that, but https://localhost/git, although the request obviously reaches the backend (which is the gitlab container).

Changing the external url directive on the inner nginx to "https://localhost/git" might work as a quick fix. Obviously you need to reconfigure gitlab afterwards:

sudo gitlab-ctl reconfig

If I were you, though, I would use a dedicated domain for gitlab (so a separate server block in nginx), as this would be cleaner to my mind. Something to the effect of:

server {
        listen 443 ssl;
        server_name gitlab.example.com;
        ssl_certificate localhost.crt;
        ssl_certificate_key localhost.key;
        ssl_protocols   TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!MD5;
        location / {
            proxy_pass http://test/;
              }
        }

And then change the external_url, of course, to gitlab.example.com.

Since you're using the localhost anyway, I don't think it would be much bother if you added this new domain in /etc/hosts : 127.0.0.1 gitlab.example.com

Lethargos
  • 455
  • 2
  • 7
  • 19
  • The problem is that I'm actively trying to avoid using subdomains to proxy containers. While I've already confirmed I can proxy using subdomains, relative paths are easier to remember. For that reason I can't accept this answer, even though it is a way to go about proxying gitlab, and that anyone else who reads this should take using subdomains into account. Only changing the `external_url` to have `/git` at the end didn't work. Looking at the logs it seems that the initial get request to the server 404's because nginx removes the `/git/` relative path when forwarding the request. – Scrambo Jan 16 '20 at 19:00
1

Cause

The reason you can't get connected is because of the location in Nginx mismatch the external_url URL defined in gitlab.rb.

In your Nginx configuration, you specify location /git/, while in the Gitlab configuration, you use https://localhost. This will cause the error: when a client requests to access https://localhost/git it will always tail the "git/" at the end of URI, however, your Gitlab is configured to access via https://localhost, so it will never be able to find the webpage!

Solution:

As either change the location /git/ to location / or change the https://localhost to https://localhost/git.

Example

Here is my full configuration on my docker, you can take this as reference:

*Note: 172.16.0.10 is the IP of the Gitlab server, gitlab.drive.nr resolves the IP of the Nginx server

Nginx docker:

$ cat nginx-r.sh 
sudo docker run -itd --name=nginx-r \
    --network=testenv \
    --ip 172.16.0.3 \
    --dns 172.16.0.2 \
    -h rproxy.drive.nr \
    -p 80:80 -p 443:443 \
    -v nginx-config:/etc/nginx \
    -v nginx-certs:/etc/ssl/private \
    nginx:alpine

Gitlab docker:

$ cat gitlab.sh 
sudo docker run --detach \
  --hostname gitlab.drive.nr \
  --name gitlab \
  --restart always \
  --network testenv \
  --ip 172.16.0.10 \
  --dns 172.16.0.2 \
  --volume gitlab-config:/etc/gitlab \
  --volume gitlab-logs:/var/log/gitlab \
  --volume gitlab-data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

Nginx configurations:

# cat conf.d/sites-enabled/gitlab.conf 
server {
  listen   80;
  location / {
    return 301 https://172.16.0.10;
    }
}

server {
  listen   443 ssl;
  server_name   gitlab.drive.nr;
  ssl_certificate /etc/ssl/private/gitlab.crt;
  ssl_certificate_key /etc/ssl/private/gitlab.key;
  ssl_session_cache builtin:1000 shared:SSL:10m;
  location /git {
    proxy_pass http://172.16.0.10;
  }
}

Gitlab configurations:

# cat gitlab-config/gitlab.rb |grep -v ^#
external_url 'http://gitlab.drive.nr/git'

Test Drive

enter image description here

Log on Nginx:

172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git HTTP/1.1" 302 102 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/users/sign_in HTTP/1.1" 302 150 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU HTTP/1.1" 200 9502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/application-aeddf31361633b3d1196c6483f25c484855e0f243e7f7e62686a4de9e10ec03b.css HTTP/1.1" 200 149097 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/print-74c3df10dad473d66660c828e3aa54ca3bfeac6d8bb708643331403fe7211e60.css HTTP/1.1" 200 382 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/webpack/runtime.ee78bc38.bundle.js HTTP/1.1" 200 2134 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/highlight/themes/white-3144068cf4f603d290f553b653926358ddcd02493b9728f62417682657fc58c0.css HTTP/1.1" 200 864 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/webpack/main.b91d0a07.chunk.js HTTP/1.1" 200 830262 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:26 +0000] "GET /git/assets/webpack/default.ca6f81b2.chunk.js HTTP/1.1" 200 159 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:27 +0000] "GET /git/assets/icons-e91700f3f1ecff110fc2c35aa62aec8f2aad69d1bfb35844186a11175a79e25f.svg HTTP/1.1" 200 25983 "https://gitlab.drive.nr/git/users/password/edit?reset_password_token=ayjmm32J3Ry_gDR8JbzU" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:27 +0000] "GET /git/assets/touch-icon-ipad-retina-8ebe416f5313483d9c1bc772b5bbe03ecad52a54eba443e5215a22caed2a16a2.png HTTP/1.1" 200 5662 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
172.16.0.1 - - [14/Jan/2020:23:35:27 +0000] "GET /git/assets/favicon-7901bd695fb93edb07975966062049829afb56cf11511236e61bcf425070e36e.png HTTP/1.1" 200 1611 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0" "-"
NeilWang
  • 383
  • 4
  • 14
  • Thank you for going into such detail to answer the question I posed. I'll be awarding you the 100 bounty for the depth of answering, but it didn't fully fix my problem, so I wont be able to mark it as correct. I'll provide another comment later in the day describing what has happened, I'm late for a meeting now. – Scrambo Jan 16 '20 at 19:02
0

I was facing the exact problem described here.

The following settings made it work.

GitLab Docker:

      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://<domain>/gitlab/'

[...]

    ports:
      - "8929:80"

NGINX:

   location /gitlab/ {

      #  ## https://github.com/gitlabhq/gitlabhq/issues/694
      #  ## Some requests take more than 30 seconds.
       proxy_read_timeout      300;
       proxy_connect_timeout   300;
       proxy_redirect          off;

       proxy_http_version 1.1;

       proxy_set_header    Host                $http_host;
       proxy_set_header    X-Real-IP           $remote_addr;
       proxy_set_header    X-Forwarded-Ssl     on;
       proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
       proxy_set_header    X-Forwarded-Proto   $scheme;

       proxy_pass http://<domain>:8929/gitlab/;
}
pat-s
  • 101
  • 3
0

I had a similar problem (both Gitlab and Nginx running as Docker containers).

The cause of my error was mistakenly using the https protocol when defining the external_url setting.

I.e. it should be:

external_url 'http://MY_DOMAIN'

rather than:

external_url 'https://MY_DOMAIN'

My Nginx config was handling the http -> https redirection for me.

I hope this helps people encountering the same problem.

Andrew Schulman
  • 8,811
  • 21
  • 32
  • 47