New user fails authentication with Gerrit setup using proxy HTTP while other users work

Question

We have an issue with authenticating a new user on our Gerrit setup.

We are running Gerrit 2.16 using proxy HTTP behind a dedicated nginx server. The nginx server listens on port 8080, does basic authentication and then relays the requests to Gerrit on port 9080 on the loopback interface.

This setup works well for all of our current users. Recently we added two new users, one works fine but the other fails authentication. The users were added in different ways and I guess there is something wrong or missing with the user that fails authentication.

The first user (that works) was added following this procedure:

• Username and password was first added to htpasswd file for nginx
• Then login to Gerrit on port 8080 (i.e. to nginx) using the new username and password
• nginx authenticates OK and relays the login request to Gerrit
• Gerrit has never seen the username before, so it initiates the new account creation and the account is created
• The new user can then edit the account profile via the GUI to add their full name and email address

The second user (that doesn't work) was added like this:

• Again, the username and password was first added to htpasswd file for nginx
• Then the 'gerrit' CLI command was used to create a Gerrit account using "ssh -p 29418 localhost gerrit create-account --full-name "'User XXXX'" --email xxxx@example.com xxxxuser" (not the real names and email!)
• The new user then tried to login to Gerrit via nginx on port 8080
• nginx authenticated OK and passed the login request to port 9080, but Gerrit rejected it with error 403

I checked logs/error_log and this is the log of the authentication error for user 2:

[2022-09-29 12:28:19,839] [HTTP-163217] ERROR com.google.gerrit.httpd.auth.container.HttpLoginServlet : Unable to authenticate user "xxxxuser"
com.google.gerrit.server.account.AccountException: Cannot assign external ID "username:xxxxuser" to account 1000186; external ID already in use.
        at com.google.gerrit.server.account.AccountManager.create(AccountManager.java:306)
        at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:140)
        at com.google.gerrit.httpd.auth.container.HttpLoginServlet.doGet(HttpLoginServlet.java:118)
        ....
    

I tried setting the HTTP password in Gerrit with "ssh -p 29418 localhost gerrit set-account xxxxuser --http-password xxxxpass" but it made no difference. I also cleared the HTTP password using "gerrit xxxxuser set-account --clear-http-password" and it also didn't help.

It seems clear that there is something missing or wrong in the account that was set up using "gerrit create-account" but I have no idea what it might be, or how to fix it.

(Wireshark captures on port 9080 confirm that the HTTP GET of the /login/ URL has the correct basic auth header with username and password from the htpasswd file.)

The config files for Gerrit and nginx are as follows:

Gerrit:

[gerrit]
        basePath = /srv/git
        serverId = 71ff3ca6-7f68-45a6-b700-6ca6604f184c
        canonicalWebUrl = http://server2:8080/
[database]
        type = postgresql
        hostname = localhost
        database = reviewdb
        username = gerrit2
[index]
        type = LUCENE
[auth]
        type = HTTP
[receive]
        enableSignedPush = false
[sendemail]
        smtpServer = smtp.gmail.com
        smtpEncryption = ssl
        smtpUser = someuser@gmail.com
        smtpPass = gjhfgjfgjsagjsgjh
        sslVerify = false
        from = Gerrit Code Review <gerrit2@server2.example.com>
[container]
        user = git
        javaHome = /usr/lib/jvm/java-8-openjdk-amd64/jre
        javaOptions = "-Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance"
        javaOptions = "-Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance"
[sshd]
        listenAddress = *:29418
[httpd]
        listenUrl = proxy-http://localhost:9080/
[cache]
        directory = cache
[plugins]
        allowRemoteAdmin = true
[plugin "events-log"]
        maxAge = 20
        returnLimit = 10000
        storeDriver = org.postgresql.Driver
        storeUrl = jdbc:postgresql:gerrit-events
        storeUsername = gerrit-events
        urlOptions = loglevel=INFO
        urlOptions = logUnclosedConnections=true
        copyLocal = true

nginx:

server {
   listen 8080;
   server_name server2;
   
   error_log  /var/log/nginx/error.log;
   access_log /var/log/nginx/access.log;

   location ^~ / {
     auth_basic "Gerrit User Authentication";
     auth_basic_user_file /etc/nginx/auth/gerrit.htpasswd;
     proxy_pass        http://127.0.0.1:9080;
     proxy_set_header  X-Forwarded-For $remote_addr;
     proxy_set_header  Host $host;
   }
}

Answer 1

I found the answer to this - posting here to help anyone else with the same issue.

It turns out that the user account requires a "gerrit:" external ID in order to successfully log in, but the "gerrit create-account" CLI command does not automatically add this external ID, and the ID cannot be added later with the "gerrit set-account" command.

The only way I have found to add the gerrit: external ID to an account is by manually creating the required file on the refs/meta/external-ids branch of the All-Users repository. More details about how to do it can be found in the top answer to this question but in summary the steps are:

1. Clone the All-Users repository
2. Fetch the refs/meta/external-ids branch and check it out
3. Create a file containing the gerrit: external ID (it's just a text file - use one of the other files as a template). The name of the file must be the SHA of the new external ID, which you can get using "echo -n 'gerrit:username' | shasum".
4. Commit the file and push to origin.