MongoDb replica set in a single image fails in GitLab CI/CD due to ReplicaSetGhost

Question

I am struggling with MongoDb 6 Replica Set and automated testing in CI/CD.

I have created a single docker image with a Replica Set for development and testing purposes.

This works well when running the docker container locally and running my tests against it. See the repository https://gitlab.com/sunnyatticsoftware/sandbox/mongo-rs And see the docker image in the public repository registry.gitlab.com/sunnyatticsoftware/sandbox/mongo-rs

Basically it has a dockerfile

FROM mongo:6.0.5-jammy as base

COPY ./init-mongodbs.sh ./init-replica.sh ./entry-point.sh /

RUN chmod +x /init-mongodbs.sh && \
    chmod +x /init-replica.sh && \
    chmod +x /entry-point.sh

# Data directory
ARG DB1_DATA_DIR=/var/lib/mongo1
ARG DB2_DATA_DIR=/var/lib/mongo2
ARG DB3_DATA_DIR=/var/lib/mongo3

# Log directory
ARG DB1_LOG_DIR=/var/log/mongodb1
ARG DB2_LOG_DIR=/var/log/mongodb2
ARG DB3_LOG_DIR=/var/log/mongodb3

# DB Ports
ARG DB1_PORT=27017
ARG DB1_PORT=27018
ARG DB1_PORT=27019

RUN mkdir -p ${DB1_DATA_DIR} && \
    mkdir -p ${DB1_LOG_DIR} && \
    mkdir -p ${DB2_DATA_DIR} && \
    mkdir -p ${DB2_LOG_DIR} && \
    mkdir -p ${DB3_DATA_DIR} && \
    mkdir -p ${DB3_LOG_DIR} && \
    chown `whoami` ${DB1_DATA_DIR} && \
    chown `whoami` ${DB1_LOG_DIR} && \
    chown `whoami` ${DB2_DATA_DIR} && \
    chown `whoami` ${DB2_LOG_DIR} && \
    chown `whoami` ${DB3_DATA_DIR} && \
    chown `whoami` ${DB3_LOG_DIR}

EXPOSE ${DB1_PORT}
EXPOSE ${DB2_PORT}
EXPOSE ${DB3_PORT}

ENTRYPOINT [ "bash", "entry-point.sh" ]

and it copies some scripts that run when executing the container

The entry-point.sh

#!/bin/bash

/bin/bash ./init-replica.sh &
/bin/bash ./init-mongodbs.sh

The init-mongodbs.sh

#!/bin/bash

# Data directory
DB1_DATA_DIR="/var/lib/mongo1"
DB2_DATA_DIR="/var/lib/mongo2"
DB3_DATA_DIR="/var/lib/mongo3"

# Log directory
DB1_LOG_DIR="/var/log/mongodb1"
DB2_LOG_DIR="/var/log/mongodb2"
DB3_LOG_DIR="/var/log/mongodb3"

REPLICA_SET="${REPLICA_SET_NAME:-rs0}"


mongod --dbpath ${DB1_DATA_DIR} --logpath ${DB1_LOG_DIR}/mongod.log --fork --port 27017 --bind_ip_all --replSet $REPLICA_SET
mongod --dbpath ${DB2_DATA_DIR} --logpath ${DB2_LOG_DIR}/mongod.log --fork --port 27018 --bind_ip_all --replSet $REPLICA_SET
mongod --dbpath ${DB3_DATA_DIR} --logpath ${DB3_LOG_DIR}/mongod.log --port 27019 --bind_ip_all --replSet $REPLICA_SET

And the init-replica.sh

#!/bin/bash

DB1_PORT=27017
DB2_PORT=27018
DB3_PORT=27019

LOCAL_HOST="${HOST:-localhost}"
REPLICA_SET="${REPLICA_SET_NAME:-rs0}"
SLEEP_INITIATE="${DELAY_INITIATE:-30}"

RS_MEMBER_1="{ \"_id\": 0, \"host\": \"${LOCAL_HOST}:${DB1_PORT}\", \"priority\": 2 }"
RS_MEMBER_2="{ \"_id\": 1, \"host\": \"${LOCAL_HOST}:${DB2_PORT}\", \"priority\": 0 }"
RS_MEMBER_3="{ \"_id\": 2, \"host\": \"${LOCAL_HOST}:${DB3_PORT}\", \"priority\": 0 }"

echo "Waiting ${SLEEP_INITIATE} seconds before initiating replica set"
sleep ${SLEEP_INITIATE} 
mongosh --eval "rs.initiate({ \"_id\": \"${REPLICA_SET}\", \"members\": [${RS_MEMBER_1}, ${RS_MEMBER_2}, ${RS_MEMBER_3}] });"

echo "Replica set initiated"
echo "$(mongosh --eval "rs.status()")"

so that it can be run as

version: '3.8'
services:
  mongors:
    image: registry.gitlab.com/sunnyatticsoftware/sandbox/mongo-rs
    container_name: mongors
    environment:
      HOST: mongors
      DELAY_INITIATE: 40
    ports:
      - 27017:27017
      - 27018:27018
      - 27019:27019

Notice it accepts an environment variable HOST which I can use to give an alias mongors so that in GitLab CI/CD I can refer to that hostname.

My app connection string could be something like this

mongodb://mongors:27017,mongors:27018,mongors:27019/?replicaSet=rs0&readPreference=primary&ssl=false

As I said, this works well when running the replica set as a docker container locally (see the docker compose).

When it runs in GitLab CI/CD it timesout. I use shared runners from GitLab.com, the built-in.

Running with gitlab-runner 16.1.0~beta.5.gf131a6a2 (f131a6a2)

Yet the connectivity seem to work fine

$ nc -zv mongors 27017
Connection to mongors (172.17.0.4) 27017 port [tcp/*] succeeded!
$ nc -zv mongors 27018
Connection to mongors (172.17.0.4) 27018 port [tcp/*] succeeded!
$ nc -zv mongors 27019
Connection to mongors (172.17.0.4) 27019 port [tcp/*] succeeded!

So connectivity is OK but I get exceptions from my app like this

System.TimeoutException: A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongors:27017" }", EndPoint: "Unspecified/mongors:27017", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 6.0.0, TopologyVersion: { "processId" : ObjectId("646f9b54fb0fbb72cb9a3b70"), "counter" : NumberLong(0) }, Type: "ReplicaSetGhost", WireVersionRange: "[0, 17]", LastHeartbeatTimestamp: "2023-05-25T17:36:03.9519134Z", LastUpdateTimestamp: "2023-05-25T17:36:03.9519144Z" }] }.

Suggesting that there is a ReplicaSetGhost or something wrong with the replica set.

Any idea why in GitLab CI/CD I cannot have my automated integration tests connect to the connection string using the service alias which matches my HOST env variable?

At some point I saw in traces this

MongoServerError: No host described in new configuration with {version: 1, term: 0} for replica set rs0 maps to this node

So maybe the replica set internally does not understand the hostname mongors?

---

UPDATE 1 (25 May): I created a sample repo with a sample pipeline that uses the standalone mongoDB single instance and also the image mongo-rs I created, to test connectivity.

https://gitlab.com/sunnyatticsoftware/sandbox/mongo-rs-tester

image: ubuntu:22.04

stages:
  - test

test_mongors:
  stage: test
  services:
    - name: registry.gitlab.com/sunnyatticsoftware/sandbox/mongo-rs
      alias: mongors
  variables:
    # MongoDB
    HOST: "mongors"
    DELAY_INITIATE: "50"
  before_script:
    - apt-get update
    - apt-get install -y curl
    - apt-get install -y gnupg
    - curl -fsSL https://www.mongodb.org/static/pgp/server-6.0.asc | tee /etc/apt/trusted.gpg.d/mongodb.asc > /dev/null
    - echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list
    - apt-get update
    - apt-get install -y mongodb-mongosh
    - sleep 30
  script: |
    output=$(mongosh --host mongors:27017 --eval "db.stats()" 2>&1); if [ $? -eq 0 ]; then echo "OK"; else echo "ERROR: $output"; fi

test_mongo:
  stage: test
  services:
    - name: mongo:6.0.5-jammy
      alias: mongostandalone
  variables:
    MONGO_INITDB_ROOT_USERNAME: root
    MONGO_INITDB_ROOT_PASSWORD: dummy

  before_script:
    - apt-get update
    - apt-get install -y curl
    - apt-get install -y gnupg
    - curl -fsSL https://www.mongodb.org/static/pgp/server-6.0.asc | tee /etc/apt/trusted.gpg.d/mongodb.asc > /dev/null
    - echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list
    - apt-get update
    - apt-get install -y mongodb-mongosh
  script: |
    output=$(mongosh --host mongostandalone:27017 --username root --password dummy --eval "db.stats()" 2>&1); if [ $? -eq 0 ]; then echo "OK"; else echo "ERROR: $output"; fi

---

UPDATE 2 (29 May): Adding the IP to the /etc/hosts didn't make any difference.

image: ubuntu:22.04

stages:
  - test

test_mongors:
  stage: test
  services:
    - name: registry.gitlab.com/sunnyatticsoftware/sandbox/mongo-rs
      alias: mongors
  variables:
    # MongoDB
    HOST: "mongors"
    DELAY_INITIATE: "50"
  before_script:
    - apt-get update
    - apt-get install -y curl
    - apt-get install -y gnupg
    - apt-get install -y dnsutils
    - MONGORS_ALIAS="mongors"
    - MONGORS_IP_ADDRESS=$(getent hosts $MONGORS_ALIAS | awk '{ print $1 }')
    - echo "$MONGORS_IP_ADDRESS  $MONGORS_ALIAS" >> /etc/hosts
    - echo "MongoDB IP $MONGORS_IP_ADDRESS added"
    - cat /etc/hosts
    - curl -fsSL https://www.mongodb.org/static/pgp/server-6.0.asc | tee /etc/apt/trusted.gpg.d/mongodb.asc > /dev/null
    - echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list
    - apt-get update
    - apt-get install -y mongodb-mongosh
    - sleep 30
  script: |
    output=$(mongosh --host mongors:27017 --eval "db.stats()" 2>&1); if [ $? -eq 0 ]; then echo "OK"; else echo "ERROR: $output"; fi

test_mongo:
  stage: test
  services:
    - name: mongo:6.0.5-jammy
      alias: mongostandalone
  variables:
    MONGO_INITDB_ROOT_USERNAME: root
    MONGO_INITDB_ROOT_PASSWORD: dummy

  before_script:
    - apt-get update
    - apt-get install -y curl
    - apt-get install -y gnupg
    - curl -fsSL https://www.mongodb.org/static/pgp/server-6.0.asc | tee /etc/apt/trusted.gpg.d/mongodb.asc > /dev/null
    - echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list
    - apt-get update
    - apt-get install -y mongodb-mongosh
  script: |
    output=$(mongosh --host mongostandalone:27017 --username root --password dummy --eval "db.stats()" 2>&1); if [ $? -eq 0 ]; then echo "OK"; else echo "ERROR: $output"; fi

---

UPDATE 3: 30 May

To simplify things, I've created this other repository to test my MongoDB Replica Set image

https://gitlab.com/sunnyatticsoftware/sandbox/mongo-rs-tester-dotnet

Basically it's a tiny .NET app with an automated test that can run with a

dotnet test

I have added a docker-compose.yml just with a container for my replica set MongoDB image

version: '3.8'
services:
  mongors:
    image: registry.gitlab.com/sunnyatticsoftware/sandbox/mongo-rs
    container_name: mongors
    environment:
      HOST: mongors
      DELAY_INITIATE: 40
    ports:
      - 27017:27017
      - 27018:27018
      - 27019:27019

and the equivalent for .gitlab-ci.yml which FAILS (and I don't know why)

image: mcr.microsoft.com/dotnet/sdk:7.0

stages:
  - test

test:
  stage: test
  services:
    - name: registry.gitlab.com/sunnyatticsoftware/sandbox/mongo-rs
      alias: mongors
  variables:
    HOST: "mongors"
    DELAY_INITIATE: "40"
  before_script:
    - sleep 30
  script: 
    - dotnet test
  allow_failure: false

The steps would be:

Requirements

• .NET 7 SDK or runtime
• Docker and docker-compose

How to run in local

1. Clone the repo
2. Add to /etc/hosts (or in Windows C:\Windows\System32\drivers\etc\hosts) this entry with the alias for localhost

127.0.0.1   mongors

3. Execute docker-compose up to spin up a mongors container
4. Run tests with dotnet test

Verify that the test is successful and it connects to the Mongo Replica Set database

Notice the connection string is mongodb://mongors:27017,mongors:27018,mongors:27019/?replicaSet=rs0&readPreference=primary&ssl=false, which uses the alias mongors.

How to run in GitLab CI/CD

Now let's try the equivalent in GitLab CI/CD

1. Run the CI/CD pipeline, which will spin up a container service with the alias mongors and which will run the dotnet test command as a script
2. Verify that the test is successful if it properly connects to the Mongo Replica Set database.

If the test is unsuccessful and the pipeline fails, it's because the Mongo RS container and tests behave differently when running within GitLab CI/CD. Why?

It fails with

 System.TimeoutException : A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongors:27017" }", EndPoint: "Unspecified/mongors:27017", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 6.0.0, TopologyVersion: { "processId" : ObjectId("6475c89d5538f6657eda5c74"), "counter" : NumberLong(0) }, Type: "ReplicaSetGhost", WireVersionRange: "[0, 17]", LastHeartbeatTimestamp: "2023-05-30T09:59:11.3576006Z", LastUpdateTimestamp: "2023-05-30T09:59:11.3576012Z" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongors:27018" }", EndPoint: "Unspecified/mongors:27018", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 6.0.0, TopologyVersion: { "processId" : ObjectId("6475c89ea07e16061b09e4ec"), "counter" : NumberLong(0) }, Type: "ReplicaSetGhost", WireVersionRange: "[0, 17]", LastHeartbeatTimestamp: "2023-05-30T09:59:11.3575679Z", LastUpdateTimestamp: "2023-05-30T09:59:11.3576205Z" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongors:27019" }", EndPoint: "Unspecified/mongors:27019", ReasonChanged: "Heartbeat", State: "Connected", ServerVersion: 6.0.0, TopologyVersion: { "processId" : ObjectId("6475c89e4cf3656cf898c899"), "counter" : NumberLong(0) }, Type: "ReplicaSetGhost", WireVersionRange: "[0, 17]", LastHeartbeatTimestamp: "2023-05-30T09:59:11.3496821Z", LastUpdateTimestamp: "2023-05-30T09:59:11.3496847Z" }] }

---

UPDATE 4: 31 May

I have disabled shared runners and created and registered my own Linux runner with docker type.

With this runner, the tests went successfully.

https://gitlab.com/sunnyatticsoftware/sandbox/mongo-rs-tester-dotnet/-/jobs/4385389688

I hoped services and CI/CD were agnostic of the runner..

Is there any explanation on why it wouldn't work with shared runners?

Answer 1

If it works well locally, that should mean the setup itself is fine, but the way GitLab CI/CD handles networking and/or the way MongoDB replica set resolves hostnames is problematic.

The error message MongoServerError: No host described in new configuration with {version: 1, term: 0} for replica set rs0 maps to this node suggests that the hostname mongors is not recognized inside the MongoDB replica set.

By default, MongoDB replica set members communicate with each other using the hostnames that were specified during the rs.initiate() command. This is typically not a problem when you are running MongoDB instances on separate machines or on a single machine using localhost, but it can cause issues when you are running MongoDB inside Docker containers, where the hostname inside the container may not be the same as the hostname outside the container.

In your case, you are initializing the MongoDB replica set using the HOST environment variable (which is set to mongors), so the MongoDB nodes should be using the mongors alias for internode communication. However, the error messages you're seeing suggest that the mongors alias is not recognized inside the MongoDB nodes.

When you initiate a MongoDB replica set, the hostnames (or aliases) you use are stored in MongoDB's replica set configuration. MongoDB uses these hostnames (or aliases) for internode communication within the replica set.

If you use an alias such as mongors, and this alias is correctly defined in the /etc/hosts file inside the Docker container (which should be the case if you are using the alias feature in GitLab CI/CD), then MongoDB should be able to use this alias for internode communication.

(Note: Alias support for the Kubernetes executor was introduced in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later.)

However, it is important to note that this depends on the alias being correctly defined inside the Docker container, and on MongoDB being able to resolve this alias to an IP address. If there is any issue with the alias resolution, this could cause the problems you are seeing.

Also, if your MongoDB nodes are running in separate Docker containers (as is often the case in a replica set), you will need to ensure that the alias is defined in the /etc/hosts file of each container, and that each container can resolve this alias to the correct IP address. If you are using Docker's default bridge network, this should happen automatically. But if you are using a custom network, you might need to add the alias manually using Docker's --add-host option.

---

Looking at your .gitlab-ci.yml:

In GitLab CI/CD, jobs within the same stage are executed in parallel, but jobs in different stages are executed sequentially.

In your pipeline, both jobs test_mongo and test_mongors are in the same stage, named test. So, by default, they would be executed in parallel.

But once test_mongo has run, its mongodb standalone instance would be no more.

Each job in a GitLab CI/CD pipeline runs in its own isolated environment. This means that the services defined in a job, like a MongoDB instance in your case, only exist while the job is running. Once the job completes, its environment, including any services that were started, is torn down.

So in your pipeline, the MongoDB instance started in the test_mongo job will not exist anymore when the test_mongors job starts.
If the test_mongors job needs to interact with the MongoDB instance from the test_mongo job, it won't be able to because that instance will have been shut down.

I tried and add the necessary sleeps for test_mongors to test access while test_mongo is still running, but it fails.

---

Using your second gitlab-ci.yml, I tried adding a mongosh --host $HOST --port 27017 --eval "db.stats()".
I got:

Accessing MongoDB instance at port 27017
Current Mongosh Log ID: 6476008e02ae6412b3aafec3
Connecting to:      mongodb://mongors:27017/?directConnection=true&appName=mongosh+1.9.1
Using MongoDB:      6.0.5
Using Mongosh:      1.9.1
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.
------
   The server generated these startup warnings when booting
   2023-05-30T13:54:57.976+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2023-05-30T13:54:57.976+00:00: You are running this process as the root user, which is not recommended
   2023-05-30T13:54:57.976+00:00: vm.max_map_count is too low
------
MongoServerError: node is not in primary or recovering state

At this stage, I would update the Docker image in order to enable the HTTP interface: you would need to start MongoDB with the --httpinterface option or net.http.enabled: true option in a configuration file.
To enable the RESTful interface, you would need to start MongoDB with the --rest option or net.http.RESTInterfaceEnabled: true option in a configuration file.

You would then be able to use curl or wget to make HTTP requests to MongoDB.

curl http://mongors:28017/serverStatus

I would check over 120 seconds how that status evolves (every 10 seconds) to try and understand the database cluster status.

---

From the comments:

It works when executed using a GitLab runner "on sasw-linux". But it was failing when executed with a GitLab Runner "on green-1.shared.runners-manager.gitlab.com/default.

• using a group GitLab runner that has a docker executor with a default docker image of dotnet core SDK will work
• using a "shared.runners-manager.gitlab.com/default" will fail

There was some issue recently with shared runner, like # 30944: unix socket errors when NETWORK_PER_BUILD is enabled on the Shared Runners. While it might not be related to the issue at hand, it illustrates a lack of stability on that particular type of runner.

Using one's own runner manually registered seems safer.