MongoDB cluster. Replicas stuck in stateStr: 'STARTUP'

Question

I've read a lot of different threads about this issue, here and at other sites, and I cannot find a solution so far.

Im running three servers at Azure, one at each zone. Server 1 is Windows, 2 and 3 (replicas) are Linux. This is not a production but a "learning" environment.

I've set this content for all three files at mongod.conf:

net:
  port: 27017
  bindIp: 0.0.0.0

security:
      authorization: enabled
      keyFile: /var/lib/mongodb/mongo.key
    
    replication:
      replSetName: "lab"

And they all share the same key.

I also created the same "root" user on each server:

{
        user: "mongodevuser",
        pwd: "ThisIsAPassword!@#123#@!",
        roles: [{ role: "root", db: "admin" }],
        authenticationRestrictions: [
            {
                clientSource: ["xx.xx.xx.xx", "127.0.0.1"]
            }
        ]
    }

And I initiated rs and added the two server IPs.

The access to the key file was setup right (I went thru sudo chown, chmod 400, etc) till I figured the right way to do this.

But now, I have both linux servers in stuck in stateStr: 'STARTUP' when I rs.status() at the "main" windows server.

Now, from the Windows server, I was able to mongosh to both linux servers, and vice versa, so it is not a firewall issue

Here is my rs.status()

members: [
    {
      _id: 0,
      name: 'mongo:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 10111,
      optime: { ts: Timestamp({ t: 1677620041, i: 1 }), t: Long("2") },
      optimeDate: ISODate("2023-02-28T21:34:01.000Z"),
      lastAppliedWallTime: ISODate("2023-02-28T21:34:01.216Z"),
      lastDurableWallTime: ISODate("2023-02-28T21:34:01.216Z"),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1677609950, i: 1 }),
      electionDate: ISODate("2023-02-28T18:45:50.000Z"),
      configVersion: 3,
      configTerm: 2,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'xx.xx.xx.xx:27017',
      health: 1,
      state: 0,
      stateStr: 'STARTUP',
      uptime: 8791,
      optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDate: ISODate("1970-01-01T00:00:00.000Z"),
      optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),
      lastAppliedWallTime: ISODate("1970-01-01T00:00:00.000Z"),
      lastDurableWallTime: ISODate("1970-01-01T00:00:00.000Z"),
      lastHeartbeat: ISODate("2023-02-28T21:34:04.912Z"),
      lastHeartbeatRecv: ISODate("1970-01-01T00:00:00.000Z"),
      pingMs: Long("36"),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: -2,
      configTerm: -1
    },
    {
      _id: 2,
      name: 'xx.xx.xx.xx:27017',
      health: 1,
      state: 0,
      stateStr: 'STARTUP',
      uptime: 9296,
      optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long("-1") },
      optimeDate: ISODate("1970-01-01T00:00:00.000Z"),
      optimeDurableDate: ISODate("1970-01-01T00:00:00.000Z"),
      lastAppliedWallTime: ISODate("1970-01-01T00:00:00.000Z"),
      lastDurableWallTime: ISODate("1970-01-01T00:00:00.000Z"),
      lastHeartbeat: ISODate("2023-02-28T21:34:03.423Z"),
      lastHeartbeatRecv: ISODate("1970-01-01T00:00:00.000Z"),
      pingMs: Long("32"),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: -2,
      configTerm: -1
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1677620041, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("c4080375be71381d41b04860817041b7b064a3c3", "hex"), 0),
      keyId: Long("7205078939239186438")
    }
  },
  operationTime: Timestamp({ t: 1677620041, i: 1 })
}

I've hide the IPs above, which are Public ones. I tried restarting the services, the servers, and I always get:

MongoServerError: node is not in primary or recovering state

At mongosh on the replicated servers.

And this is the error I get at the log files

"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.

and

Failed to reap transaction table","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}

Thanks.

You are in a "learning" environment but using **public** IP-Addresses? Maybe start with `localhost`,especially for security reasons. — Wernfried Domscheit, Mar 01 '23 at 08:35
The company is paying for it and want me to do it that way, it's not my call. Specially to setup the firewalls right, etc. It goes beyond Mongo. — Danielle, Mar 01 '23 at 14:35
For production/public accessible MongoDB you should also secure connections with TLS/SSL and also use x.509 certificates for membership authentication rather than `keyFile` as recommended by MongoDB — Wernfried Domscheit, Mar 01 '23 at 15:07

Danielle · Answer 1 · 2023-03-01T06:52:53.400

0

I usually do not answer my questions, but I was dealing with this the entire day until I found the solution and this may help someone. Admins, please consider if it should be deleted or kept.

The problem I had was related to the hosts file.

I called the servers Mongo1, Mongo2 and Mongo3, but at the hosts file, I was using the internet IP in all of them, so I pointed each one to the local address on each hosts file as follows.

Mongo 1:

127.0.0.1 Mongo1
20.xx.xx.xx Mongo2
20.xx.xx.xx Mongo3

Mongo 2

20.xx.xx.xx Mongo1
127.0.0.1 Mongo2
20.xxx.xx.xx Mongo3

Mongo3

20.xx.xx.xx Mongo1
20.xx.xx.xx Mongo2
127.0.0.1 Mongo3

Then, I remarked everything related to replication at mongod.conf, and deleted the local DB.

I started over the replication and added each instance with the names at the hosts file: Mongo1, Mongo2 and Mongo3 and everything worked perfect and instantly.

I hope this helps someone.

edited Mar 01 '23 at 06:52

answered Mar 01 '23 at 05:37

Danielle

1,386
8
22

1

According to MongoDB documentation (e.g. [Considerations When Deploying a Replica Set](https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set/#considerations-when-deploying-a-replica-set)) you should prefer hostnames in configuration rather than IP-Addresses. Looks like you fall exactly into this trap. – Wernfried Domscheit Mar 01 '23 at 08:33
Exactly. I was watching several videos that were using IP addresses instead, but on a local environment, not over the internet. – Danielle Mar 01 '23 at 14:36
What they dont way, which they are right to cause its common sense and that's where I failed, is that the hosts for each local should point to 127.0.0.1 at /etc/hosts – Danielle Mar 01 '23 at 14:39
I think "I called the servers Mongo1, Mongo2 and Mongo3" it totally wrong. Use the hostname as returned by `hostname` or `hostname -s` command. – Wernfried Domscheit Mar 01 '23 at 15:04
That is what I did Wernfried, hostname returns Mongo1, Mongo2 and Mongo3. The cluster replicates perfect, but Im unable to connect thru node, but that's another story Im trying to solve. – Danielle Mar 01 '23 at 15:26
I also found out Azure provides you with a static IP address, but when it comes to the IP utilized by the server, it is an internal IP, like 10.0.0.4 which I believe is being a problem for me. – Danielle Mar 01 '23 at 15:28

MongoDB cluster. Replicas stuck in stateStr: 'STARTUP'

1 Answers1