MongoDB refuses to backup

Question

I have a MongoDB instance inside a docker container to run some local services and use mongodump to perform backups. Recently this backup process, and applications running on top, have broken because MongoDB no longer thinks its the master. I need the (up-to-date) backup contents to migrate the services to a new server. How can I get MongoDB to dump its database contents so I can redeploy the instance?

This is the command I run to perform a backup and the error message:

    mongodump --host "db" --port 27017 --db rocketchat                    
2019-05-17T11:28:59.950+0000    Failed: error counting rocketchat.rocketchat_apps_persistence: node is not in primary or recovering state

I've tried running a few variations of the mongodump command including treating the server as a secondary instance:

 mongodump --host "db" --port 27017 --readPreference secondaryPreferred
2019-05-17T11:16:23.066+0000    Failed: error counting admin.system.version: node is not in primary or recovering state

When I try to initiate the replicaset for the server, it already thinks it is initialized.

rs0:OTHER> rs.initiate()
{
    "info2" : "no configuration specified. Using a default configuration for the set",
    "me" : "1d70c5ced11b:27017",
    "info" : "try querying local.system.replset to see current configuration",
    "ok" : 0,
    "errmsg" : "already initialized",
    "code" : 23,
    "codeName" : "AlreadyInitialized",
    "operationTime" : Timestamp(1558063935, 1),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1558063935, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}

Rsyncing the contents of the docker volume that contains the WiredTiger data unfortunately also doesn't' work. I end up with the same problem where the database errors out thinking that its not a master. Sadly this also means some low level backups (on top of the mongodump backups) also don't work.

Setup:

• Single MongoDB instance as a docker-compose service.
• Replicaset "rs0" for various databases with oplog.

Edit:

Result of rs.status() and rs.config()

rs0:OTHER> rs.status()
{
    "operationTime" : Timestamp(1558063935, 1),
    "ok" : 0,
    "errmsg" : "Our replica set config is invalid or we are not a member of it",
    "code" : 93,
    "codeName" : "InvalidReplicaSetConfig",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1558063935, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}

rs0:OTHER> rs.config()
{
    "_id" : "rs0",
    "version" : 1,
    "protocolVersion" : NumberLong(1),
    "writeConcernMajorityJournalDefault" : true,
    "members" : [
        {
            "_id" : 0,
            "host" : "8d6d6fda7542:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatIntervalMillis" : 2000,
        "heartbeatTimeoutSecs" : 10,
        "electionTimeoutMillis" : 10000,
        "catchUpTimeoutMillis" : -1,
        "catchUpTakeoverDelayMillis" : 30000,
        "getLastErrorModes" : {

        },
        "getLastErrorDefaults" : {
            "w" : 1,
            "wtimeout" : 0
        },
        "replicaSetId" : ObjectId("5ccfbd9f66a45fbddfa5c0f0")
    }
}