0

I created a new docker network on my AWS EC2 instance by

docker network create testnet

I have the following docker-compose:

version: '2'

services:
  mongodb:
    image: mongo:3
    container_name: mongodb
    environment:
      - MONGO_DATA_DIR=/data/db
      - MONGO_LOG_DIR=/dev/null
    volumes:
      - mongodb_data_db:/data/db
    ports:
      - 27017:27017
    command: mongod --smallfiles --logpath=/dev/null --replSet rs0 # --quiet

volumes:
  mongodb_data_db:

networks:
  default:
    external:
      name: testnet

A second container running on the same network is trying to connect to 'mongodb' using this docker-compose:

version: "2"
services:
  monstache:
    image: rwynn/monstache
    container_name: monstache
    command: -mongo-url=mongodb -elasticsearch-url=http://elasticsearch:9200 -direct-read-namespace=db.heartbeat -direct-read-split-max=2

networks:
  default:
    external:
      name: testnet

It worked since the last time AWS decided to reboot my instance. After that, I had to restart all containers again, but since then I get an error message from the monstache container saying:

Unable to connect to MongoDB using URL MongoDB: timed out after 15 seconds

Which means that I somehow can not access to the MongoDB container anymore. Also, another container in the network can not connect to 'mongodb' anymore, so I don´t think it´s only a problem with the 'monstache' container. It seems like something has changed in my docker system generally. At least when I run. 

docker network inspect testnet

I can see that all containers are listed.

What I have done so far:

  • Restarted docker service (many times)
  • Manually removed and created the docker network (also with a new name)
  • docker system prune

Really need help because I am stuck now for 2 days trying to solve this :(

UPDATE: docker-compose.yml for Elasticsearch (and Kibana)

version: "2"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
    container_name: elasticsearch
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
  kibana:
    image: docker.elastic.co/kibana/kibana:6.4.1
    container_name: kibana
    depends_on:
      - elasticsearch
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
    ports:
      - 5601:5601

volumes:
  elasticsearch:
    driver: local

networks:
  default:
    external:
      name: testnet

UPDATE: docker network inspect testnet

:~$ docker network inspect testnet
[
  {
    "Name": "testnet",
    "Id": "448018003d92c8802dd701931e21da018618abce360a147808a5c6b4b51f4b6d",
    "Created": "2018-10-08T12:40:10.163231318Z",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
      "Driver": "default",
      "Options": {},
      "Config": [
        {
          "Subnet": "172.18.0.0/16",
          "Gateway": "172.18.0.1"
        }
      ]
    },
    "Internal": false,
    "Attachable": false,
    "Containers": {
      "34df14ddf4ef004115b6e66b35177356a7c0c5e5d0d94d2c05406aa61cd1d744": {
        "Name": "kibana",
        "EndpointID": "bb38deafbd1929d268ba55c8fb28064d9b0afe7bbfb95289a6893ca62f91ff8b",
        "MacAddress": "02:42:ac:12:00:03",
        "IPv4Address": "172.18.0.3/16",
        "IPv6Address": ""
      },
      "95034d04c4f6c07527f725436a84b20a1514d8aaf70d4e19c54344eb07c7632f": {
        "Name": "elasticsearch",
        "EndpointID": "269e42333b20dd01152f58329c87060059471a8ea68e3cd97cb45c502b102879",
        "MacAddress": "02:42:ac:12:00:02",
        "IPv4Address": "172.18.0.2/16",
        "IPv6Address": ""
      },
      "c3153881f2a8925bb74718afa9b33c5e9cfcc10f58b2fa7a5157e45b83bea343": {
        "Name": "mongodb",
        "EndpointID": "44c6ac5755897c056d7285eba83a0934e1871b6c2ca671cbbe846fc55e23ff3e",
        "MacAddress": "02:42:ac:12:00:04",
        "IPv4Address": "172.18.0.4/16",
        "IPv6Address": ""
      }
    },
    "Options": {},
    "Labels": {}
  }
]
Ikar Pohorský
  • 4,617
  • 6
  • 39
  • 56
tonyskulk
  • 11
  • 7
  • Are you sure mongodb container is running already? – Mani Oct 08 '18 at 14:56
  • Yes, all I can see that all containers are running with docker ps -a – tonyskulk Oct 08 '18 at 15:13
  • So you already tried running mongo container first and then monstache container manually. In these cases they are working fine right? – Mani Oct 08 '18 at 15:26
  • Unfortunately it´s not working even when starting each container manually. Actually there is also an elasticsearch container running on the same network. Monstache is going to synchronize mongodb data to elasticsearch indices, so it has to know 'mongodb' and 'elasticsearch' as hosts. But the 'mongodb' is somehow not connectable. Anyway, the setup is: mongodb + elasticsearch running as their own containers. And after that I start monstache container – tonyskulk Oct 08 '18 at 15:51
  • Can u provide Docker compose for elastic search? And if I understand ur question properly. All the containers are starting properly but none of them are accessible through service discovery. Am I right? – Mani Oct 08 '18 at 16:06
  • Can you provide more information on the networking setup? VPC/subnets, etc...? – maxm Oct 08 '18 at 16:23
  • @Mani Added docker-compose.yml for elasticsearch. – tonyskulk Oct 08 '18 at 16:24
  • @maxm Added docker network inspect – tonyskulk Oct 08 '18 at 16:31
  • @Mani All containers are starting property, yes. mongodb is the only one that can not be accessed through service discovery. – tonyskulk Oct 08 '18 at 16:34
  • check mongo.conf is it allowing external trafic, check ur securyt group, check mapping port and finaly connect with mongo from localhost will lead these step to actual issue – Adiii Oct 08 '18 at 16:44
  • seems monstache is not connected to the network. As said by Adii check the mongo.conf to allow traffic from remote hosts and other security policies. – Mani Oct 08 '18 at 17:00
  • I have checked the aws security group and port exposition / mapping. I can connect to mongodb from localhost and also from external ip. I have started _mongod_ with _--bind_ip=0.0.0.0_ and also got no effect. Is there another way to allow external traffic? I´m a bit lost, because with all those settings it worked well before the reboot – tonyskulk Oct 09 '18 at 13:25

2 Answers2

1

I have fixed my problem.

Since I am starting mongod with 

--replSet rs0

after the reboot somehow I had to re-instantiate the mongodb replica set. After using

mongo --eval "rs.initiate()"

on the mongodb container I was able to connect from other containers by 'mongodb' container service url.

That´s also the reason why it happened after the reboot. It seems that I have to re-initiate the replicat set always after rebooting occurres. Actually I thought that this should be conserved in the mongodb volume directory, so rebooting would not affect this... But seems I was wrong :)

Thank you all for your time.

tonyskulk
  • 11
  • 7
0

Try this, order matters, basically starting fresh... (copy and past all at once)

docker-compose kill

docker-compose down

docker rm $(docker ps -aq)

docker network rm $(docker network ls)

docker volume rm $(docker volume ls --format {{.Name}})

docker-compose up --force-recreate

If possible you should avoid using ip-addresses for replicas.

Hostnames

Use a logical DNS hostname instead of an ip address, particularly when configuring replica set members or sharded cluster members. The use of logical DNS hostnames avoids configuration changes due to ip address changes.

More info --> replica-set-architectures

You shouldn't need to initiate after a reboot, unless all nodes are shutdown at the same time. I'm assuming this is what happened if all containers are running on 1 instance.

If you leave one up, and replSet is set in the config file, others will rejoin automagically.

For High availability, consider running the Replica Set in Swarm. Here is nice walkthrough of setting up a replica set using swarm.

D. Vinson
  • 1,090
  • 6
  • 8