Docker registry on marathon insecure-registry

Question

I have installed mesosphere in an AWS cluster with one master and two nodes, call them master, node1 and node2. The master is visible, the nodes are private.

I get a marathon instance running by default and docker support on the back. Docker version is 1.6 (verified).

So I installed a docker registry image using marathon as described in Marathon docs, with a few variants for S3 support, like this:

  {
      "id": "/docker/registry",
      "instances": 1,
      "cpus": 0.5,
      "mem": 1024.0,
      "disk": 128,
      "container": {
        "docker": {
          "type": "DOCKER",
          "image": "registry:2",
          "network": "BRIDGE",
          "portMappings": [
            {
              "containerPort": 5000,
              "hostPort": 0,
              "protocol": "tcp",
              "servicePort": 5000
            }
          ]
        },
        "volumes": []
      },
      "env": {
        "APPLICATION_WEB_PROXY_BASE": "/service/docker_registry",
        "SETTINGS_FLAVOR": "s3",
        "REGISTRY_VERSION": "0.1",
        "REGISTRY_LOG_LEVEL": "debug",
        "REGISTRY_STORAGE": "s3",
        "REGISTRY_STORAGE_S3_ACCESSKEY": "ACCESSKEY",
        "REGISTRY_STORAGE_S3_SECRETKEY": "SECRETKEY",
        "REGISTRY_STORAGE_S3_REGION": "us-east-1",
        "REGISTRY_STORAGE_S3_BUCKET": "my-docker-registry",
        "REGISTRY_STORAGE_S3_ROOTDIRECTORY": "/docker",
        "AWS_PATH": "/docker",
        "STORAGE_PATH": "/docker"
      },
      "ports": [ 0 ]
    }

The docker registry service starts in one of the nodes. Say... node1.amazonaws.com:23456. Note the node and port is randomly selected by marathon. Nothing guarantees the same node and port will be picked again if I restart and I don't know which node and port will be picked beforehand.

So the registry works and I can use its rest API (from an ssh line in the master) as: wget node1.amazonaws.com:23456/v2 and I get the proper json response {}.

But then I would like to do a push to the server and I get this kind of message

"FATA[0000] Error response from daemon: v1 ping attempt failed with error:
Get https://myregistrydomain.com:5000/v1/_ping: tls: oversized record received with length 20527. 
If this private registry supports only HTTP or HTTPS with an unknown CA certificate,please add 
`--insecure-registry myregistrydomain.com:5000` to the daemon's arguments.
In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag;
simply place the CA certificate at /etc/docker/certs.d/myregistrydomain.com:5000/ca.crt"

So I guess I need to set the insecure-registry flag. or setup a CA cert. As I don't need a cert at this point I just need to make it insecure. But then the flag has to be set as --insecure-registry myregistrydomain.com:5000.

But then you get the problem, the host and port are dynamically set. So I can't put this into a config file anywhere (and I don't know where anyway a this uses CoreOS and there is no /etc/default/docker file).

I tried to passing it into the Marathon JSON App. I recall it is possible to use the $HOST and $PORT env variables so I tried adding them as parameters in the container as:

...
container:
"network": "BRIDGE",
"parameters": [{ "key": "insecure-registry", "value": "$HOST:$PORT" }],
"portMappings": [
...

But the service just won't start. Then I tried setting it in the args:

...
  "disk": 128,
  "args": ["--insecure-registry $HOST:$PORT"],
  "container": {
...

But I got a message saying:

Executor registered on slave 20150928-002726-2617573386-5050-1278-S1 flag provided but not defined: -insecure-registry

So I tried setting a DOCKER_OPTS env variable as:

  "env": {
    "DOCKER_OPTS": "--insecure-registry $HOST:$PORT",
    "APPLICATION_WEB_PROXY_BASE": "/service/docker_registry",

But it just ignores the flag.

In other words, I have no idea where to make it insecure.

Any ideas would be greatly appreciated.

Thanks!

Answer 1

I think this question is wrong in concept. I was trying to add the --insecure-registry flag to the registry when it has to be added to the docker daemon that will access the registry. In the case of DCOS it has to be added to each one of the slave nodes. By Michael suggestion, I think it can be solved by locking the registry port and using the mesos-dns to resolve the slave host name, so for example I can use the DNS name dockerregistry.marathon.mesos that will resolve to the proper server and the locked port 5000, i.e:

--insecure-registry dockerregistry.marathon.mesos:5000

Of course this will restrict me to only run one instance in one server at any time but that is all ok, I only need one instance. The important thing is that when a new instance starts, it points to the proper S3 bucket.

So, the problem is now how do I restart the docker daemon running in CoreOS across the whole cluster adding this flag, but I think that is another question for another time.

Answer 2

I believe this is answered in DCOS documentation

Copy and paste from the example linked above:

Run this in all agent nodes of your cluster:

$ sudo tee /etc/systemd/system/docker.service.d/override.conf  <<-'EOF'
[Service]
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// $OPTIONS \
     $DOCKER_STORAGE_OPTIONS \
     $DOCKER_NETWORK_OPTIONS \
     $BLOCK_REGISTRY \
     $INSECURE_REGISTRY \
     --storage-driver=overlay \
     --insecure-registry registry.marathon.l4lb.thisdcos.directory:5000 
EOF

systemctl daemon-reload
systemctl restart docker

Answer 3

You can force the deployment on certain nodes via constraints in your Marathon app specification, such as shown in the following example:

...
"constraints": [
    [
        "hostname",
        "LIKE",
        "node1.amazonaws.com"
    ]
]
...