1

I have setup the mesos and marathon on my local system. Also, I have docker engine running on my system, and when I do, sudo docker images, I get the following, 

REPOSITORY                                                 TAG                 IMAGE ID            CREATED             SIZE
myrepo/hello                                               latest              b7ce0084dbaf        4 weeks ago         330.3 MB
fluent/new                                                 latest              adc4b7b4b0eb        4 weeks ago         1.589 GB
<none>                                                     <none>              3a87799875ed        4 weeks ago         1.589 GB
<none>                                                     <none>              16a573cd3b85        4 weeks ago         330.3 MB
769348183957.dkr.ecr.us-east-1.amazonaws.com/ruby          2.1.10              77bf121d484e        8 weeks ago         1.535 GB
769348183957.dkr.ecr.us-east-1.amazonaws.com/centos-base   7                   9ab68a0dd16a        10 weeks ago        330.3 MB
hello-world                                                latest              c54a2cc56cbb        12 weeks ago        1.848 kB
docker/whalesay                                            latest              6b362a9f73eb        16 months ago       247 MB

Now, through the Marathon UI, I am trying to make an application, and after going doing all the steps in the documentation at https://mesosphere.github.io/marathon/docs/native-docker-private-registry.html, when I finally create the application, it's status varies between Deploying and Waiting intermittently and finally shows delayed. I never get it in the running stage. I try to figure out the logs on my machine but I am not able to make sense of them seeing their volume.

My JSON config for the app was as follows: 

{
  "id": "/123",
  "cmd": "sudo docker run -itd 9ab68a0dd16a /bin/bash",
  "cpus": 1,
  "mem": 128,
  "disk": 200,
  "instances": 1,
  "container": {
    "docker": {
      "image": "769348183957.dkr.ecr.us-east-1.amazonaws.com/centos-base",
      "network": "HOST",
      "forcePullImage": true
    },
    "type": "DOCKER"
  }
}

Am I missing anything? AFAIK, what should happen is the application should automatically pull this image from my local repo and finally get deployed on the machine. Should I also create a task for my application because I haven't made any task specifically? If yes, won't I have to first create an application and then add tasks to it?

Also, I restarted the mesos slave like this: sudo ./bin/mesos-slave.sh --master=127.0.0.1:5050 --work_dir=/var/lib/mesos --containerizers=docker,mesos --executor_registration_timeout=5mins

Also, I tried making a tar.gz file of the docker repo on my localhost and copied it to /etc and passed the URI: file///etc/docker.repo.tar.gz

Where am I going wrong? In the marathon logs, I am able to see this None of the enabled containerizers (mesos) could create a container for the provided TaskInfo/ExecutorInfo message. Hence, I think this is where I am going wrong.

Edit: Marathon logs for the Application suggested: 

[2016-09-26 12:29:30,418] INFO Task launch for 'task [nginx.c59c7403-83b6-11e6-a834-0a0027000000]' was accepted. 0 tasksToLaunch, 0 in flight, 1 confirmed.  not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-8)
I0926 12:29:30.418325 90841088 master.cpp:3104] Processing ACCEPT call for offers: [ d165ac5e-93dc-4b7b-bf36-071ce75aa44d-O17 ] on slave d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0 at slave(1)@172.26.35.124:63837 (172.26.35.124) for framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 (marathon) at scheduler-c27b723e-e9d3-428a-b7d3-c8c184b1ed7c@172.26.35.124:63837
I0926 12:29:30.419334 90841088 master.hpp:177] Adding task nginx.c59c7403-83b6-11e6-a834-0a0027000000 with resources cpus(*):0.1; mem(*):256; ports(*):[31835-31835] on slave d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0 (172.26.35.124)
I0926 12:29:30.419400 90841088 master.cpp:3589] Launching task nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 (marathon) at scheduler-c27b723e-e9d3-428a-b7d3-c8c184b1ed7c@172.26.35.124:63837 with resources cpus(*):0.1; mem(*):256; ports(*):[31835-31835] on slave d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0 at slave(1)@172.26.35.124:63837 (172.26.35.124)
I0926 12:29:30.419661 89767936 slave.cpp:1361] Got assigned task nginx.c59c7403-83b6-11e6-a834-0a0027000000 for framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000
I0926 12:29:30.421689 90304512 gc.cpp:83] Unscheduling '/tmp/mesos/0/slaves/d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0/frameworks/fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000' from gc
I0926 12:29:30.422006 89767936 gc.cpp:83] Unscheduling '/tmp/mesos/0/meta/slaves/d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0/frameworks/fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000' from gc
I0926 12:29:30.422173 88158208 slave.cpp:1480] Launching task nginx.c59c7403-83b6-11e6-a834-0a0027000000 for framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000
I0926 12:29:30.422904 88158208 paths.cpp:528] Trying to chown '/tmp/mesos/0/slaves/d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0/frameworks/fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000/executors/nginx.c59c7403-83b6-11e6-a834-0a0027000000/runs/0548c84c-40ad-40cd-bbd8-b1330e66f348' to user 'bhjain'
I0926 12:29:30.445124 88158208 slave.cpp:5352] Launching executor nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/0/slaves/d165ac5e-93dc-4b7b-bf36-071ce75aa44d-S0/frameworks/fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000/executors/nginx.c59c7403-83b6-11e6-a834-0a0027000000/runs/0548c84c-40ad-40cd-bbd8-b1330e66f348'
I0926 12:29:30.446513 88158208 slave.cpp:1698] Queuing task 'nginx.c59c7403-83b6-11e6-a834-0a0027000000' for executor 'nginx.c59c7403-83b6-11e6-a834-0a0027000000' of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000
E0926 12:29:30.446702 88158208 slave.cpp:3784] Container '0548c84c-40ad-40cd-bbd8-b1330e66f348' for executor 'nginx.c59c7403-83b6-11e6-a834-0a0027000000' of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 failed to start: **None of the enabled containerizers (mesos) could create a container for the provided TaskInfo/ExecutorInfo message**
E0926 12:29:30.446846 90841088 slave.cpp:3855] Termination of executor 'nginx.c59c7403-83b6-11e6-a834-0a0027000000' of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 failed: Unknown container: 0548c84c-40ad-40cd-bbd8-b1330e66f348
I0926 12:29:30.453965 90841088 slave.cpp:3012] Handling status update TASK_FAILED (UUID: 90717092-98c0-4eab-9967-f43e005159b5) for task nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 from @0.0.0.0:0
W0926 12:29:30.454391 91914240 containerizer.cpp:1295] Ignoring update for unknown container: 0548c84c-40ad-40cd-bbd8-b1330e66f348
I0926 12:29:30.454927 91377664 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 90717092-98c0-4eab-9967-f43e005159b5) for task nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000
I0926 12:29:30.455751 91377664 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FAILED (UUID: 90717092-98c0-4eab-9967-f43e005159b5) for task nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000
I0926 12:29:30.456254 90304512 slave.cpp:3410] Forwarding the update TASK_FAILED (UUID: 90717092-98c0-4eab-9967-f43e005159b5) for task nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2-8f1c-c24057bcfb58-0000 to master@172.26.35.124:63837
I0926 12:29:30.456456 91377664 master.cpp:4763] Status update TASK_FAILED (UUID: 90717092-98c0-4eab-9967-f43e005159b5) for task nginx.c59c7403-83b6-11e6-a834-0a0027000000 of framework fa8c0ef7-651b-41c2
John Dui
  • 541
  • 3
  • 8
  • 15
  • 1
    Can you please fix the app definition JSON in this question? Also, are you using a base Docker image? You need an application that runs in the foreground.. What happens if you run a standard nginx image (from Docker Hub)? – Tobi Sep 26 '16 at 06:38
  • @Tobi, oh, my bad, I will do it. – John Dui Sep 26 '16 at 06:41
  • I meant pasting a "real" JSON... – Tobi Sep 26 '16 at 06:43
  • @Tobi, is the JSON still not ok? I haven't tried nginx, but AFAIK, when I try with the centos-image, atleast it should pull that image. Instead it shows me error and terminates the task. I can post the marathon logs if that would help. – John Dui Sep 26 '16 at 06:45

1 Answers1

0

The JSON seems fine from a JSON schema perspective. TBH, the cmd property contents doesn't really make sense if you want to test running containers on Mesos.

I even think the command you're using will not be able to work, executing Docker within the application context...

Please use a standard image like nginx like this:

{
  "id": "nginx",
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "nginx",
      "network": "BRIDGE",
      "portMappings": [
        { "hostPort": 0, "containerPort": 80 }
      ],
      "forcePullImage":true
    }
  },
  "instances": 1,
  "cpus": 0.1,
  "mem": 256,
  "healthChecks": [{
      "protocol": "HTTP",
      "path": "/",
      "portIndex": 0,
      "timeoutSeconds": 10,
      "gracePeriodSeconds": 10,
      "intervalSeconds": 2,
      "maxConsecutiveFailures": 10
  }]
}

to check the overall functionality. If this works, tag and push the nginx image to your local registry, and try using it via the uris credential passing method and the local image name.

Tobi
  • 31,405
  • 8
  • 58
  • 90
  • I still face the same problem when I use this JSON. Check my edit in the original post. Do you think it might be a problem with the config? – John Dui Sep 26 '16 at 07:02
  • Can you manually launch a Docker container on the slave? The `**None of the enabled containerizers (mesos) could create a container for the provided TaskInfo/ExecutorInfo message**` error hints to problems launching containers via Docker – Tobi Sep 26 '16 at 07:29
  • since the slave is running on my local itself, yes, when I run an image, I am able to run the container on the machine which is running the mesos slave. – John Dui Sep 26 '16 at 07:32
  • I ran the mesos slave with --containerizers=docker,mesos flag. Is there anything else that I should do as well? – John Dui Sep 26 '16 at 07:34
  • I am stuck on this issue for quite long. Can you please help me out? It will be a very very big help for me. Thanks! – John Dui Sep 26 '16 at 07:45
  • I think you're having a problem with the `docker` containerizer, because it's not listed in the error message. Also, I wonder why that the task is launched with `resources cpus(*):0.1; mem(*):32` instead of 256mb. What version of Docker are you using? – Tobi Sep 26 '16 at 07:57
  • I am using V1.12.0 of docker. Also, is there any way I can debug my way out of this? Thanks! – John Dui Sep 26 '16 at 08:23
  • Afaik DC/OS is not supporting Docker 1.12, for compatibility reasons. Do you have the possibility to downgrade to 1.11? And on what OS are you on? – Tobi Sep 26 '16 at 08:41
  • I can do that if that will work. I am using Mac OS X El Capitan. Also, I wasn't really able to get my head around DC/OS but, when I downloaded Mesos from this `http://mesos.apache.org/gettingstarted/`, (using `wget http://www.apache.org/dist/mesos/0.28.2/mesos-0.28.2.tar.gz`, will I be setting it up with DC/OS? Thanks! – John Dui Sep 26 '16 at 08:58
  • Nope, but as you're running this on MacOS I hardly doubt that this is even supported... IMO you'd have to compile Mesos on you Mac, not using the Linux versions. And Docker on Mac is a whole topic for itself. See: http://mesos.apache.org/gettingstarted/ – Tobi Sep 26 '16 at 09:00
  • To be honest, I'd recommend using https://github.com/mesoshq/mesosctl together with a Vagrant-based cluster, e.g. https://github.com/mesoshq/vagrant-cluster-centos – Tobi Sep 26 '16 at 09:03
  • http://mesos.apache.org/gettingstarted/ mentions the packages for OS-X Yosemite and El-Capitan. So I guess it should be supported. I have worked with Docker for Mac before, so I will be able to do that. Also, thanks for the advice for an alternate to Marathon but since I am trying to mimic the actual production system on my local, I can't switch to another GUI. – John Dui Sep 26 '16 at 09:08
  • It mentions the downloading and compiling of the sources, not "packages" in the sense of rpms or debs. So, did you **compile** Mesos on your Mac? I'm using MacOS as well, and see nothing but problems with the *native* Docker. I can't really help you any further. Also, `mesosctl` is not an alternate to Marathon. I can bootstrap Mesos installations on 1 to n nodes... Marathon is a framework/scheduler running on Mesos. – Tobi Sep 26 '16 at 09:12
  • I didn't really compile it. But, I did follow these steps: `# Change working directory. $ cd mesos # Bootstrap (Only required if building from git repository). $ ./bootstrap # Configure and build. $ mkdir build $ cd build $ ../configure $ make`. Is this what you are talking about? If not, how can I start with compilation? Is there any documentation for that? And which system do you recommend for this? This will probably be my last question, so thanks :) – John Dui Sep 26 '16 at 09:17
  • Well, with that you build Mesos on you host. But you apparently didn't run a `make install`. I somehow don't see you argument concerning to mimic the production. I guess that isn't run on MacOS, and probably not on one host. IMO, using a vagrant based setup is much cleaner, and you even have the choice to use the same OS as "production". This would be much closer to production than what you're currently doing – Tobi Sep 26 '16 at 09:22
  • I did run make install but didn't write it here. That makes sense. With vagrant based install, is there any documentation for that? – John Dui Sep 26 '16 at 09:29
  • Not sure what you mean with "documentation". In the projects is linked there's an inline documentation. Setting up Vagrant/VirutalBox is documented on the respective sites – Tobi Sep 26 '16 at 09:34