0

I've been trying to configure Apache Ignite on DC/OS (1.8.7) marathon using the official docs at http://apacheignite.gridgain.org/docs/mesos-deployment but short of some hacks I haven't been able to get it to work following the docs. One of the core reasons appear to be that the cmd 

"cmd": "java -jar ignite-mesos-1.8.0.jar"

will through an error "sh: java: command not found". This would indicate that java is not in the path but on the marathon hosts I've validated that java is in fact accessible on the path for my regular user at least.

I suspect that somehow java needs to be added to the path of mesos-container that is trying to run the cmd but I've been unable to find any documentation on how to set the path or default environment variables (ignite-mesos spawns tasks that need JAVA_HOME set as well, which is also missing in the tasks) in the containers that get created. For reference my marathon.json file is below...

{
  "id": "/ignition",
  "cmd": "java -jar ignite-mesos-1.8.0.jar",
  "args": null,
  "user": null,
  "env": {
    "IGNITE_MEMORY_PER_NODE": "2048",
    "IGNITE_NODE_COUNT": "3",
    "IGNITE_VERSION": "1.8.0",
    "MESOS_MASTER_URL": "zk://master.mesos:2181/mesos",
    "IGNITE_RUN_CPU_PER_NODE": "0.1"
  },
  "instances": 0,
  "cpus": 0.25,
  "mem": 2048,
  "disk": 0,
  "gpus": 0,
  "executor": null,
  "constraints": null,
  "fetch": [
    {
      "uri": "http://SERVER_HERE/ignite-mesos-1.8.0.jar"
    }
  ],
  "storeUrls": null,
  "backoffSeconds": 1,
  "backoffFactor": 1.15,
  "maxLaunchDelaySeconds": 3600,
  "container": null,
  "healthChecks": null,
  "readinessChecks": null,
  "dependencies": null,
  "upgradeStrategy": {
    "minimumHealthCapacity": 1,
    "maximumOverCapacity": 1
  },
  "labels": {
    "HAPROXY_GROUP": "external"
  },
  "acceptedResourceRoles": null,
  "ipAddress": null,
  "residency": null,
  "secrets": null,
  "taskKillGracePeriodSeconds": null,
  "portDefinitions": [
    {
      "protocol": "tcp",
      "port": 10108
    }
  ],
  "requirePorts": false
}
Dance machine
  • 70
  • 5
  • Try one (or both) of adding `PATH` to the `env`, or use the full path to the `java` executable. – Elliott Frisch Jan 10 '17 at 01:00
  • Also `$JAVA_HOME\bin` variable should be enlisted in `PATH` – Nikolay Tikhonov Jan 10 '17 at 09:53
  • I've checked and java is on the path as well as JAVA_HOME is set (on the root account as well which I believe is what Marathon runs as) – Dance machine Jan 10 '17 at 17:58
  • I'm thinking this may be a case where the mesos containerizer has changed to be more isolated than in the past when the Ignite documentation has been written and environment variables that were on the agent are no long being passed into the container (which seems like correct behaviour) – Dance machine Jan 10 '17 at 18:06

1 Answers1

1

Ignite seems to expect a JDK 1.7/1.8 installation on each agent node, and the JAVA_HOME environment variable set accordingly.

Unfortunately, the Mesos framework doesn't seem to be well-maintained, as it still uses Mesos 0.22 libraries.

Tobi
  • 31,405
  • 8
  • 58
  • 90
  • each agent node does appear to have JAVA_HOME set. when you say the Mesos framework isn't maintained well do you mean the Ignite implementation specifically? – Dance machine Jan 10 '17 at 17:59
  • Yes, the Ignite framework... It's using a quite old Mesos library version. What I meant is that you have to apparently install a JDK on each agent manually – Tobi Jan 11 '17 at 06:10
  • I'm not 100% sure if this would even work with newer Mesos versions, as you pointed out in the comment above – Tobi Jan 11 '17 at 10:47
  • I've just worked this out. Each agent machine did have java installed and was available on the PATH of the correct users. However when running inside of the mesos containerizer it uses a different path (it's possible to see this by creating a task that just runs 'export' and looking at it) after making sure the relevant java binaries were where the mesos container was expecting I was able to get past this roadblock. – Dance machine Jan 19 '17 at 18:20