2

I would really like to add the following settings to our spark-defaults.conf using OpsCenter 6.0.2 in order to avoid configuration drift. Is there a way to add these config items to the config profile template?

spark.cores.max 4
spark.driver.memory 2g
spark.executor.memory 4g
spark.python.worker.memory 2g
fcnorman
  • 1,154
  • 9
  • 19

1 Answers1

1

NOTE: As Mike Lococo has pointed out in the comments for this answer -- this answer may work to update the config profile values but will not result in those values being written to spark-defaults.conf.

The following is not a solution!

You can; you have to update the config profile via the LCM Config Profile API (https://docs.datastax.com/en/opscenter/6.0/api/docs/lcm_config_profile.html#lcm-config-profile).

First, identify the config profile that needs updating:

$ curl http://localhost:8888/api/v1/lcm/config_profiles

Get the href for the specific config profile that needs updating, request it, and save the response body to a file:

$ curl http://localhost:8888/api/v1/lcm/config_profiles/026fe8e3-0bb8-49c1-9888-8187b1624375 > profile.json

Now, in the profile.json file you just saved to, you add or edit the key at json > spark-defaults-conf to include the following keys: 

"spark-defaults-conf": {
  "spark-cores-max": 4,
  "spark-python-worker-memory": "2g",
  "spark-ssl-enabled": false,
  "spark-drivers-memory": "2g",
  "spark-executor-memory": "4g" 
}

Save the updated profile.json. Finally, execute an HTTP PUT to the same config profile URL, using the edited file as the request data:

$ curl -X PUT http://localhost:8888/api/v1/lcm/config_profiles/026fe8e3-0bb8-49c1-9888-8187b1624375 -d @profile.json
elijah
  • 2,904
  • 1
  • 17
  • 21
  • Lifecycle Manager developer here. I'm skeptical that this does what you want. This will add the fields to the config-profile and they'll get echoed back via the API, but it WON'T write them to your target nodes. spark-defaults.conf is template-based and ignores unsupported keys. You can see the list of possible keys in /etc/opscenterd/definitions/spark-defaults-conf/dse/spark-defaults-conf-dse-5.0.2.template I've filed an improvement request under OPSC-10290 to add support for these keys and to consider the possibility of looping through all provided keys so we can set all possible options. – Mike Lococo Sep 12 '16 at 15:23