0

I have created Datastax Enterprise cluster on AWS using 3 m3.large instances where all 3 nodes are Analytical nodes running Spark.

3 m3.large amazon instances each have 2 cores, so in total 6 cores.

On Spark Master web ui I see only 3 cores available in total -> on the web ui of each of the worker nodes, I see only 1 available core.

What happened with 3 other cores? Is DSE reserving 1 core per machine for Cassandra or some other operations?

Thank you

Srdjan Nikitovic
  • 853
  • 2
  • 9
  • 19

1 Answers1

0

The number of cores available to Spark in DSE is defaulted to (.7 * total_number_of_cores). In your case, this would lead to 1.4 which is rounded down to a single core. You can modify your percentage by editing

/etc/dse/dse.yaml

or  

install_location/resources/dse/conf/dse.yaml

and adjusting

initial_spark_worker_resources: .7

The documentation below also goes into greater detail on how to modify your Spark settings. However, it is important to note that if all of your machine's cores get allocated to Spark, your system will end up context switching and degrade performance.

https://docs.datastax.com/en/datastax_enterprise/4.5/datastax_enterprise/spark/sparkConf.html#sparkConf__sparkCfgMem

peytoncas
  • 755
  • 3
  • 9