Why is Spark detecting 8 cores, when I only have 4?

Question

I have a Apache Spark 1.6.1 standalone cluster set on a single machine with the following specifications:

I set nothing so Spark can take the default values, which for cores is "all the available cores", based on that, the question is:

do you have hyperthreading enabled? this is a feature that is available on your processor and basically allows each core to act as two. That would be my first guess. Try toggling hyperthreading off and on in the bios and see if the number of cores changes. — Rob S., Jun 04 '16 at 16:23
Thanks! The hyperthreading topic helped too as complement to the answer given below. — User2130, Jun 04 '16 at 20:55

score 1 · Accepted Answer · answered Jun 04 '16 at 16:27

1

I assume that setting all available cores means that Spark is also using Virtual cores

And since your CPU does support Hyperthreading it has 8 virtual cores available.

If you want to only use physical cores I assume there is a specific setting for that.

answered Jun 04 '16 at 16:27

SilverWarior

1 Answers1