5

Intel strongly recommends using their distribution for Python, instead of manually building Python modules for yourself. An obvious advantage is that there are many optimized modules available from their distribution, a non-trivial task if you want to compile them yourself.

But I haven't find out what kind of optimizations are available. If I would manually build the packages I am certain all the instructions available to my processor (AVX, SSE) where taken into consideration while compiling the code, and making the binaries as optimized as possible, hence I wanted to know:

Does Intel ship the distribution with binaries compiled for all their processors? (e.g. a different set of binaries for each generation: Nehalem, Sandy Bridge, etc) or...

Does Intel ship some sort of generic optimization? (to all their processors) or...

something in the middle?

To be more specific, Intel says it "accelerates" Python by the way of MKL, TBB and DAAL. A manual build will at least take advantage of MKL and customization of instruction sets to the host processor.

So, from my point of view there is a trade-of between these two approaches: if using the distribution, a lot of time would be saved and would have all "accelerators" but manual compiling would take advantage of MKL and instructions sets (at least). So, what is your take?

J. Ramos
  • 63
  • 7

1 Answers1

2

Both are right - most of libs are compiled with multiple ISA targets as AVX2, AVX512 and internal dispatcher select particular codepath in runtime.

Also there is lower level sse2 codepath that would work on any modern x86 systems including AMD CPUs. Depending on dispatcher implementation in particular library particular codepath would be selected for non Intel CPUs as well.

In terms of messaging - it's mixed a bit. From python level there is no expectation that user would be compiling anything specific. But there is additional perf benefit available if application would be ported from Python to C++ - in this case all those libs would be used through C++ APIs

Nikolay Petrov
  • 58
  • 5