Dynamic option for blas_set_num_threads()

[hiccup7]

I have been using Python with MKL BLAS, which defaults to dynamically setting the number of threads used by each BLAS function. See:
https://software.intel.com/en-us/node/528546
https://software.intel.com/en-us/node/528547

In Julia, I would expect comparable speed from OpenBLAS by calling blas_set_num_threads(CPU_CORES), but I would get a cold slap in the face. I learned that OpenBLAS will automatically use one thread for small arrays, but otherwise use exactly the number of threads specified by blas_set_num_threads(). In other words, my understanding is that OpenBLAS doesn't have MKL BLAS' dynamic option.

As an example, on my Haswell CPU, where Julia reports CPU_CORES as 8, OpenBLAS' dgemv() function (from the develop branch) runs fastest with blas_set_num_threads(2). It is not practical or realistic for me to put blas_set_num_threads() before each (hidden) call to a BLAS function.

For an Intel CPU with hyperthreading, better OpenBLAS performance would occur by using the number of physical cores instead of logical cores. For example, blas_set_num_threads(CPU_CORES >> 1) For portability, I suggest Julia includes a new constant: CPU_PHYSICAL_CORES.

I realize that there is already an effort to provide MKL BLAS as a build and shipping option for Julia (JuliaLang/julia#10969). Assuming that OpenBLAS won't go away, however, it would be helpful if Julia provided a layer of abstraction to make OpenBLAS as performant and easy to use as MKL BLAS, and to make Julia code portable between builds with either one.

Specifically, I suggest that blas_set_num_threads(-1) causes Julia to use a dynamic number of threads. When built with MKL BLAS, this would cause MKL BLAS to effectively act like MKL_DYNAMIC is True. When built with OpenBLAS, a lookup table for each BLAS function would determine the maximum number of threads to use. There would be a different lookup table for each CPU architecture, such as Intel Haswell or Intel SandyBridge. Preferably, the lookup table would be in source code so that it could be tuned by each user. I would expect a lot of pull requests by the community for several months on the lookup tables for various CPUs, but I believe they would appreciate the speedup.

[sturlamolden]

The analysis here is wrong. You generally get best performance by using the number of logical cores. That is what MKL does as well.

[sturlamolden]

See here what happens when we go from 4 threads (number of physical cores) to 8 threads (number of logical cores):

http://nbviewer.ipython.org/urls/dl.dropboxusercontent.com/u/12464039/kdtree-jan21-simd.ipynb

Why? Because if you only have half as many threads as logical cores, each thread has a 50 % chance of being scheduled to an idle core.

[stevengj]

cc: @xianyi ... it really seems like this should be filed as an OpenBLAS issue.

[tknopp]

yes I also do not see why this is a Julia issue. This would also be a very local optimization that would certainly fail in situations, where thread pools are fighting with each other.

[tkelman]

related: JuliaLang/julia#5991

[ScottPJones]

I also would be very concerned about this being done as a fixed amount, or with only local knowledge.
I saw some other issues with overloading a machine, using @async. If you have hundreds of Julia processes running, and they are all setting the number of threads to use to the number of virtual processors, I think you'll get worse performance than simply using 1 thread per process.

[hiccup7]

While MKL BLAS only needs to optimize the number threads for Intel CPUs, OpenBLAS also supports AMD, MIPS and ARM CPUs. It is a big task for OpenBLAS to implement a dynamic option with all the CPU architectures (and all the BLAS functions) it supports. Also, OpenBLAS has a very small team of volunteer contributors. Therefore, OpenBLAS has wisely chosen to push the responsibility of optimizing the number threads up to parent (language).

BLAS functions dominate my speed experience of a scientific programming language. If Julia starts supporting MKL BLAS on Windows, then, for me, Julia begins to be competitive with Python, which is already has free distributions with MKL BLAS. But if Julia remains OpenBLAS-only on Windows, and the Julia team points the finger at OpenBLAS for lack of a dynamic threading option, then the Julia language becomes a non-competitive no-go for me. For Julia users without an Intel CPU, an OpenBLAS dynamic threading option is the only way I see to make Julia scientific language both speed-competitive and usable.

My opening post is an attempt to provide a technical solution, taking into account the large number of Julia contributors & users vs. the tiny team of OpenBLAS contributors. My suggestion of using a Julia source file is designed to empower all Julia users. Compare this to a solution that is only testable by contributors with the ability to build OpenBLAS from source. In other words, I propose that we will more likely get success by leveraging a large number of unskilled users than to rely on a tiny team of highly-skilled developers (who don't have time for this big task).

For those who point out the possibility of the dynamic option providing non-optimum performance in some cases, I would stress that this is a proposal for an option. I am in favor of still providing the present means of specifying the number of threads used by OpenBLAS.

[tkelman]

@hiccup7 Julia is also a volunteer project with a relatively small number of contributors. The suggestions are fine, but the changes you want to see are much more likely to happen sooner if you start rolling up your sleeves and contributing code. You don't need to write pages of arguments to convince us that this would be a good idea, you need to make our lives easier by doing some of the work yourself. We'd rather review pull requests than read long issue threads that state facts we are already well aware of.

But if Julia remains OpenBLAS-only on Windows

As I've already said multiple times, it isn't. You can use any library you have access to via ccall. Unless you can resolve licensing questions and help get a binary build of Julia that uses MKL to pass its tests (and as I've said elsewhere, this is an especially difficult task on Windows), you can do this today, and it would be a much more productive way to spend your time.

[tknopp]

Furthermore I don't see the actual issue. One can simply compile Julia with MKL and use it. Why is it so important that Julia comes distributed with MKL? We do see very very rarely discussions where the BLAS performance has been an issue for a user.

[nalimilan]

@hiccup7 You seem to start with the premise that it would be easier to write this in Julia than in OpenBLAS, which I don't think is really true. Indeed, OpenBLAS developers are a small team a highly skilled people, but implementing the kind of crude performance tuning that you describe should not require a deep understanding of the hand-optimized kernels for all architectures. Somebody able to implement this in Julia would probably be able to do this in OpenBLAS too.

[hiccup7]

@tkelman , using ccall to access MKL BLAS functions instead of having Julia use MKL BLAS under the hood for linear algebra functions causes Julia to no longer be a high-level language. I wouldn't take a backwards step from Python in ease of code development and readability.

@tknopp , BLAS performance dominates the language performance for DSP code like mine. Also, compare the price of $29 for MKL with the Anaconda distribution of Python to $1199 for an MKL license from Intel for C++ and FORTRAN to support Julia builds. For hobby Windows and OSX users, the choice is clear!

@nalimilan , Compare the number of persons with the skill and time to change pure-Julia source files (all Julia users have the skill to use a text editor) to those with the skill, time and disk space to build OpenBLAS from source. I am trying to improve the convenience and odds of community support.