OpenBLAS becomes single-threaded

[tjoli]

Hi everybody,

I am using standard lapack routines to diagonalize moderately large matrices with openBLAS.
The box is a xeon with 6 cores and hyperthreading. At runtime execution starts with 12 threads
by default and after some reproducible time goes down to one thread with miserable performance
but execution is without errors. I have observed the same behavior with OMP_NUM_THREADS=2,4,6
and using also OPENBLAS_NUM_THREADS. I tried to set OMP_DYNAMIC=FALSE with no change of behavior. It is a locally compiled version of openBLAS 0.3.3 with gcc/gfortran 7.3.0

any hints would be appreciated !

[martin-frbg]

Weird. Does matrix size change by the time you notice the number of active threads going down ? Many BLAS drivers now fall back to using a single thread when the overhead of multithreading is expected to exceed the gain from parallelizing, but this is always local to the individual function. Also most LAPACK functions are copied unmodified from the netlib reference implementation, so the LAPACK itself is not multithreaded.
Offhand I can only think of "something in your code" tying all threads to the same core via sched_setaffinity. Some minimal code sample that reproduces the problem would be great, but it would probably already help to know if you see this same behaviour with older versions (e.g 0.3.0, 0.2.20)

[tjoli]

In fact it happens during treatment of a single matrix. I noticed the same behavior with 0.2.20 hence I upgraded. The inference of single-thread behavior comes from looking at output of "top"
also I used repeatedly the same code on a local computer center and it gives full speed with expected performance.

added:
I checked that a standard openMP code runs all the time with the correct number of threads, by standard I mean without calling any openBLAS.

[martin-frbg]

So what are the dimensions of that particular matrix, and which function is it that appears to execute single-threaded ? (Could be a missed or misguided optimization in OpenBLAS, but if you already saw this with 0.2.20 at least it is not a recent regression.) I presume your "standard openMP code" will be using some other implementation of BLAS (like MKL or ATLAS), or is it avoiding BLAS completely ?

[tjoli]

I checked matrices of sizes 1800 and 3800.

The problem appears with lapack routine ZHEEV:
if I want all eigenvectors (so ZHEEV('V'....)) then it slows down to one thread while if I want eigenvalues only ZHEEV now with argument
ZHEEV('N"...) it runs full speed i.e. all threads remain active all the time. I understand that it is more costly to get the eigenvectors but why should this reduce the threads to unity ?

the standard openMP code I referred to uses no BLAS at all, just to be sure.

[martin-frbg]

For that one needs to look at what ZHEEV in netlib LAPACK does - at first glance it uses DSTERF if only eigenvalues are requested, and ZSTEQR when both eigenvalues and eigenvectors. I have not yet traced the BLAS calls (if any) in the two call trees. (Does your standard code avoid LAPACK calls as well, or which LAPACK does it link to when not using OpenBLAS ?)

[tjoli]

my "standard" code uses no LAPACK stuff at all. Just filling matrices, a dumb case as a test.
it is not a test of any LAPACK/BLAS stuff.

[tjoli]

Calling LAPACK DSTEQR leads to the same problem when I want all eigenvectors of a real matrix.
It turns single-threaded after a while. Tested on a order=5600 matrix

[martin-frbg]

I now remember that we have #1560 as a related issue, and from what I managed to find out there it was the call to ZLASR in the eigenvectors case that was the bottleneck.

[brada4]

... which more or less comes from unnecessary parallelism for small samples given to blas functions ....
@tjoli can you run
perf record sample
perf report
With and without OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 set
And post the reports, esp. if any BLAS function (as opposed to lapack) consumes relatively lots more of time in parallel case.

[tjoli]

so perf record without any settings NUM_THREADS set

30.43%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zlasr_
19.90%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zhemv_U
17.16%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_kernel_l
16.65%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_incopy
 8.06%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_kernel_r
 5.22%  lapack.exe  [kernel.vmlinux]                [k] entry_SYSCALL_64_fastpath
 0.40%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemv_kernel_4x4
 0.21%  lapack.exe  libc-2.26.so                    [.] pthread_mutex_lock
 0.20%  lapack.exe  [kernel.vmlinux]                [k] native_irq_return_iret
 0.19%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemv_kernel_4x4
 0.18%  lapack.exe  libpthread-2.26.so              [.] __pthread_mutex_lock
 0.16%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_itcopy
 0.16%  lapack.exe  libpthread-2.26.so              [.] __pthread_mutex_unlock_usercnt
 0.10%  lapack.exe  libc-2.26.so                    [.] __sched_yield
 0.09%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] blas_thread_server
 0.08%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zswap_k
 0.06%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dlartg_
 0.05%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zscal_k
 0.04%  lapack.exe  lapack.exe                      [.] __hamiltoniantorus_MOD_hamfermitorus

[tjoli]

now with OPENBLAS_NUM_THREADS=1 and OMP_NUM_THREADS=1

81.34%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zlasr_
 7.77%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_kernel_r
 5.46%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zhemv_U
 3.19%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_kernel_l
 0.38%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_incopy
 0.30%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemv_kernel_4x4
 0.18%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemv_kernel_4x4
 0.17%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dlartg_
 0.15%  lapack.exe  [kernel.vmlinux]                [k] native_irq_return_iret
 0.07%  lapack.exe  lapack.exe                      [.] __hamiltoniantorus_MOD_hamfermitorus
 0.05%  lapack.exe  lapack.exe                      [.] __toolz_MOD_laguerre
 0.05%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zsteqr_
 0.05%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] lsame_
 0.04%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] __powidf2
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_itcopy
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dlamch_
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zaxpy_k
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] ztrmm_kernel_RC
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] ztrmm_kernel_RR
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] ztrmm_kernel_RN
 0.03%  lapack.exe  lapack.exe                      [.] __toolz_MOD_search_8
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemv_kernel_4x1
 0.03%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zlarfb_
 0.02%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zcopy_k
 0.02%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zswap_k
 0.02%  lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] zgemm_otcopy
 0.02%  lapack.exe  lapack.exe                      [.] __fockfermitorus_MOD_fermitorusspace

[brada4]

zhemv has no ththreading threshold
@tjoli - do you see execution time halved with single thread?

[martin-frbg]

Execution times would complete the picture, but I doubt they will be markedly different (note that the first message has "goes down to one thread with miserable performance"). Once the program enters the non-parallelized netlib ZLASR all the work is done by a single core, no BLAS involved.

[brada4]

I am into thinking zlasr work is same for both cases, just that it takes 30 or 80% depending on zhemv ?waste

[tjoli]

similar behavior for DSTEQR first without any settings:

28.86%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dlasr_
22.70%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dgemm_kernel
18.92%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dgemm_incopy
17.56%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dsymv_kernel_4x4
 7.62%  su2lapack.exe  [kernel.vmlinux]                [k] entry_SYSCALL_64_fastpath
 0.44%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dgemm_itcopy
 0.40%  su2lapack.exe  libpthread-2.26.so              [.] __pthread_mutex_lock
 0.30%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dgemv_kernel_4x4
 0.26%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dlartg_
 0.26%  su2lapack.exe  libpthread-2.26.so              [.] __pthread_mutex_unlock_usercnt
 0.24%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dgemv_t
 0.24%  su2lapack.exe  su2lapack.exe                   [.] __su2hamiltonianspherelong_MOD_hamfermisphere
 0.22%  su2lapack.exe  [kernel.vmlinux]                [k] native_irq_return_iret
 0.16%  su2lapack.exe  libopenblas_nehalemp-r0.3.3.so  [.] dsymv_U
 0.15%  su2lapack.exe  libc-2.26.so                    [.] __sched_yield