CUDA Setup failed despite GPU being available.

[WangStone1]

System Info

$ nvidia-smi
Mon Apr 28 15:30:49 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA L20 On | 00000000:45:00.0 Off | 0 |
| N/A 37C P0 88W / 350W | 6158MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA L20 On | 00000000:58:00.0 Off | 0 |
| N/A 39C P0 92W / 350W | 290MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA L20 On | 00000000:C0:00.0 Off | 0 |
| N/A 39C P0 87W / 350W | 290MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA L20 On | 00000000:D4:00.0 Off | 0 |
| N/A 36C P0 86W / 350W | 290MiB / 46068MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 4180 C python 6152MiB |
| 1 N/A N/A 96486 C python 284MiB |
| 2 N/A N/A 98525 C python 284MiB |
| 3 N/A N/A 98732 C python 284MiB |
+---------------------------------------------------------------------------------------+

$ python --version
Python 3.11.9
(comfyui_prd)

$ pip show torch
Name: torch
Version: 2.3.1
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /data/conda/envs/comfyui_prd/lib/python3.11/site-packages
Requires: filelock, fsspec, jinja2, networkx, nvidia-cublas-cu12, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-cufft-cu12, nvidia-curand-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, nvidia-nccl-cu12, nvidia-nvtx-cu12, sympy, triton, typing-extensions
Required-by: accelerate, clip-interrogator, compel, deepspeed, facexlib, fairscale, kornia, onnx2torch, open-clip-torch, peft, pytorch-lightning, rf-clip, simple-lama-inpainting, spandrel, timm, tomesd, torchaudio, torchdiffeq, torchmetrics, torchsde, torchvision, transparent-background, ultralytics, ultralytics-thop, xformers
(comfyui_prd)

$ pip show bitsandbytes
Name: bitsandbytes
Version: 0.42.0
Summary: k-bit optimizers and matrix multiplication routines.
Home-page: https://github.com/TimDettmers/bitsandbytes
Author: Tim Dettmers
Author-email: dettmers@cs.washington.edu
License: MIT
Location: /data/conda/envs/comfyui_prd/lib/python3.11/site-packages
Requires: scipy
Required-by:
(comfyui_prd)

Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-693.el7.x86_64

conda

Reproduction

python -m bitsandbytes

warn(msg)

/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: /data/conda/envs/comfyui_prd did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: /usr/local/cuda-12.2/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
The following directories listed in your path were found to be non-existent: {PosixPath('//hf-mirror.com'), PosixPath('https')}
The following directories listed in your path were found to be non-existent: {PosixPath('%M'), PosixPath('\"$command\"";echo $msg >>$HISTORY_FILE;logger -p local6.notice -t history "$msg";export HISTORY HISTFILE HISTSAVE HISTZONE HISTORY HISTLOG;export HISTFILE=$HOME/.bash_history; export HISTSIZE=1000; export HISTFILESIZE=1000; export HISTTIMEFORMAT="%F %T $(whoami)'), PosixPath('\"$curuser\",\"ppid\"'), PosixPath('\"$SSH_TTY\",\"pwd\"'), PosixPath(' ";}'), PosixPath('\"$loginuser\",\"curuser\"'), PosixPath('\"$loginip\",\"tty\"'), PosixPath('\"$PWD\",\"command\"'), PosixPath('%S");whoStr=(who -u am i);ip=$(hostname -I);loginip=$(echo $SSH_CLIENT | { read x y; echo $x; });if [ ! -n "$loginip" ];then tmpip=${whoStr[6]};tmpip=${tmpip/"("/};loginip=${tmpip/")"/};fi;loginuser=${whoStr[0]};curuser=$USER;if [ ! -n "$curuser" ];then curuser=$(id -un);fi;msg="\"curtime\"'), PosixPath('{ command=$(history 1 | { read a b c d e; echo $e; });command=${command/\"/%22 };curtime=$(date "+%Y-%m-%d %H'), PosixPath('\"$curtime\",\"loginuser\"'), PosixPath('\"$ip\",\"loginip\"'), PosixPath('\"$PPID\",\"ip\"')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: 8.9.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary /data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda121.so)
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=121
python setup.py install
Traceback (most recent call last):
File "", line 189, in _run_module_as_main
File "", line 148, in _get_module_details
File "", line 112, in _get_module_details
File "/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/init.py", line 6, in
from . import cuda_setup, utils, research
File "/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/research/init.py", line 1, in
from . import nn
File "/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/research/nn/init.py", line 1, in
from .modules import LinearFP8Mixed, LinearFP8Global
File "/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/research/nn/modules.py", line 8, in
from bitsandbytes.optim import GlobalOptimManager
File "/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/optim/init.py", line 6, in
from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/cextension.py", line 20, in
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

(comfyui_prd)

but i run in another same PC successfull，the conda enviroment、hardware、OS and cuda is same with this ， another PC output like this:

$ python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++ ANACONDA CUDA PATHS ++++++++++++++++++++
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/torch/lib/libtorch_cuda_linalg.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/torch/lib/libc10_cuda.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda110.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda110_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda111.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda114.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda114_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda115.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda115_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda117_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda120.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda120_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda121_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda122.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda122_nocublaslt.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda123.so
/data/conda/envs/comfyui_prd/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda123_nocublaslt.so

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudart.so
/usr/local/cuda-12.2/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda-12.2/nsight-compute-2023.2.0/target/linux-desktop-glibc_2_11_3-x64/libcuda-injection.so
/usr/local/cuda-12.2/nsight-compute-2023.2.0/target/linux-desktop-glibc_2_19_0-ppc64le/libcuda-injection.so
/usr/local/cuda-12.2/nsight-compute-2023.2.0/target/linux-desktop-t210-a64/libcuda-injection.so

+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++
/home/ipa_sudo/miniforge3/pkgs/icu-75.1-he02047a_0/lib/libicudata.so
/home/ipa_sudo/miniforge3/lib/libicudata.so

++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
++++++++++ /usr/local/cuda-12.2/lib64 CUDA PATHS +++++++++++

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['8.9', '8.9', '8.9', '8.9']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
+ library is importable
+ CUDA function is callable

WARNING: Please be sure to sanitize sensible info from any such env vars!

i don't know why got the different result

Expected behavior

please help me to solve the problem，i tried a lot of methods but not soleved yet, i can't find the reason

[WangStone1]

i unistall bitsandbytes and install bitsandbytes==0.42.0 again,the error changed as this：
$ python -m bitsandbytes
Could not find the bitsandbytes CUDA binary at PosixPath('/data1/prd/bitsandbytes/bitsandbytes/libbitsandbytes_cuda121.so')
Could not load bitsandbytes native library: /data1/prd/bitsandbytes/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "/data1/prd/bitsandbytes/bitsandbytes/cextension.py", line 87, in
lib = get_native_library()
^^^^^^^^^^^^^^^^^^^^
File "/data1/prd/bitsandbytes/bitsandbytes/cextension.py", line 74, in get_native_library
dll = ct.cdll.LoadLibrary(str(binary_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/conda/envs/comfyui_test/lib/python3.11/ctypes/init.py", line 454, in LoadLibrary
return self._dlltype(name)
^^^^^^^^^^^^^^^^^^^
File "/data/conda/envs/comfyui_test/lib/python3.11/ctypes/init.py", line 376, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /data1/prd/bitsandbytes/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory

CUDA Setup failed despite CUDA being available. Please run the following command to get more information:

python -m bitsandbytes

Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
and open an issue at: https://github.com/bitsandbytes-foundation/bitsandbytes/issues

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: CUDASpecs(highest_compute_capability=(8, 9), cuda_version_string='121', cuda_version_tuple=(12, 1))
PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: (8, 9).
Library not found: /data1/prd/bitsandbytes/bitsandbytes/libbitsandbytes_cuda121.so. Maybe you need to compile it from source?
If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION,
for example, make CUDA_VERSION=113.

The CUDA version for the compile might depend on your conda install, if using conda.
Inspect CUDA version via conda list | grep cuda.
To manually override the PyTorch CUDA version please see: https://github.com/TimDettmers/bitsandbytes/blob/main/docs/source/nonpytorchcuda.mdx
The directory listed in your path is found to be non-existent: //hf-mirror.com
The directory listed in your path is found to be non-existent: { command=$(history 1 | { read a b c d e; echo $e; });command=${command/"/%22 };curtime=$(date "+%Y-%m-%d %H
The directory listed in your path is found to be non-existent: %S");whoStr=(who -u am i);ip=$(hostname -I);loginip=$(echo $SSH_CLIENT | { read x y; echo $x; });if [ ! -n "$loginip" ];then tmpip=${whoStr[6]};tmpip=${tmpip/"("/};loginip=${tmpip/")"/};fi;loginuser=${whoStr[0]};curuser=$USER;if [ ! -n "$curuser" ];then curuser=$(id -un);fi;msg=""curtime"
The directory listed in your path is found to be non-existent: "$command"";echo $msg >>$HISTORY_FILE;logger -p local6.notice -t history "$msg";export HISTORY HISTFILE HISTSAVE HISTZONE HISTORY HISTLOG;export HISTFILE=$HOME/.bash_history; export HISTSIZE=1000; export HISTFILESIZE=1000; export HISTTIMEFORMAT="%F %T $(whoami)
Found duplicate CUDA runtime files (see below).

We select the PyTorch default CUDA runtime, which is 12.1,
but this might mismatch with the CUDA version that is needed for bitsandbytes.
To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variable.

For example, if you want to use the CUDA version 122,
BNB_CUDA_VERSION=122 python ...

OR set the environmental variable in your .bashrc:
export BNB_CUDA_VERSION=122

In the case of a manual override, make sure you set LD_LIBRARY_PATH, e.g.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2,

• Found CUDA runtime at: /usr/local/cuda-12.2/lib64/libcudart.so
• Found CUDA runtime at: /usr/local/cuda-12.2/lib64/libcudart.so.12.2.53
• Found CUDA runtime at: /usr/local/cuda-12.2/lib64/libcudart.so.12
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  ++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  Checking that the library is importable and CUDA is callable...
  Couldn't load the bitsandbytes library, likely due to missing binaries.
  Please ensure bitsandbytes is properly installed.

For source installations, compile the binaries with cmake -DCOMPUTE_BACKEND=cuda -S ..
See the documentation for more details if needed.

Trying a simple check anyway, but this will likely fail...
Traceback (most recent call last):
File "/data1/prd/bitsandbytes/bitsandbytes/diagnostics/main.py", line 66, in main
sanity_check()
File "/data1/prd/bitsandbytes/bitsandbytes/diagnostics/main.py", line 40, in sanity_check
adam.step()
File "/data/conda/envs/comfyui_test/lib/python3.11/site-packages/torch/optim/optimizer.py", line 391, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/conda/envs/comfyui_test/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data1/prd/bitsandbytes/bitsandbytes/optim/optimizer.py", line 291, in step
self.update_step(group, p, gindex, pindex)
File "/data/conda/envs/comfyui_test/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data1/prd/bitsandbytes/bitsandbytes/optim/optimizer.py", line 521, in update_step
F.optimizer_update_32bit(
File "/data1/prd/bitsandbytes/bitsandbytes/functional.py", line 1317, in optimizer_update_32bit
optim_func = str2optimizer32bit[optimizer_name][0]
^^^^^^^^^^^^^^^^^^
NameError: name 'str2optimizer32bit' is not defined
Above we output some debug information.

[Titus-von-Koeller]

is there a reason why you install 0.42? Could you try with the newest version?

[matthewdouglas]

Generally speaking, we cannot support EOL platforms like CentOS 7. The wheels we publish on PyPI currently have a minimum glibc requirement of 2.24.

It's probably possible to build from source, if you choose.

I'm curious about the differences in your environments. Maybe in one of them there's something in LD_LIBRARY_PRELOAD that isn't in the other. If this is an HPC environment, I'd suggest contacting the support staff.

[WangStone1]

is there a reason why you install 0.42? Could you try with the newest version?
I can only install to version 0.42.0 through pip installation

[WangStone1]

Generally speaking, we cannot support EOL platforms like CentOS 7. The wheels we publish on PyPI currently have a minimum glibc requirement of 2.24.

It's probably possible to build from source, if you choose.

I'm curious about the differences in your environments. Maybe in one of them there's something in LD_LIBRARY_PRELOAD that isn't in the other. If this is an HPC environment, I'd suggest contacting the support staff.

The environment is the same. I recreated a clean conda environment, first installing bitsandbytes and dependencies in req.txt, then installing torch2.3.1, then installing my own dependencies, and running Python -m bitsandbytes works normally, so I feel strange

[zhuyumanman]

Generally speaking, we cannot support EOL platforms like CentOS 7. The wheels we publish on PyPI currently have a minimum glibc requirement of 2.24.
It's probably possible to build from source, if you choose.
I'm curious about the differences in your environments. Maybe in one of them there's something in LD_LIBRARY_PRELOAD that isn't in the other. If this is an HPC environment, I'd suggest contacting the support staff.

The environment is the same. I recreated a clean conda environment, first installing bitsandbytes and dependencies in req.txt, then installing torch2.3.1, then installing my own dependencies, and running Python -m bitsandbytes works normally, so I feel strange

Hello, may I ask if you've solved this problem? My technical skills are limited, and I still haven't been able to resolve this issue.