[BUG]: pybind11_add_module incorrectly always uses system python for venv

[QuantuMope]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.13.6

Problem description

I'v encountered a bug concerning pybind11_add_module where the module is always created using the system python rather than the found python folders.

These examples are on Ubuntu 22.04 where the system python version is 3.10. My cmake version is 3.31.2.
I've confirmed that I run across similar issues on different machines as well, some of which are Ubuntu 24.04.

Using a minimal example minimal_example.cpp

#include <pybind11/pybind11.h>

int do_something() {
    return 42;
}

PYBIND11_MODULE(min_example, m) {
    m.def("do_something", &do_something);
}

with a simple CMakeLists.txt

cmake_minimum_required(VERSION 3.10)
project(pybind11_error)

set(CMAKE_CXX_FLAGS "-O3 -DNDEBUG -Wno-deprecated-declarations -fPIC")
set(CMAKE_CXX_STANDARD 17)

# Use Python from venv if one is active
if(DEFINED ENV{VIRTUAL_ENV})
    set(Python3_EXECUTABLE $ENV{VIRTUAL_ENV}/bin/python)
    message(STATUS "Using Python from: $ENV{VIRTUAL_ENV}")
endif()

# Find required packages
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
find_package(pybind11 REQUIRED)

pybind11_add_module(min_example
    min_example.cpp
)

I then create a fresh venv that matches my system python version and pip install pybind11[global]. Creating min_example and running it works perfectly.

(venv310) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ cmake ..
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using Python from: /home/asjchoi/Desktop/pybind11_error/venv310
-- Found Python3: /home/asjchoi/Desktop/pybind11_error/venv310/bin/python (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/asjchoi/Desktop/pybind11_error/venv310/include (found version "2.13.6")
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/asjchoi/Desktop/pybind11_error/build
(venv310) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ make -j16
[ 50%] Building CXX object CMakeFiles/min_example.dir/min_example.cpp.o
[100%] Linking CXX shared module min_example.cpython-310-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
[100%] Built target min_example
(venv310) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ python
Python 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import min_example
>>> min_example.do_something()
42
>>> 

Now let's say I create a venv with a different python version (3.12) and pip install pybind11[global]. Observe below that the proper python3 and pybind11 files are being found within the venv. Even the .so file is properly named as min_example.cpython-312-x86_64-linux-gnu.so, but you can see upon importing, that the system python was used.

((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ cmake ..
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using Python from: /home/asjchoi/Desktop/pybind11_error/venv312
-- Found Python3: /home/asjchoi/Desktop/pybind11_error/venv312/bin/python (found version "3.12.10") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/asjchoi/Desktop/pybind11_error/venv312/include (found version "2.13.6")
-- Configuring done (0.4s)
-- Generating done (0.0s)
-- Build files have been written to: /home/asjchoi/Desktop/pybind11_error/build
((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ make -j16
[ 50%] Building CXX object CMakeFiles/min_example.dir/min_example.cpp.o
[100%] Linking CXX shared module min_example.cpython-312-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
[100%] Built target min_example
((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ python
Python 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import min_example
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: Python version mismatch: module was compiled for Python 3.10, but the interpreter version is incompatible: 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0].
>>> 

For cmake build systems, I've gotten around this by replacing pybind11_add_module with

# Pybind module
add_library(min_example MODULE min_example.cpp)
target_include_directories(min_example PRIVATE
    ${Python3_INCLUDE_DIRS}
    ${pybind11_INCLUDE_DIRS}
)
target_link_libraries(min_example PRIVATE
    pybind11::headers
)

# Set module properties
set_target_properties(min_example PROPERTIES
    PREFIX ""
    SUFFIX "${PYTHON_MODULE_EXTENSION}"
)

If I rebuild using the python3.12 venv, everything works as expected. The same works for 3.10, just not shown.

((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ cmake ..
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using Python from: /home/asjchoi/Desktop/pybind11_error/venv312
-- Found Python3: /home/asjchoi/Desktop/pybind11_error/venv312/bin/python (found version "3.12.10") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/asjchoi/Desktop/pybind11_error/venv312/include (found version "2.13.6")
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/asjchoi/Desktop/pybind11_error/build
((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ make -j16
[ 50%] Building CXX object CMakeFiles/min_example.dir/min_example.cpp.o
[100%] Linking CXX shared module min_example.cpython-312-x86_64-linux-gnu.so
[100%] Built target min_example
((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ python3
Python 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import min_example
>>> min_example.do_something()
42
>>> 

Regardless of having a working solution, it'd be much more convenient if we could reliably use pybind11_add_module. Furthermore, I'm now encountering the same python version conflict issue when using torch.utils.cpp_extension while inside a non-system python matching venv. For this, I'm forced to use a same python version venv, which is not ideal. Below is the same error from torch:

  File "/home/asjchoi/venvs/horizon3.12-venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1380, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/home/asjchoi/venvs/horizon3.12-venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 1823, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/asjchoi/venvs/horizon3.12-venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py", line 2245, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: Python version mismatch: module was compiled for Python 3.10, but the interpreter version is incompatible: 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0].
>>> 

Again, I've confirmed that this is an issue on

• Ubuntu 22.04 (3.10 default) with venvs for
  • python3.9
  • python3.12
• Ubuntu 24.04 (3.12 default) with venvs for
  • python3.10

Reproducible example code

Already provided in description

Is this a regression? Put the last known working version here if it is.

Not a regression

[cmoineau]

Hello @QuantuMope,

I just got the same problem, following the comment found on this thread: #2661 (comment)

I set -DPYBIND11_FINDPYTHON=ON and it solved my issue. I hope it can help you too.

Note: This issue seems to have appeared when we moved from PyBind 2.10.4 to 2.13.6 due to cmake deprecation issues.

Cheers,

Cyril

[QuantuMope]

Hello @cmoineau

Unfortunately, setting that option does not seem to help. Were you able to import properly using the above example on your machine?

((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ cmake -DPYBIND11_FINDPYTHON=on ..
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using Python from: /home/asjchoi/Desktop/pybind11_error/venv312
-- Found Python3: /home/asjchoi/Desktop/pybind11_error/venv312/bin/python (found version "3.12.10") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/asjchoi/Desktop/pybind11_error/venv312/include (found version "2.13.6")
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/asjchoi/Desktop/pybind11_error/build
((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ make -j16
[ 50%] Building CXX object CMakeFiles/min_example.dir/min_example.cpp.o
[100%] Linking CXX shared module min_example.cpython-312-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
[100%] Built target min_example
((venv312) ) asjchoi@quantumopehordt:~/Desktop/pybind11_error/build $ python3
Python 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import min_example
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: Python version mismatch: module was compiled for Python 3.10, but the interpreter version is incompatible: 3.12.10 (main, Apr  9 2025, 08:55:05) [GCC 11.4.0].
>>> 

[cmoineau]

> cmake ..
-- Found pybind11: /usr/include (found version "2.11.1")
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /test_pybind/build
> make
[ 50%] Building CXX object CMakeFiles/min_example.dir/min_example.cpp.o
[100%] Linking CXX shared module min_example.cpython-312-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
[100%] Built target min_example
> python
Python 3.12.6 | packaged by conda-forge | (main, Sep 22 2024, 14:16:49) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import min_example
>>> min_example.do_something()
42

Your minimal example work for me (I have also tried with PyBind 2.13.6).

Note: I am using conda env instead of venv which may interract differently, when I had your issue I was using CIBuildWheel to compile.

[QuantuMope]

I see. Then this problem may be specifically for venv. I've adjusted the issue title to reflect this.

[BacktoMars]

The behavior of FindPython3 regarding venv is actually defined here:
https://cmake.org/cmake/help/latest/module/FindPython3.html#module:FindPython3

Set the hint Python3_FIND_VIRTUALENV to FIRST and your example should work.

[QuantuMope]

@BacktoMars thanks for your reply, though that documentation states that the hint Python3_FIND_VIRTUALENV is set to FIRST by default. Just to check, I reran my minimal example setting the flag explicitly and still encounter the version mismatch error.