Merge pull request #155 from YannickJadoul/manylinux2010-alternative

Adding support for building manylinux2010 wheels - alternative

Author: joerick

README.md

@@ -221,8 +221,8 @@ All being well, you should get wheels delivered to you in a few minutes.
 | **Build parameters** | `CIBW_BUILD_VERBOSITY` | Increase or decrease the output of `pip wheel` |
 | **Build environment** | `CIBW_ENVIRONMENT` | Set environment variables needed during the build |
 |   | `CIBW_BEFORE_BUILD` | Execute a shell command preparing each wheel's build |
-|   | `CIBW_MANYLINUX1_X86_64_IMAGE` | Specify an alternative manylinx1 x86_64 docker image |
-|   | `CIBW_MANYLINUX1_I686_IMAGE` | Specify an alternative manylinux1 i686 docker image |
+|   | `CIBW_MANYLINUX_X86_64_IMAGE` | Specify an alternative manylinux x86_64 docker image |
+|   | `CIBW_MANYLINUX_I686_IMAGE` | Specify an alternative manylinux i686 docker image |
 | **Tests** | `CIBW_TEST_COMMAND` | Execute a shell command to test all built wheels |
 |   | `CIBW_TEST_REQUIRES` | Install Python dependencies before running the tests |
 |   | `CIBW_TEST_EXTRAS` | Install Python dependencies before running the tests using ``extras_require``|
@@ -231,10 +231,10 @@ A more detailed description of the options, the allowed values, and some example
 
 ### Linux builds on Docker
 
-Linux wheels are built in the [`manylinux1` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 513](https://www.python.org/dev/peps/pep-0513/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
+Linux wheels are built in the [`manylinux` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 571](https://www.python.org/dev/peps/pep-0571/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
 - Programs and libraries cannot be installed on the Travis CI Ubuntu host with `apt-get`, but can be installed inside of the Docker image using `yum` or manually. The same goes for environment variables that are potentially needed to customize the wheel building. `cibuildwheel` supports this by providing the `CIBW_ENVIRONMENT` and `CIBW_BEFORE_BUILD` options to setup the build environment inside the running Docker image. See [below](#options) for details on these options.
 - The project directory is mounted in the running Docker instance as `/project`, the output directory for the wheels as `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system.  Note that this is not available on CircleCI due to their Docker policies.
-- Alternative dockers images can be specified with the `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [below](#options) for more details.
+- Alternative dockers images can be specified with the `CIBW_MANYLINUX_X86_64_IMAGE` and `CIBW_MANYLINUX_I686_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [below](#options) for more details.
 
 
 Options
@@ -291,17 +291,19 @@ For `linux` you need Docker running, on Mac or Linux. For `macos`, you need a Ma
 
 Optional.
 
-Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux1_x86_64` or `cp34-macosx_10_6_intel` - you can list specific ones to build and `cibuildwheel` will only build those, and/or list ones to skip and `cibuildwheel` won't try to build them.
+Space-separated list of builds to build and skip. Each build has an identifier like `cp27-manylinux_x86_64` or `cp34-macosx_10_6_intel` - you can list specific ones to build and `cibuildwheel` will only build those, and/or list ones to skip and `cibuildwheel` won't try to build them.
 
 When both options are specified, both conditions are applied and only builds with a tag that matches `CIBW_BUILD` and does not match `CIBW_SKIP` will be built.
 
-The format is `python_tag-platform_tag`. The tags are as defined in [PEP 0425](https://www.python.org/dev/peps/pep-0425/#details).
+The format is `python_tag-platform_tag`. The tags are similar but not identical to the ones defined in [PEP 425](https://www.python.org/dev/peps/pep-0425/#details).
 
 Python tags look like `cp27` `cp34` `cp35` `cp36` `cp37`
 
-Platform tags look like `macosx_10_6_intel` `manylinux1_x86_64` `manylinux1_i686` `win32` `win_amd64`
+Platform tags look like `macosx_10_6_intel` `manylinux_x86_64` `manylinux_i686` `win32` `win_amd64`
 
-You can also use shell-style globbing syntax (as per `fnmatch`) 
+You can also use shell-style globbing syntax (as per `fnmatch`).
+
+The list of supported and currently selected build identifiers can be retrieved by passing the `--print-build-identifiers` flag to `cibuildwheel`.
 
 Examples:
 - Only build on Python 3.6: `CIBW_BUILD`:`cp36-*`
@@ -311,7 +313,7 @@ Examples:
 - Skip Python 2.7 on 32-bit Windows: `CIBW_SKIP`:`cp27-win32`
 - Skip Python 3.4 and Python 3.5: `CIBW_SKIP`:`cp34-* cp35-*`
 - Skip Python 3.6 on Linux: `CIBW_SKIP`:`cp36-manylinux*`
-- Only build on Python 3 and skip 32-bit builds: `CIBW_BUILD`:`cp3?-*` and `CIBW_SKIP`:`*-win32 *-manylinux1_i686`
+- Only build on Python 3 and skip 32-bit builds: `CIBW_BUILD`:`cp3?-*` and `CIBW_SKIP`:`*-win32 *-manylinux_i686`
 
 ***
 
@@ -370,15 +372,20 @@ Platform-specific variants also available:
 
 ***
 
-| Environment variables: `CIBW_MANYLINUX1_X86_64_IMAGE` and `CIBW_MANYLINUX1_I686_IMAGE`
+| Environment variables: `CIBW_MANYLINUX_X86_64_IMAGE` and `CIBW_MANYLINUX_I686_IMAGE`
 | ---
 
 Optional.
 
-An alternative docker image to be used for building [`manylinux1`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the official images, [`quay.io/pypa/manylinux1_x86_64`](https://quay.io/pypa/manylinux1_i686) and [`quay.io/pypa/manylinux1_i686`](https://quay.io/pypa/manylinux1_i686).
+An alternative Docker image to be used for building [`manylinux`](https://github.com/pypa/manylinux) wheels. `cibuildwheel` will then pull these instead of the default images, [`quay.io/pypa/manylinux2010_x86_64`](https://quay.io/pypa/manylinux2010_x86_64) and [`quay.io/pypa/manylinux2010_i686`](https://quay.io/pypa/manylinux2010_i686).
+
+The value of this option can either be set to `manylinux1` or `manylinux2010` to use the [official `manylinux` images](https://github.com/pypa/manylinux), or any other valid Docker image name.
+
+Beware to specify a valid Docker image that can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1- or manylinux2010-compatible in order to produce valid `manylinux1`/`manylinux2010` wheels (see https://github.com/pypa/manylinux, [PEP 513](https://www.python.org/dev/peps/pep-0513/), and [PEP 571](https://www.python.org/dev/peps/pep-0571/) for more details).
 
-Beware to specify a valid docker image that can be used the same as the official, default docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the `auditwheel` tool needs to be present for `cibuildwheel` to work. Apart from that, the architecture and relevant shared system libraries need to be manylinux1-compatible in order to produce valid `manylinux1` wheels (see https://github.com/pypa/manylinux and [PEP 513](https://www.python.org/dev/peps/pep-0513/) for more details).
+Note that `auditwheel` detects the version of the `manylinux` standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as `cibuildwheel` has no way of detecting the correct `--plat` command line argument to pass to `auditwheel` for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT="manylinux2010_$(uname -m)"`).
 
+Example: `manylinux1`  
 Example: `dockcross/manylinux-x64`  
 Example: `dockcross/manylinux-x86`
 

cibuildwheel/__main__.py

@@ -54,6 +54,8 @@ def main():
 
     args = parser.parse_args()
 
+    detect_obsolete_options()
+
     if args.platform != 'auto':
         platform = args.platform
     else:
@@ -141,11 +143,17 @@ def main():
     )
 
     if platform == 'linux':
-        manylinux1_x86_64_image = os.environ.get('CIBW_MANYLINUX1_X86_64_IMAGE', None)
-        manylinux1_i686_image = os.environ.get('CIBW_MANYLINUX1_I686_IMAGE', None)
+        manylinux_x86_64_image = os.environ.get('CIBW_MANYLINUX_X86_64_IMAGE', 'manylinux2010')
+        manylinux_i686_image = os.environ.get('CIBW_MANYLINUX_I686_IMAGE', 'manylinux2010')
+
+        default_manylinux_images_x86_64 = {'manylinux1': 'quay.io/pypa/manylinux1_x86_64',
+                                           'manylinux2010': 'quay.io/pypa/manylinux2010_x86_64'}
+        default_manylinux_images_i686 = {'manylinux1': 'quay.io/pypa/manylinux1_i686',
+                                         'manylinux2010': 'quay.io/pypa/manylinux2010_i686'}
 
         build_options.update(
-            manylinux1_images={'x86_64': manylinux1_x86_64_image, 'i686': manylinux1_i686_image},
+            manylinux_images={'x86_64': default_manylinux_images_x86_64.get(manylinux_x86_64_image) or manylinux_x86_64_image,
+                              'i686': default_manylinux_images_i686.get(manylinux_i686_image) or manylinux_i686_image},
         )
     elif platform == 'macos':
         pass
@@ -170,6 +178,26 @@ def main():
         raise Exception('Unsupported platform')
 
 
+def detect_obsolete_options():
+    # Check the old 'MANYLINUX1_*_IMAGE' options
+    for (deprecated, alternative) in [('CIBW_MANYLINUX1_X86_64_IMAGE', 'CIBW_MANYLINUX_X86_64_IMAGE'),
+                                      ('CIBW_MANYLINUX1_I686_IMAGE', 'CIBW_MANYLINUX_I686_IMAGE')]:
+        if deprecated in os.environ:
+            print("'{}' has been deprecated, and will be removed in a future release. Use the option '{}' instead.".format(deprecated, alternative))
+            if alternative not in os.environ:
+                print("Using value of option '{}' as replacement for '{}'".format(deprecated, alternative))
+                os.environ[alternative] = os.environ[deprecated]
+            else:
+                print("Option '{}' is not empty. Please unset '{}'".format(alternative, deprecated))
+                exit(2)
+
+    # Check for 'manylinux1' in the 'CIBW_BUILD' and 'CIBW_SKIP' options
+    for deprecated in ['CIBW_BUILD', 'CIBW_SKIP']:
+        if deprecated in os.environ and 'manylinux1' in os.environ[deprecated]:
+            print("Build identifiers with 'manylinux1' been deprecated. Replacing all occurences of 'manylinux1' by 'manylinux' in the option '{}'".format(deprecated))
+            os.environ[deprecated] = os.environ[deprecated].replace('manylinux1', 'manylinux')
+
+
 def print_preamble(platform, build_options):
     print(textwrap.dedent('''
              _ _       _ _   _       _           _

cibuildwheel/linux.py

@@ -12,25 +12,25 @@
 def get_python_configurations(build_selector):
     PythonConfiguration = namedtuple('PythonConfiguration', ['identifier', 'path'])
     python_configurations = [
-        PythonConfiguration(identifier='cp27-manylinux1_x86_64', path='/opt/python/cp27-cp27m'),
-        PythonConfiguration(identifier='cp27-manylinux1_x86_64', path='/opt/python/cp27-cp27mu'),
-        PythonConfiguration(identifier='cp34-manylinux1_x86_64', path='/opt/python/cp34-cp34m'),
-        PythonConfiguration(identifier='cp35-manylinux1_x86_64', path='/opt/python/cp35-cp35m'),
-        PythonConfiguration(identifier='cp36-manylinux1_x86_64', path='/opt/python/cp36-cp36m'),
-        PythonConfiguration(identifier='cp37-manylinux1_x86_64', path='/opt/python/cp37-cp37m'),
-        PythonConfiguration(identifier='cp27-manylinux1_i686', path='/opt/python/cp27-cp27m'),
-        PythonConfiguration(identifier='cp27-manylinux1_i686', path='/opt/python/cp27-cp27mu'),
-        PythonConfiguration(identifier='cp34-manylinux1_i686', path='/opt/python/cp34-cp34m'),
-        PythonConfiguration(identifier='cp35-manylinux1_i686', path='/opt/python/cp35-cp35m'),
-        PythonConfiguration(identifier='cp36-manylinux1_i686', path='/opt/python/cp36-cp36m'),
-        PythonConfiguration(identifier='cp37-manylinux1_i686', path='/opt/python/cp37-cp37m'),
+        PythonConfiguration(identifier='cp27-manylinux_x86_64', path='/opt/python/cp27-cp27m'),
+        PythonConfiguration(identifier='cp27-manylinux_x86_64', path='/opt/python/cp27-cp27mu'),
+        PythonConfiguration(identifier='cp34-manylinux_x86_64', path='/opt/python/cp34-cp34m'),
+        PythonConfiguration(identifier='cp35-manylinux_x86_64', path='/opt/python/cp35-cp35m'),
+        PythonConfiguration(identifier='cp36-manylinux_x86_64', path='/opt/python/cp36-cp36m'),
+        PythonConfiguration(identifier='cp37-manylinux_x86_64', path='/opt/python/cp37-cp37m'),
+        PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27m'),
+        PythonConfiguration(identifier='cp27-manylinux_i686', path='/opt/python/cp27-cp27mu'),
+        PythonConfiguration(identifier='cp34-manylinux_i686', path='/opt/python/cp34-cp34m'),
+        PythonConfiguration(identifier='cp35-manylinux_i686', path='/opt/python/cp35-cp35m'),
+        PythonConfiguration(identifier='cp36-manylinux_i686', path='/opt/python/cp36-cp36m'),
+        PythonConfiguration(identifier='cp37-manylinux_i686', path='/opt/python/cp37-cp37m'),
     ]
 
     # skip builds as required
     return [c for c in python_configurations if build_selector(c.identifier)]
 
 
-def build(project_dir, output_dir, test_command, test_requires, test_extras, before_build, build_verbosity, build_selector, environment, manylinux1_images):
+def build(project_dir, output_dir, test_command, test_requires, test_extras, before_build, build_verbosity, build_selector, environment, manylinux_images):
     try:
         subprocess.check_call(['docker', '--version'])
     except:
@@ -42,8 +42,8 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
 
     python_configurations = get_python_configurations(build_selector)
     platforms = [
-        ('manylinux1_x86_64', manylinux1_images.get('x86_64') or 'quay.io/pypa/manylinux1_x86_64'),
-        ('manylinux1_i686', manylinux1_images.get('i686') or 'quay.io/pypa/manylinux1_i686'),
+        ('manylinux_x86_64', manylinux_images['x86_64']),
+        ('manylinux_i686', manylinux_images['i686']),
     ]
 
     for platform_tag, docker_image in platforms:
@@ -62,9 +62,9 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
             for PYBIN in {pybin_paths}; do
                 # Setup
                 rm -rf /tmp/built_wheel
-                rm -rf /tmp/delocated_wheel
+                rm -rf /tmp/delocated_wheels
                 mkdir /tmp/built_wheel
-                mkdir /tmp/delocated_wheel
+                mkdir /tmp/delocated_wheels
 
                 if [ ! -z {before_build} ]; then
                     PATH="$PYBIN:$PATH" sh -c {before_build}
@@ -79,11 +79,11 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
                 # the first element
                 if [[ "$built_wheel" == *none-any.whl ]]; then
                     # pure python wheel - just copy
-                    mv "$built_wheel" /tmp/delocated_wheel
+                    mv "$built_wheel" /tmp/delocated_wheels
                 else
-                    auditwheel repair "$built_wheel" -w /tmp/delocated_wheel
+                    auditwheel repair "$built_wheel" -w /tmp/delocated_wheels
                 fi
-                delocated_wheel=(/tmp/delocated_wheel/*.whl)
+                delocated_wheels=(/tmp/delocated_wheels/*.whl)
 
                 if [ ! -z {test_command} ]; then
                     # Set up a virtual environment to install and test from, to make sure
@@ -100,7 +100,12 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
                         echo "Running tests using `which python`"
 
                         # Install the wheel we just built
-                        pip install "$delocated_wheel"{test_extras}
+                        # Note: If auditwheel produced two wheels, it's because the earlier produced wheel
+                        # conforms to multiple manylinux standards. These multiple versions of the wheel are
+                        # functionally the same, differing only in name, wheel metadata, and possibly include
+                        # different external shared libraries. so it doesn't matter which one we run the tests on.
+                        # Let's just pick the first one.
+                        pip install "${{delocated_wheels[0]}}"{test_extras}
 
                         # Install any requirements to run the tests
                         if [ ! -z "{test_requires}" ]; then
@@ -118,8 +123,8 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
                 fi
 
                 # we're all done here; move it to output
-                mv "$delocated_wheel" /output
-                chown {uid}:{gid} "/output/$(basename "$delocated_wheel")"
+                mv "${{delocated_wheels[@]}}" /output
+                for delocated_wheel in "${{delocated_wheels[@]}}"; do chown {uid}:{gid} "/output/$(basename "$delocated_wheel")"; done
             done
         '''.format(
             pybin_paths=' '.join(c.path+'/bin' for c in platform_configs),

test/01_basic/cibuildwheel_test.py

@@ -17,6 +17,10 @@ def test():
 def test_build_identifiers():
     # check that the number of expected wheels matches the number of build
     # identifiers
-    expected_wheels = utils.expected_wheels('spam', '0.1.0')
+    # after adding CIBW_MANYLINUX_IMAGE to support manylinux2010, there
+    # can be multiple wheels for each wheel, though, so we need to limit
+    # the expected wheels
+    expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0')
+                       if not '-manylinux' in w or '-manylinux1' in w]
     build_identifiers = utils.cibuildwheel_get_build_identifiers(project_dir)
     assert len(expected_wheels) == len(build_identifiers)

test/06_docker_images/cibuildwheel_test.py

@@ -5,14 +5,17 @@ def test():
     project_dir = os.path.dirname(__file__)
 
     if utils.platform != 'linux':
-        pytest.skip('the docker test is only relevant to the linux build')
+        pytest.skip('the test is only relevant to the linux build')
 
     utils.cibuildwheel_run(project_dir, add_env={
-        'CIBW_MANYLINUX1_X86_64_IMAGE': 'dockcross/manylinux-x64',
-        'CIBW_MANYLINUX1_I686_IMAGE': 'dockcross/manylinux-x86',
+        'CIBW_MANYLINUX_X86_64_IMAGE': 'dockcross/manylinux2010-x64',
+        'CIBW_MANYLINUX_I686_IMAGE': 'dockcross/manylinux1-x86',
+        'CIBW_BEFORE_BUILD': '/opt/python/cp36-cp36m/bin/pip install -U auditwheel',  # Currently necessary on dockcross images to get auditwheel 2.1 supporting AUDITWHEEL_PLAT
+        'CIBW_ENVIRONMENT': 'AUDITWHEEL_PLAT=`if [ $(uname -i) == "x86_64" ]; then echo "manylinux2010_x86_64"; else echo "manylinux1_i686"; fi`',
     })
 
     # also check that we got the right wheels built
-    expected_wheels = utils.expected_wheels('spam', '0.1.0')
+    expected_wheels = [w for w in utils.expected_wheels('spam', '0.1.0')
+                       if '-manylinux2010_i686' not in w]
     actual_wheels = os.listdir('wheelhouse')
     assert set(actual_wheels) == set(expected_wheels)