Skip to content

Vendor distutils stubs #4691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions _distutils_hack/__init__.py
Original file line number Diff line number Diff line change
@@ -3,8 +3,7 @@
import sys

report_url = (
"https://github.com/pypa/setuptools/issues/new?"
"template=distutils-deprecation.yml"
"https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml"
)


@@ -74,7 +73,7 @@

# check that submodules load as expected
core = importlib.import_module('distutils.core')
assert '_distutils' in core.__file__, core.__file__

Check warning on line 76 in _distutils_hack/__init__.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Operator "in" not supported for types "Literal['_distutils']" and "str | None"   Operator "in" not supported for types "Literal['_distutils']" and "None" (reportOperatorIssue)

Check warning on line 76 in _distutils_hack/__init__.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Operator "in" not supported for types "Literal['_distutils']" and "str | None"   Operator "in" not supported for types "Literal['_distutils']" and "None" (reportOperatorIssue)
assert 'setuptools._distutils.log' not in sys.modules


@@ -213,7 +212,7 @@


def add_shim():
DISTUTILS_FINDER in sys.meta_path or insert_shim()

Check warning on line 215 in _distutils_hack/__init__.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Expression value is unused (reportUnusedExpression)

Check warning on line 215 in _distutils_hack/__init__.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Expression value is unused (reportUnusedExpression)


class shim:
14 changes: 5 additions & 9 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
strict = False

# Early opt-in even when strict = False
# warn_unused_ignores = True # Disabled until we have distutils stubs for Python 3.12+
warn_unused_ignores = True
warn_redundant_casts = True
enable_error_code = ignore-without-code

@@ -18,6 +18,9 @@ disable_error_code =

## local

# Use our custom stubs for distutils
mypy_path = $MYPY_CONFIG_FILE_DIR/typings
Comment on lines +21 to +22
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used pyright's default of typings so that I didn't have to configure it there. But that folder name can be arbitrary.


# CI should test for all versions, local development gets hints for oldest supported
# But our testing setup doesn't allow passing CLI arguments, so local devs have to set this manually.
# python_version = 3.8
@@ -48,17 +51,10 @@ disable_error_code =
[mypy-pkg_resources.tests.*]
disable_error_code = import-not-found

# - distutils doesn't exist on Python 3.12, unfortunately, this means typing
# will be missing for subclasses of distutils on Python 3.12 until either:
# - support for `SETUPTOOLS_USE_DISTUTILS=stdlib` is dropped (#3625)
# for setuptools to import `_distutils` directly
# - or non-stdlib distutils typings are exposed
[mypy-distutils.*]
ignore_missing_imports = True

# - wheel: does not intend on exposing a programmatic API https://github.com/pypa/wheel/pull/610#issuecomment-2081687671
[mypy-wheel.*]
ignore_missing_imports = True

# - The following are not marked as py.typed:
# - jaraco: Since mypy 1.12, the root name of the untyped namespace package gets called-out too
# - jaraco.develop: https://github.com/jaraco/jaraco.develop/issues/22
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -133,6 +133,8 @@ type = [

# local

# Referenced in distutils-stubs
"types-docutils",
# pin mypy version so a new version doesn't suddenly cause the CI to fail,
# until types-setuptools is removed from typeshed.
# For help with static-typing issues, or mypy update, ping @Avasam
@@ -203,6 +205,8 @@ include-package-data = true
include = [
"setuptools*",
"pkg_resources*",
# TODO: Include distutils stubs with package once we're confident in them
# "typings/distutils-stubs",
Comment on lines +208 to +209
Copy link
Contributor Author

@Avasam Avasam Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to ship anything as long as typeshed provides stubs for setuptools/_distutils. We can synchronize once ready.
As I mentioned in the issue, there's also always the option to ship it as a different package (in which case it could live completely separately from setuptools and pypa/distutils)

"_distutils_hack*",
]
exclude = [
2 changes: 2 additions & 0 deletions pyrightconfig.json
Original file line number Diff line number Diff line change
@@ -12,6 +12,8 @@
],
// Our testing setup doesn't allow passing CLI arguments, so local devs have to set this manually.
// "pythonVersion": "3.8",
// Allow using distutils-stubs on Python 3.12+
"reportMissingModuleSource": false,
// For now we don't mind if mypy's `type: ignore` comments accidentally suppresses pyright issues
"enableTypeIgnoreComments": true,
"typeCheckingMode": "basic",
3 changes: 3 additions & 0 deletions ruff.toml
Original file line number Diff line number Diff line change
@@ -34,6 +34,9 @@ ignore = [
# Only enforcing return type annotations for public functions
"ANN202", # missing-return-type-private-function
"ANN204", # missing-return-type-special-method
# Typeshed doesn't want complex or non-literal defaults for maintenance and testing reasons.
# This doesn't affect us, let's have more complete stubs.
"PYI011", # typed-argument-default-in-stub

# https://docs.astral.sh/ruff/formatter/#conflicting-lint-rules
"W191",
7 changes: 1 addition & 6 deletions setuptools/__init__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,4 @@
"""Extensions to the 'distutils' for large or complex distributions"""
# mypy: disable_error_code=override
# Command.reinitialize_command has an extra **kw param that distutils doesn't have
# Can't disable on the exact line because distutils doesn't exists on Python 3.12
# and mypy isn't aware of distutils_hack, causing distutils.core.Command to be Any,
# and a [unused-ignore] to be raised on 3.12+

from __future__ import annotations

@@ -213,7 +208,7 @@ def ensure_string_list(self, option: str):
"'%s' must be a list of strings (got %r)" % (option, val)
)

@overload
@overload # type: ignore[override] # Has an extra **kw param that distutils doesn't have
def reinitialize_command(
self, command: str, reinit_subcommands: bool = False, **kw
) -> _Command: ...
2 changes: 1 addition & 1 deletion setuptools/command/build_py.py
Original file line number Diff line number Diff line change
@@ -143,7 +143,7 @@ def find_data_files(self, package, src_dir):
)
return self.exclude_data_files(package, src_dir, files)

def get_outputs(self, include_bytecode: bool = True) -> list[str]: # type: ignore[override] # Using a real boolean instead of 0|1
def get_outputs(self, include_bytecode: bool = True) -> list[str]:
"""See :class:`setuptools.commands.build.SubCommand`"""
if self.editable_mode:
return list(self.get_output_mapping().keys())
7 changes: 3 additions & 4 deletions setuptools/command/install_lib.py
Original file line number Diff line number Diff line change
@@ -95,10 +95,9 @@ def copy_tree(
self,
infile: StrPath,
outfile: str,
# override: Using actual booleans
preserve_mode: bool = True, # type: ignore[override]
preserve_times: bool = True, # type: ignore[override]
preserve_symlinks: bool = False, # type: ignore[override]
preserve_mode: bool = True,
preserve_times: bool = True,
preserve_symlinks: bool = False,
level: object = 1,
) -> list[str]:
assert preserve_mode and preserve_times and not preserve_symlinks
5 changes: 1 addition & 4 deletions setuptools/config/setupcfg.py
Original file line number Diff line number Diff line change
@@ -24,10 +24,8 @@
Generic,
Iterable,
Iterator,
List,
Tuple,
TypeVar,
cast,
)

from packaging.markers import default_environment as marker_env
@@ -112,8 +110,7 @@ def _apply(
filenames = [*other_files, filepath]

try:
# TODO: Temporary cast until mypy 1.12 is released with upstream fixes from typeshed
_Distribution.parse_config_files(dist, filenames=cast(List[str], filenames))
_Distribution.parse_config_files(dist, filenames=filenames)
handlers = parse_configuration(
dist, dist.command_options, ignore_option_errors=ignore_option_errors
)
8 changes: 4 additions & 4 deletions setuptools/errors.py
Original file line number Diff line number Diff line change
@@ -30,15 +30,15 @@
BaseError = _distutils_errors.DistutilsError


class InvalidConfigError(OptionError): # type: ignore[valid-type, misc] # distutils imports are `Any` on python 3.12+
class InvalidConfigError(OptionError):
"""Error used for invalid configurations."""


class RemovedConfigError(OptionError): # type: ignore[valid-type, misc] # distutils imports are `Any` on python 3.12+
class RemovedConfigError(OptionError):
"""Error used for configurations that were deprecated and removed."""


class RemovedCommandError(BaseError, RuntimeError): # type: ignore[valid-type, misc] # distutils imports are `Any` on python 3.12+
class RemovedCommandError(BaseError, RuntimeError):
"""Error used for commands that have been removed in setuptools.

Since ``setuptools`` is built on ``distutils``, simply removing a command
@@ -48,7 +48,7 @@ class RemovedCommandError(BaseError, RuntimeError): # type: ignore[valid-type,
"""


class PackageDiscoveryError(BaseError, RuntimeError): # type: ignore[valid-type, misc] # distutils imports are `Any` on python 3.12+
class PackageDiscoveryError(BaseError, RuntimeError):
"""Impossible to perform automatic discovery of packages and/or modules.

The current project layout or given discovery options can lead to problems when
Empty file.
25 changes: 25 additions & 0 deletions typings/distutils-stubs/_modified.pyi
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from collections.abc import Callable, Iterable
from typing import Literal, TypeVar

from _typeshed import StrOrBytesPath

_SourcesT = TypeVar("_SourcesT", bound=StrOrBytesPath)
_TargetsT = TypeVar("_TargetsT", bound=StrOrBytesPath)

def newer(source: StrOrBytesPath, target: StrOrBytesPath) -> bool: ...
def newer_pairwise(
sources: Iterable[_SourcesT],
targets: Iterable[_TargetsT],
newer: Callable[[_SourcesT, _TargetsT], bool] = newer,
) -> tuple[list[_SourcesT], list[_TargetsT]]: ...
def newer_group(
sources: Iterable[StrOrBytesPath],
target: StrOrBytesPath,
missing: Literal["error", "ignore", "newer"] = "error",
) -> bool: ...
def newer_pairwise_group(
sources: Iterable[_SourcesT],
targets: Iterable[_TargetsT],
*,
newer: Callable[[_SourcesT, _TargetsT], bool] = newer,
) -> tuple[list[_SourcesT], list[_TargetsT]]: ...
38 changes: 38 additions & 0 deletions typings/distutils-stubs/archive_util.pyi
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
from typing import Literal, overload

from _typeshed import StrOrBytesPath, StrPath

@overload
def make_archive(
base_name: str,
format: str,
root_dir: StrOrBytesPath | None = None,
base_dir: str | None = None,
verbose: bool = False,
dry_run: bool = False,
owner: str | None = None,
group: str | None = None,
) -> str: ...
@overload
def make_archive(
base_name: StrPath,
format: str,
root_dir: StrOrBytesPath,
base_dir: str | None = None,
verbose: bool = False,
dry_run: bool = False,
owner: str | None = None,
group: str | None = None,
) -> str: ...
def make_tarball(
base_name: str,
base_dir: StrPath,
compress: Literal["gzip", "bzip2", "xz"] | None = "gzip",
verbose: bool = False,
dry_run: bool = False,
owner: str | None = None,
group: str | None = None,
) -> str: ...
def make_zipfile(
base_name: str, base_dir: str, verbose: bool = False, dry_run: bool = False
) -> str: ...
3 changes: 3 additions & 0 deletions typings/distutils-stubs/bcppcompiler.pyi
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from .ccompiler import CCompiler

class BCPPCompiler(CCompiler): ...
Loading

Unchanged files with check annotations Beta

"""
Undo secondary effect of `extra_path` adding to `install_lib`
"""
suffix = os.path.relpath(self.install_lib, self.install_libbase)

Check warning on line 79 in setup.py

GitHub Actions / pyright (3.8, ubuntu-latest)

No overloads for "relpath" match the provided arguments (reportCallIssue)

Check warning on line 79 in setup.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Argument of type "str | None" cannot be assigned to parameter "path" of type "StrPath" in function "relpath"   Type "str | None" is not assignable to type "StrPath"     Type "None" is not assignable to type "StrPath"       "None" is not assignable to "str"       "None" is incompatible with protocol "PathLike[str]"         "__fspath__" is not present (reportArgumentType)

Check warning on line 79 in setup.py

GitHub Actions / pyright (3.12, ubuntu-latest)

No overloads for "relpath" match the provided arguments (reportCallIssue)

Check warning on line 79 in setup.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Argument of type "str | None" cannot be assigned to parameter "path" of type "StrPath" in function "relpath"   Type "str | None" is not assignable to type "StrPath"     Type "None" is not assignable to type "StrPath"       "None" is not assignable to "str"       "None" is incompatible with protocol "PathLike[str]"         "__fspath__" is not present (reportArgumentType)
if suffix.strip() == self._pth_contents.strip():
self.install_lib = self.install_libbase
def finalize_options(self):
ei_cmd = self.ei_cmd = self.get_finalized_command("egg_info")
self.egg_info = ei_cmd.egg_info

Check warning on line 106 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Cannot access attribute "egg_info" for class "Command"   Attribute "egg_info" is unknown (reportAttributeAccessIssue)

Check warning on line 106 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Cannot access attribute "egg_info" for class "Command"   Attribute "egg_info" is unknown (reportAttributeAccessIssue)
if self.bdist_dir is None:
bdist_base = self.get_finalized_command('bdist').bdist_base

Check warning on line 109 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Cannot access attribute "bdist_base" for class "bdist"   Attribute "bdist_base" is unknown (reportAttributeAccessIssue)

Check warning on line 109 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Cannot access attribute "bdist_base" for class "bdist"   Attribute "bdist_base" is unknown (reportAttributeAccessIssue)
self.bdist_dir = os.path.join(bdist_base, 'egg')
if self.plat_name is None:
if self.egg_output is None:
# Compute filename of the output egg
basename = ei_cmd._get_egg_basename(

Check warning on line 121 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Cannot access attribute "_get_egg_basename" for class "Command"   Attribute "_get_egg_basename" is unknown (reportAttributeAccessIssue)

Check warning on line 121 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Cannot access attribute "_get_egg_basename" for class "Command"   Attribute "_get_egg_basename" is unknown (reportAttributeAccessIssue)
py_version=get_python_version(),
platform=self.distribution.has_ext_modules() and self.plat_name,
)
self.egg_output = os.path.join(self.dist_dir, basename + '.egg')

Check warning on line 126 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.8, ubuntu-latest)

No overloads for "join" match the provided arguments (reportCallIssue)

Check warning on line 126 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.8, ubuntu-latest)

Argument of type "None" cannot be assigned to parameter "a" of type "BytesPath" in function "join"   Type "None" is not assignable to type "BytesPath"     "None" is not assignable to "bytes"     "None" is incompatible with protocol "PathLike[bytes]"       "__fspath__" is not present (reportArgumentType)

Check warning on line 126 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.12, ubuntu-latest)

No overloads for "join" match the provided arguments (reportCallIssue)

Check warning on line 126 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.12, ubuntu-latest)

Argument of type "None" cannot be assigned to parameter "a" of type "BytesPath" in function "join"   Type "None" is not assignable to type "BytesPath"     "None" is not assignable to "bytes"     "None" is incompatible with protocol "PathLike[bytes]"       "__fspath__" is not present (reportArgumentType)
def do_install_data(self):
# Hack for packages that install data to install's --install-lib
to_compile = []
for p, ext_name in enumerate(ext_outputs):
filename, ext = os.path.splitext(ext_name)
pyfile = os.path.join(self.bdist_dir, strip_module(filename) + '.py')

Check warning on line 185 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.8, ubuntu-latest)

No overloads for "join" match the provided arguments (reportCallIssue)

Check warning on line 185 in setuptools/command/bdist_egg.py

GitHub Actions / pyright (3.12, ubuntu-latest)

No overloads for "join" match the provided arguments (reportCallIssue)
self.stubs.append(pyfile)
log.info("creating stub loader for %s", ext_name)
if not self.dry_run: