[FR] pip options in extra_requires

[msftcangoblowm]

setuptools version

setuptools==77.0.3

Python version

Python 3.9

OS

Void Linux

Additional environment information

packaging==24.2

pip==25.0.1

pip-tools==7.4.1

Description

When hosting .whl locally or on a warehouse, besides pypi.org, how to inform pip where to search for those packages:

1. cli provided options --find-links --extra-index-url or --index-url

2. $HOME/.pip/pip-conf

[global]
extra-index-url = http://localhost:8080/simple/

3. Specified within each and every requirements.txt file

When .whl are hosted locally or on non-pypi.org warehouses, pip-compile will insert lines looking like,

--extra-index-url http://localhost:8080/simple/

setuptools.dist.Distribution._normalize_requires does not filter out those lines!

Then packaging.requirements.Requirement.parse does not recognize those lines as packages.

other reports

wreck#30

packaging#884

Expected behavior

setuptools.dist.Distribution._normalize_requires, filter out (sourced from config file) lines that start with --find-links --extra-index-url or --index-url.

Avoiding packaging.requirements.Requirement raising a packaging.requirements.InvalidRequirement exception.

How to Reproduce

1. setup pypiserver service locally, including $HOME/.pip/pip-conf file

[global]
extra-index-url = http://localhost:8080/simple/

1. pip-compile -o requirements.in requirements.txt

requirements.in

# available on pypi.org
packaging

# not available on pypi.org Available locally or on another remote warehouse
packagenotonpypiorg

requirements.txt

--extra-index-url http://localhost:8080/simple/

# available on pypi.org
packaging

# not available on pypi.org Available locally or on another remote warehouse
packagenotonpypiorg

2. pyproject.toml

[build-system]
requires = ["setuptools>=77.0.3", "wheel", "build"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.0.1.dev0"
dynamic = [
    "dependencies",
]
[tool.setuptools.dynamic]
dependencies = { file = ["requirements.txt"] }

3. place a package, not available on pypi.org, within the configured folder holding .whl files

packagenotonpypiorg.whl

4. python -m build a project with requirement.txt containing a dependency only hosted locally.

build asks setuptools to search for packages. First http://localhost:8080/simple/ then pypi.org

setuptools.dist.Distribution._normalize_requires does not filter out warehouse location lines.

packaging.requirements.Requirement does not recognize those lines as packages.

In this example, a local pypiserver is configured. When python -m pip install runs, the --extra-index-url option is necessary to find locally hosted .whl (packages). Which is great for pip, not so great for python -m build.

Output

Traceback (most recent call last):
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/packaging/_parser.py", line 71, in _parse_requirement
    name_token = tokenizer.expect(
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/packaging/_tokenizer.py", line 142, in expect
    raise self.raise_syntax_error(f"Expected {expected}")
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
packaging._tokenizer.ParserSyntaxError: Expected package name at the start of dependency specifier
    --extra-index-url http://localhost:8080/simple/
    ^
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/sda1/dev_parent/sqa_cf/.venv/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/mnt/sda1/dev_parent/sqa_cf/.venv/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/mnt/sda1/dev_parent/sqa_cf/.venv/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 317, in get_requires_for_build_sdist
    return hook(config_settings)
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/build_meta.py", line 337, in get_requires_for_build_sdist
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/build_meta.py", line 320, in run_setup
    exec(code, locals())
  File "<string>", line 1, in <module>
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 160, in setup
    dist.parse_config_files()
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/dist.py", line 757, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 73, in apply_configuration
    return _apply(dist, config, filepath)
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 60, in apply
    dist._finalize_requires()
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/dist.py", line 383, in _finalize_requires
    self._normalize_requires()
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/dist.py", line 404, in _normalize_requires
    self.extras_require = dict_(
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/setuptools/dist.py", line 405, in <genexpr>
    (k, list(map(str, _reqs.parse(v or [])))) for k, v in extras_require.items()
  File "/tmp/build-env-ve1j18i8/lib/python3.9/site-packages/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier
    --extra-index-url http://localhost:8080/simple/

[msftcangoblowm]

Proposed solution

In setuptools.dist.Distribution._normalize_requires, filter out (sourced from config file) lines that start with --find-links --extra-index-url or --index-url.

Since those lines are not valid packages, packaging.requirements.Requirement will raise a packaging.requirements.InvalidRequirement exception.

Potential fix

In setuptools/setuptools/dist.py,

Distribution._normalize_requires

    def _normalize_requires(self):
        """Make sure requirement-related attributes exist and are normalized"""
        extras_require = getattr(self, "extras_require", None) 
        dict_ = _static.Dict if _static.is_static(extras_require) else dict
        self.extras_require = dict_(
               (k, list(map(str, _reqs.parse(v or [])))) for k, v in extras_require.items() if not v.startswith("--extra-index-url") and not v.startswith("--index-url") and not v.startswith("--find-links")
        )

        install_requires = getattr(self, "install_requires", None) 
        # Preserve the "static"-ness of values parsed from config files
        list_ = _static.List if _static.is_static(install_requires) else list
        install_requires_filtered = [
            line
            for line in install_requires
            if line.startswith("--extra-index-url") and not line.startswith("--index-url") and not line.startswith("--find-links")
        ]
        self.install_requires = list_(map(str, _reqs.parse(install_requires_filtered)))

Not sure how to create a pytest test to cover the situation of a project with requirement.txt containing a dependency which is locally hosted.

[msftcangoblowm]

In the meantime, after pip-compile and before calling python -m build, manually edit requirements.txt to comment out the pip option lines.

To install the package

python -m pip install --extra-index-url='http://localhost:8080/simple/' dist/mypackage-0.0.1.dev0-py3-none-any.whl

[abravalheri]

Hi @msftcangoblowm thank you very much for opening this issue e for the initial investigation.

Please note however that this is a documented limitation instead of a bug. In the docs page you can read:

Supporting file for dependencies is meant for a convenience for packaging applications with possibly strictly versioned dependencies.

Library packagers are discouraged from using overly strict (or “locked”) dependency versions in their dependencies and optional-dependencies.

Currently, when specifying optional-dependencies dynamically, all of the groups must be specified dynamically; one can not specify some of them statically and some of them dynamically.

Also note that the file format for specifying dependencies resembles a requirements.txt file, however please keep in mind that all non-comment lines must conform with PEP 508 (pip specific syntaxes, e.g. -c/-r/-e and other flags, are not supported).

So I am reclassifing this as a feature request instead of a bug.

[msftcangoblowm]

From setuptools POV can understand and accept, as a documented limitation, is considered a FR.

Arguments for

Would like to advocate for this feature, so laying out the arguments, eventhough it's not news.

There are other perspectives:

1. the FR is to ignore -r/-c/-e/ and other pip options. Rather than crash unnecessarily with a traceback.

2. Supporting packages being hosted on multiple servers, as well as locally, is the way it should be. Hosting only on one is why initially labelled this issue as a planet sized missed opportunity.

3. working locally shouldn't be punished. Hope i'm not alone in hosting some packages locally.

4. The tiresome argument against a single point of failure. A scenario where this can come to pass. Tomorrow every country puts every other country on their respective terrorist sponsoring organization lists. And poof pypi.org is doomed as every govt decides they must have their own and maintainers from other countries are persona non grata. We've already reached clown world, would like to live in a world where even mentioning this in a low whisper at a bar is a complete embarrassment and a step too far and over any reasonable line of decency. But here we are.

5. self-hosting shouldn't be punished unless hosting malicious packages.

Counter arguments

1. The test of this feature might turn out to be slightly novel.

These require system services: pypiserver or dumb-pypi
(a web server)

this requires nothing but a folder and no additional dependencies.

pip install --no-index --find-links=wheels/ -r requirements.txt

2. package limitation is a limitation for a really good reason

3. place the burden on downstream requirements rendering packages to comment out the lines thereby disabling any and all pip options

[webknjaz]

Setuptools is a build backend. Its purpose is to create an artifact with spec-compliant metadata. It is not an installer and has no ability to know where the deps are coming from. Similarly, the installers can only see the abstract names of the deps. And installers can decide where to get the deps. Build backends are an absolutely incorrect place for storing/processing/special-casing settings of arbitrary build front-ends. This separation is standardized and exists for a reason. Start at PEP 517 to understand more.

[msftcangoblowm]

This is not a build backend build frontend issue. The build backend is setuptools.build_meta

Not passing any config settings to a custom build backend and/or thru to the build front end subprocess.

setuptools requirements parsing is crashing when encountering garbage requirements lines. Would merely like to filter out the garbage before setuptools requirements parsing occurs.

This is not an add support request it's to gracefully ignore all options appearing in requirements files. The status quo of do nothing results in setuptools being a dedicated tool for pypi.org only.

[msftcangoblowm]

A way to reframe this is consider any requirement line beginning with -- to be equivalent to # , a comment. setuptools does filter out requirement file comments. Once package lines are sent to packaging.requirements.Requirement, it's too late

So this FR is build front end agnostic.

Is this chain of thought acceptable?

There is no package named --

https://pypi.org/project/--/

https://pypi.org/search/?q=--

Sorry this FR did not start off by proposing requirements lines starting with -- are comments.

[webknjaz]

The status quo of do nothing results in setuptools being a dedicated tool for pypi.org only.

Not really. Pip settings don't get into core packaging metadata. I think I missed that you were talking about a specific feature of setuptools loading the entries from pip-specific requirement files. So it might make sense to look into disregarding those flags on parsing.

Another way would be declaring this unsupported and telling people to restructure layering of the requirement files in a way that allows pointing to files that don't include pip flags.

Arguably, your setuptools config should be loading entries of requirements.in and not the venv definitions of requirements.txt. It's just semantically wrong to do otherwise.

[msftcangoblowm]

load .in instead of .txt

requirements.in contains -c/-r/-e pip options. So requirements.in is unsuitable without resolving to something safe for setuptools.

package wreck renamed .txt to .lock

The workflow is: .in --> .lock. Then .in and .lock --> .unlock

.unlock is a resolved/flat .in

.unlock doesn't contain pip option flags at all. But actually that's a wreck bug. Where is the other package warehouse locations? Guilty of well it works on my machine.

Believe i'm the only person on the planet using wreck for requirement file management, but it's producing what you consider safe for setuptools requirement files.

A package will use both .lock and .unlock.

.unlock for library dependencies

.lock for all the various optional dependencies.

A package author chooses which, to use, for what.

When a dependency package has dodgy/misbehaving transitive dependencies, those can be thrown into separate venv.

Example commands to render .lock and .unlock requirement files. With one venv for the docgy package with misbehaving transitive dependencies.

reqs fix --venv-relpath='.venv'
reqs fix --venv-relpath='.doc'
reqs fix --venv-relpath='.rst2html'

Assumes pyproject.toml has those three venv configured and a tox-reqs.ini is running a command with the appropriate python version. So tox is run three times.

declaring this unsupported

This is equivalent to do nothing.

There is no third option. A package contains only dependencies and optional-dependencies. Using only safe for setuptools requirement files means not being able to support host locally or on other package warehouses.

make sense to look into disregarding those flags on parsing

Lets do this. Pretty please. Treat -- lines as a comment.

Can i get consensus that this is the way to move forward.

[abravalheri]

Hi @msftcangoblowm thank you for the clarifications. I think I understand your motivations and how it would help to move your use case forward.

However we also need to consider 2 points:

1. While the implementation of ingore lines starting with -- is pretty straight forward, and some users may be aware that setuptools is simply ignoring unknow flags, other users will mistakenly believe that the semantics of those flags are being taken into consideration.

   The understanding of why these semantics cannot be handled by setuptools requires a more in dept understanding of the Python packaging landscape.

2. Before the feature of reading dependencies from text files was introduced there was a lot of debate. So historically this feature has been very controversial and one of the conditions for the community contribution that introduced the feature to be merged was that Setuptools would not handle pip-specific flags or syntaxes.

With that in mind, I believe that would be best to explore some of the other options mentioned by @webknjaz, for example:

restructure layering of the requirement files in a way that allows pointing to files that don't include pip flags.

Can you have 2 files, one without the flags intended to be consumed by setuptools and another one with the flags that includes the first one via -r ..., intended to be consumed by the installer?

[msftcangoblowm]

if setuptools would cooperate by allowing two sets of dependencies and optional-dependencies. One setuptools safe. And current pair consumed by build front ends that can include build front end options.

With two sets of dependencies and optional-dependencies, .in aside, there would be four file extensions for requirements files

For example

.unlock
.lock
.safe.unlock
.safe.lock

How would setuptools find the safe requirements files during python -m build? Could there be two set in [tool.setuptools.dynamic]?

If setuptools says yes. I am willing to retool wreck to always output both safe and non-safe requirements files.

Lets call this best of both worlds scenario

[tool.setuptools.dynamic]
dependencies = { file = ["requirements/prod.unlock"], safe = ["requirements/prod.safe.unlock"] }
optional-dependencies.dev = { file = ["requirements/dev.lock"], safe = ["requirements/dev.safe.lock"] }
optional-dependencies.manage = { file = ["requirements/manage.lock"], safe = ["requirements/manage.safe.lock"] }

If safe key exists, setuptools uses that instead of file. The build front ends can then have access to file dependencies and optional-dependencies.

file would still not contain -r/-c/-e options.

The advantage of best of both worlds is requirements file management is kept simple.