Governance and Change Management Review After Breaking Change in #4870 #4919
Comments
|
To clarify, are you offering resources in the form of long term support or funding to setuptools to support your urgent request of these seemingly resource intensive corporate-like governance structures? Separately, while I do think there are open questions about build backends being able to make backwards compatible changes, I also think this is an opportunity for users to self reflect on:
|
Respectfully, this response misses the point. Setuptools is a foundational project with deep, transitive integration across the Python ecosystem. The concern raised is not about the technical direction of the project, but about process, governance, and the way this specific breaking change was introduced:
This is about the standard of care expected from maintainers of critical infrastructure. To your question: if the setuptools team lacks the resources to implement safeguards before merging breaking changes, that is a valid and important discussion to have. But the right response to limited resources is caution, not recklessness. If you're open to support, I’d be happy to help facilitate discussion with Python community stakeholders who are willing to support long-term stability—because what's at risk here is not just a broken build, but trust in the reliability of Python's core tooling. Let’s not turn this into a false binary between modernization and accountability. We can and must have both. |
|
More data points:
Linux distributions and some companies do not rely on wheels from PyPI. Instead they are rebuilding all packages from sources and store the sources + wheels in an internal package index. I detected bugs #4892 and #4910 within an hour of the setuptools release, because latest setuptools release broke the test instance of our build pipeline that rebuilds all our wheels from sdists. We perform regular tests to ensure that we can build and rebuilds latest versions. If you are interested to learn more about our tool chain: https://github.com/python-wheel-build/fromager |
Thank you for the opportunity to self-reflect, the whole community is forever grateful! - on a more serious note, I believe this is exactly what the author meant when he said "dismissive attitude". But to take things one at a time:
|
|
@ianballard @SamuZad To be clear I am not a setuptools maintainer, so please don't take my replies as such.
Besides the last point, in which I can't speak for setuptools, I do largely discuss the issues of build backends making breaking changes in the post I linked to. Edit: Fixed hyperlink
Yes, most Python packaging foundational projects are severely under resourced, that's why I was asking you to clarify if you are going to provide funding for making these urgent requests.
Sure, but the user reports did not sound like Linux distros, who often vendor their own version of setuptools and have custom build scripts, nor typically use a pip or python front end tools to build. Are you a Linunx distro that had this problem yesterday? Why did you have these problems? Are you not vendoring your own version of setuptools?
Wheels weren't broken, which is what most users are expected to use to install Python packages. It is still unclear to me why most users reporting the situation were depending on building sdists, and this should be understood as part of this “post-mortem”, we should be encouraging users to move to wheels.
Remember, I'm not a setuptools maintainer, but I am invested in Python packaging, so this is "just my opinion", but yes, I would strongly recommend looking at using flit or hatchling instead of setuptools. I have already done so for all my work projects last year. |
|
I am flabbergasted to why v78.0.1 still have not been yanked from pypi. https://pypi.org/project/setuptools/#history For much less impact v75.9.0 was yanked and yet the problematic v78 survives. There is zero reason they should have kept it during the abrupt event and even more so now the feature was rolled back |
I'm working on a Linux distro. The test instance of our build pipeline was affected by two breaking changes in setuptools in the past 7 days. Our test pipeline builds with the latest version of all our dependencies to detect these kinds of breaking changes. See #4919 (comment) |
Thanks for the info, I read your earlier comment but I found something a little ambiguous, do you use the latest setuptools in a test pipeline to validate everything works? Or is it part of the real build pipeline and this caused the Linux distro to be unable to publish new packages? Any post-mortem would need to understand the impact, and understand the state of current practise, to fix things and to advise on new best practises. |
|
I understand that people are frustrated, but I don't think this tone is really a good way to start a constructive discussion. If you start with hostility and demands, the discussion is likely to derail into further hostility, counter-hostility and we will be unlikely to reach any solutions, or even focus on the right problems. Yes, I am often frustrated too. Gentoo Linux is a source-first distribution, so the vast majority of our users are building from source, and are downstream to all Python build systems. Major disruptive changes bring a lot of frustration, because in the end they put the burden of fixing the issues on others. But I do realize that setuptools wants to go forward, and doesn't want to have to support deprecated functionality forever. However, I feel like it's an ecosystem problem. Firstly, we do not have a reliable way of reporting deprecations at build system level. The way it is currently done, deprecation warnings are printed in the middle of a very verbose build log produced by setuptools, and are easily missed. On top of that, some tools don't print build log at all. And even if we solved that and made wheel building/installing tools catch deprecation warnings and make them more visible, a lot of packages are only building wheels as part of their CD pipeline, and rarely look at the output. Of course, there's the option of having a Secondly, Python packages get abandoned. We're dealing with dozens of packages that are either not maintained at all or barely maintained — and many of them obviously still "work", at least as far as users are concerned, and are dependencies of many projects. They often use setuptools; some of them don't supply wheels for one reason or another. Breaking changes break these packages, and when there's no active maintainer to quickly make a new release, things go very wrong (pipelines get broken, people start independently forking the same package with no intent on future maintenance…). |
|
@mgorny This really is an ecosystem problem, and that’s exactly why setuptools is in a great position to lead the way. The pushback isn’t about avoiding progress—it’s about wanting changes to be smoother, more transparent, and less disruptive. This could be an opportunity to do things better: clearer deprecation paths, better tooling, and more open comms so we move forward without breaking trust along the way. |
We do. Almost all packages just pass through, unless we discover something that causes us to decide otherwise. But every package that lives on the blocked list increases the burden on us (across thousands of end users). And if a tool doesn't state what versions deprecations will take place, it is impossible to choose what version constraints to choose. <-- Edited to add this
Those are all less mature projects than setuptools. We are expecting setuptools to act like a mature project with ecosystem-wide dependents. |
For better or worse setuptools comes with a lot of legacy as it predates "modern" Python packaging standards, so in some cases it is less mature in supporting specific standards as it has taken the maintainers time to migrate. Flit is now over 10 years old, designed to be simple, and focus on standards: https://github.com/pypa/flit. If you are looking for stability, which is what I assume you are using "mature" as a proxy for, then flit might be a better choice for your own build system. |
We're depending on uv, but uv in a CI environment apparently installs the latest setuptools as part of the bootstrap process for its build environment. We depend on Are we seriously suggesting every end-user of Python audits all their third-party deps, figures out how to repackage all of those that don't provide wheels, self-host them, and set up CI processes to automate all that? As a simple end-user of Python, that sounds like a pretty crazy solution. Almost what's even the point of having a package manager and PyPI index/repo at that point? |
It doesn't, you should outline your scenario to uv and see what that's happening and fix it. If you create a project with uv the default build backend is currently hatchling (soon uv_build). Otherwise it should take whatever you've specified in your pyproject.toml.
It absolutely respects build dependencies, there's a bug that it doesn't respect |
Maybe I don't understand how this stuff works, it's certainly complicated enough. I think what I meant to write is that by default, uv builds all packages in isolated virtual environments, as per PEP 517. For pylatex and seemingly many many other projects that are distributed as sdists, this environment pulls in the latest setuptools as part of what I might incorrectly be labelling the "bootstrap" phase of building the module. This was reproducible locally if you simply blew away your UV cache and ran a
Apologies, I meant to write |
Ah, the point of confusion is you replied to my point saying for: "For your own projects", so I assumed you were talking about your own project, but it appears you're not. For installing other projects the build system is out of your control, other than in general it's better to install from wheels rather than sdists where possible, and then the buld system isn't involved. |
I feel compelled to link @hynek's brilliant article here: https://hynek.me/articles/semver-will-not-save-you/#taking-responsibility. Using P.S. @ianballard since you've created the issue using the blank template, you probably didn't see the CoC link present in other forms. So I'll leave it here for you: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md. I'm sure you didn't mean it, but your OP reads like an attack. Please, make an effort to interact constructively. This is no place for making demands towards maintainers that gifted you their free labor. |
Thanks for linking Hynek’s article—it makes a strong case for why clarity and predictability around changes matter, especially in a critical part of the ecosystem like this. I’m deeply grateful to the contributors who maintain and improve setuptools. I recognize the time, care, and effort involved—and that much of it goes unacknowledged. Your work is appreciated. I want to be clear that nothing in my original post was meant as an attack. The concerns I raised were serious because the impact was serious. Calling for transparency and accountability—especially in light of how dismissive some contributor responses were—is both reasonable and necessary. This change caused widespread breakage without clear timelines, migration paths, or safeguards. Raising that isn’t hostility—it’s about protecting trust and pushing for processes that match the project’s role in the ecosystem. Again, I truly appreciate the work that goes into this project. But as a community, we need to address how this was handled so it doesn’t happen again—especially the way concerns were dismissed after the fact. |
|
I oppose @ianballard's suggestion. I'd rather volunteers build whatever they feel like building without the burden of feeling that they owe anything to anyone. Without them there would not be any setuptools at all. These people should have good, honest fun and nothing else. |
Like @ianballard, I appreciate all of the work provided by the PyPA. Even so, I don't think this issue reads as an attack. It was well-reasoned, constructive, and calm. Especially so given the amount of disruption that was caused yesterday. Yes, there are serious concerns being raised. Those concerns do lead one to wish for changes. But how does one advocate for changes without stating what those changes are? Nevertheless, all of the questions are in many of our minds. Refusal to engage is one way of addressing them, and if that is what comes of it, then we know where we stand. As for the urgent requests, they are requests, not demands. If the PyPA feels they are unreasonable, or cannot be met with present resources, that is fair, we can have a discussion about it. What can we, as a community, do to help so that we can have the requests met? What can I do? I am willing to contribute. |
One good solution users who can afford to invest the time in it, is to trust updates less, if a faulty update from a volunteer project is going to cause you unacceptable losses, I strongly suggest you design your build system not to trust updates or to minimize the amount it has to trust. At my work we had no impact, even though several of our dependencies were affected, largely because we avoid sdists, cache built sdists, and use docker layers to cache all dependencies. I appreciate though this is not free, but if the cost of failing CI is high enough, it's probably worth it.
As part of the community are you going to organize, or provide, funding or resources to the setuptools maintainers to address these urgent requests and reviews? You haven’t clarified on that, and if not, what expectations do you have for volunteers to work on these urgent requests you've asked for? Maybe I'm misguided here and setuptools maintainers are happy to work on your urgent requests out of their own time, but when I request others do work for me urgently I come with a proposal on how to fund it. |
I did touch on this earlier, but to clarify: yes, I understand that maintainers are volunteers, and I deeply appreciate the work that goes into sustaining this project. I’m advocating for stronger processes around high-impact changes—especially when those changes break downstream projects, including setuptools’ own tests, as seen in PR #4909. Based on this, it seems like some of these processes are already in place so no extra work needed, but they were missed in this case. To move forward, we need to be honest about what happened so we can learn from it and make sure it doesn’t happen again. This isn’t about blaming individuals or holding volunteer maintainers to unrealistic standards. It’s about recognizing that critical infrastructure needs extra care, and that trust depends on how change is introduced and communicated. If resources are a blocker to improving those processes, I’d be happy to help facilitate that conversation. But we can’t just ignore the problem. |
|
Maybe I can clarify a little bit by saying what kind of governance changes I would like to see. I opened a new issue with a specific change I would like made about the deprecation in question. Can we get some open and honest discussion of that issue? |
Sure, let's talk about that.
It's hard to imagine "migration tooling" for the task of replacing a hyphen with an underscore in An "opt-out path" was indeed provided: not using a new major version of Setuptools. That's why Setuptools follows semver: to advertise breaking changes. Without the ability to make breaking changes, there is no hope of ever cleaning up legacy cruft in projects. I would wager that for the overwhelming majority of projects using Setuptools, only a small fraction of Setuptools code paths are relevant, or ought to be relevant. Setuptools is freshly downloaded ~20 million times a day for all these isolated build environments (it seems like local caches aren't doing their job a lot of the time, which is a separate issue) so being able to cut down the wheel size would save quite a bit of resources. Removing unnecessary code also tends to speed up Python projects, because importing it also has a cost. (If you don't believe me, try timing how much overhead the Many users struggled to fix broken builds because their transitive dependencies didn't specify a Setuptools version cap and they either couldn't restrict the Setuptools version their tools used for isolated build environments, didn't know how to set up a non-isolated build environment, etc. It's hard to say that this is Setuptools' fault. But that's for users who actually have to build those packages locally. And, as @notatallshaw pointed out, at least part of that problem is caused by package maintainers not pre-building trivial projects (e.g. ones that are completely in Python aside from dependencies) when they should. Which would also flag the issue for those maintainers. (They're the ones the deprecation notices are intended for in the first place!) (By the way, Damian: you say correctly that your local caching isn't free, but I can imagine it being less expensive in the long run than repeatedly downloading things and building them fresh - including new downloads for build dependencies. Not to mention the costs borne by others. Bandwidth isn't free either, and PyPI operates at petabyte scale - as I'm sure you're aware. Let me take this opportunity to say: thank you, Fastly!)
The underlying deprecation occurred four years (twice as long as was recently deemed appropriate for the Python standard library) and 24 major versions ago. How long is enough?
Yes. That's why it was explicitly labelled as a breaking change, and published following accepted standard practices for breaking changes (i.e., a bump of the major version number, in a project that advertises that it uses semver).
The response was initially dismissive because the scope of the problem wasn't immediately obvious. But the change (again: following up on a deprecation from four years ago) was reverted within less than six hours. In my book, actions speak louder than words - I find it hard to call that "dismissive". I didn't see anything from the developers that I'd consider unprofessional. I did see a ton of noise from third parties, typically in forms like "this also affects XYZ package", "this is outrageous", etc. None of that helps. (What does help is e.g. the comment that included search results for the now-broken key in
Organizations that would be severely disrupted by a bleeding-edge version of a package being unusable for their purposes for a few hours, ought to already know how to avoid having their tooling automatically pull in bleeding-edge versions. It could just as easily happen to their install-time dependencies, after all. (And in that case, they could spend considerable time on a build only to discover a problem in testing.) The same sort of thing happened with Setuptools 72. (It took a bit over 13 hours to revert that change, but it was overnight in North America.) And very little of the outrage seems to be getting directed at that tooling. Instead, the Setuptools issue filled with comments from people trying to help each other with workarounds for that tooling. I actually agree that the specific change in Setuptools 78 shouldn't have been made - at all, without a compelling case that not breaking it would get in the way of something useful. It's pretty trivial to leave in a few lines of code to normalize case, hyphens vs underscores, etc. Going forward, I'd like to see Setuptools batch together planned breaking changes and announce them. (Not that I'm particularly hopeful that the right people would pay attention to the announcements.) It seems a little ridiculous to me that we're on major version 78 of Setuptools, and trying to make changes like this only to be forced into reverting them is certainly a contributing factor to that. But the response I've seen so far looks like a pitchfork mob, and a largely misdirected one at that. @tiran :
Did this cause a significant problem? Having discovered in the test pipeline that the latest version of a build-time dependency fails to build a given package, is there not any system in place to roll back the build-time dependency? (I assume you are not reliant on Pip etc.'s build isolation.) Is there an organizational reason the package can't be built and released with the prior, known-working dependencies? After all: the package can't really be expected to know which (if any) future version of Setuptools will break on its current configuration data. The maintainers don't have a crystal ball. They can be conservative, but some people think that's a bad idea, even for build-time dependencies.
I assume you're talking about the person who runs
Once Pip succeeds in building a wheel, it's cached - so I can't fathom why people would be concerned with self-hosting wheels or setting up CI processes unless they already have those things. In the hypothetical world where, say, Requests doesn't ship a wheel, end users can obtain a wheel this way and install it in arbitrarily many virtual environments. |
At least for Gentoo, it is certainly possible to roll back the dependency. Actually it's a generalization of the appropriate caution to take with new versions of a foundational package which has an above-average tendency to have major versions with breaking changes that do, as it happens, break things. Simply put: have a globally consistent ecosystem of thousands of packages where every package is expected to work with the latest versions of all its dependencies, and when packages are not compatible:
This is necessary mostly not because of deliberate breaking changes, but because setuptools historically used the Monkeypatch Oriented Programming (MOP) model and now making changes tends to affect other components at a distance (plus literally all of its code is a public API for people writing setuptools plugins, and any package using setup.py counts as writing a setuptools plugin). So it's become relatively clear that any new setuptools version, whether it's a semver major or not, should be treated with caution and the default approach should be to deny it until it's been proven. (A distro can get away with this because a distro can do repodata patching. For example, Conda is a distro, and can do it as described at https://prefix.dev/blog/repodata_patching and https://bioconda.github.io/developer/repodata_patching.html. Some Linux distros use incremental patch updates to an entire package including the metadata, or simply rebuild that one package with the same version but fully fixed metadata. However, with PyPI it is not possible to patch metadata at all, only upload new versions, since every uploaded asset is immutable.) ... But, figuring out that there is a breaking change is still annoying to deal with.
The suggestion that build isolation is a problem that needs to be solved by ceasing its use, is humorous to me for a variety of reasons. |
You have alternatives! You could use Linux distributions to provide your dependencies! We did not make the setuptools version with the breaking change available at all, period, end of story. We went from 77.0.3 to 78.0.2, skipping the ones that broke the world. Our motivations are different from upstream: we need to keep all packages working together, so we delay upgrades until the ecosystem is ready, even if those upgrades are good ones. All the more so if the upgrades are bad upgrades. |
Have you ever heard of the concept of long tail? I think you are just trying to force your view without actually having the data to back it up. Google big query gives you that: https://console.cloud.google.com/marketplace/product/gcp-public-data-pypi/pypi I checked that from versions published today March 26th 2025 (110K) 5800 are sdist only. That is ~5% That is not even considering all the packages from the past that stopped updating. |
I don't think this is about the long tail at all. It seems to me that this is actually about "only consider the common use case". If we focus on only the top downloaded packages then we can optimize for the the packages that bill the most CI hours (which is indeed where wheels are the most important, due to reducing duplication of effort i.e. energy and time consumption from repetitively performing the same computations). The framing is pretty clearly not about whether packages are still in use today, that's the whole point of limiting to a subset such as "top one thousand". Note: I make no claims of my own about whether the association between CI hours spent, and willingness to make breaking changes, is a useful association. I'm just trying to analyze the claims made. |
pradyunsg
changed the title
|
I've taken the liberty of retitling this issue to not say urgent. |
|
With respect, @ianballard, if there are measures that you think would help to protect the ecosystem from a mistake like this, then a contribution of an implementation of one or more of those measures would likely be very welcome. Over in Rust land, there’s “crater” which builds a ton of packages to check the impacts of changes. Perhaps you could contribute a similar tool or infrastructure to run such a thing? Sounds like several folks in this thread already have building blocks in their CI. Or, you could organize a postmortem: collect feedback from the affected folks, analyze it, and provide guidance on the most valuable dependency chains to address. You could also proactively monitor the changes going into setuptools, looking for potential issues and verifying assumptions. Humans are human, after all. They don’t always get it right. If you truly think setuptools needs to provide stronger guarantees to the ecosystem, then the next step is to start to work on providing those guarantees. After all, setuptools was created because at one point someone tried to come up with a way to make it easier to package up Python software… |
|
@dwink that's been discussed a bit at #4920 in both senses. With specific regard to proactively monitoring the changes going into setuptools, this works out well for rust and is something I advocated there based on experience with Meson as well, but won't work at all for setuptools without actual process changes... because it assumes that people have time to monitor and react to those changes. Given a specific release schedule that spans a month and a half (or three months!) and incorporates beta or release candidate channels, it's possible to watch changes and do downstream testing and report results in time to, well, do something about it. Your options are extremely limited and amount to approximately very slightly better than "nothing", if the time from merging a new change to the time of sending that change out into production as a stable release ranges from "about a minute, because that's approximately how long it takes to switch to the tags page after clicking the merge button" to "sometime today, after I merge this other pending change I'd like to make as well". It's unworkable and untenable. If you want to monitor changes you can't do it like this. As a fallback plan you can monitor ALL submitted PRs, just in case some of them get merged. There will be a lot of noise if you do that, because you'll be spending most of your time reviewing changes that the project maintainers don't want to make or want to see major changes to, as well as discovering all the typos that were going to get fixed before merging anyway. |
I see the challenge but I suspect that IF such a tool existed, then a process change to use such a tool would be something to consider. It’s a bit chicken-vs-egg, but IME processes form out of what is available and what maintainers are willing to do. If they could take some clear actions and get a clear response to regular calls for testing, etc, then I doubt there would be a problem with adjusting the process. What I see is that the process currently expects the community to test new major versions before blindly adopting them (hence semver), but the community doesn’t and/or won’t do that — there’s an unwritten expectation that before major versions are released, broad validation is done and the results are communicated, more like a commercial product release. If you’re among those who want to have that work done, then step up and offer it, or provide software automating it, or something. |
There's no ask about any work being done. This strawman argument that users are demanding some perfect unbreakable software is just not what the issue is about. This issue is about having clear guidelines about what is and what isn't considered acceptable. Users use software because it's reliable, it's basically why Linus is so adamant about the "we do not break userspace" mantra, because that's what users want out of their software. Sometimes breakage is inevitable and justifiable, if there's something positive that comes out of it. However, in this case, it was just disrupting the entire ecosystem for nothing. There's absolutely no benefit that came from that change itself. The benefit that might come now is that build tools out there will hopefully realize they cannot rely on setuptools, and will provide mechanisms to specify the version used when building packages. |
|
I proposed an annual cadence for breaking changes in this issue. I think it is a small, viable change that can come of the discussion so far. |
The OP clearly demanded a list of things to be done.
Hold up, please reread what I wrote and reconsider what it says. The ask is to change how this software is released to fit different expectations from what the project has historically supported. I’m not arguing for or against any of that, but I am asserting that if you want to change something, you should expect to do and/or coordinate the work you want done.
Here is the what the project states:
I think it’s pretty clear what the policy is (Adopt new releases at your own risk), and stand by my assertion that if you want that to change, then you should work on establishing the improved approach by demonstrating it. Contribute what you want to see! |
I counter-assert that your stance is bizarre from beginning to end, and your own proofs are what prove you wrong. Your quotes clearly and unambiguously, in the strongest possible terms, demonstrate that it is categorically impossible to contribute what one wishes to see. There's nothing to change outside of a policy. The policy is what people want to see changed. One does not change a policy by "putting in the work", one changes a policy by persuading the policymaker to change it. |
Ah, okay. So once the new policy is decided, then maintainers can tackle adjusting how they choose what gets released, how to track changes that they aren’t releasing separately from the ones they are, handling the bitrot in those changes when the official time for changes comes around, handling the complaints from folks who really need/want this breaking change sooner because reasons… I’m sure it’ll be no work at all. Zero. |
|
I suggested a different policy here. One way to apply that policy without a great deal of extra changes is to release everything as beta except for the annual release. Obviously, it goes against this sentiment,
But after the impact on Monday, I don't think this is true. Instead, we should have beta releases. That allows consumers of |
|
None of that is work that can be "demonstrated" as part of a pitch to change things. So I'm baffled what you think your point is when you say
Am I saying that the setuptools maintainers must be persuaded by arguments that they should do those things? If so, I don't see where. Can you show me where I've said that? It may very well be that the things you point out are reasons why the setuptools maintainers don't want to change. There's nothing anyone else can do to influence that one way or another, aside for saying "I totally promise I'll stay around and be available to expend time on answering those questions and figuring out those changes, and you should believe my promise". What I don't understand is how you think you're helping the discussion by proudly lecturing people about how they should "simply" contribute the changes they wish to see and demonstrate that it works, as though it's just... a PR someone can submit. Do you really think that if someone could submit a PR to setuptools to
that the reason they haven't submitted such a PR is because they're unwilling to contribute and are simply complainers? Please, tell me where I submit such a PR. Is there a repository called I dunno, https://github.com/world-history/setuptools-users where I can submit PRs to the Or maybe I can just borrow Guido's Time Machine. :) |
That’s precisely my point. The “change” you’re asking for will require more thought and effort than just what shows up in the code. It takes ongoing effort, as time goes on and this particular incident fades in memory. If consumers of this package won’t pin versions and adopt updates incrementally in their own projects/pipelines as advised by the project, then I’m skeptical they will adjust to whatever annual/quarterly/whatever new process is defined. The breakage will just happen whenever that release comes out. So I’m suggesting that passionate folks invest in this project and contribute on an ongoing basis, helping the maintainers to avoid ecosystem breakage. Maybe that leads to release process changes, maybe new announcements, who knows? It depends what you end up doing. |
|
I tend to think that submitting a pull request is often helpful to compliment a discussion. It can demonstrate the discussion points in a more visual way. But OTOH, its acceptance would still be blocked on review/discussions. In general, I don't think that setuptools is in a position to dramatically change their release process, since it's a huge paradigm change. And the underlying technical problem isn't necessarily with setuptools itself, but with the lack of ways to pin versions in ephemeral build envs that aren't hacks. I'd call for a more organized effort on standardizing mechanisms for pinning build dependencies on the user side. There's also a separate discussion on checking important downstreams in the ecosystem: #4920. I think, it may be a valuable tool to let the maintainers know of some bits of impact automatically. This is something that the maintainers seem to be open to considering, but somebody has to put together a comprehensive demonstration. And I believe this one could be a PR, but it should start small until the initial review happens. On a related note, I recently read https://alexgaynor.net/2021/oct/07/whats-in-a-version-number/ and like how it puts the problem being faced here:
And so I don't believe that making maintainers to jump between versioning schemas is sustainable in any way. Such changes are inevitably connected with additional maintenance burden. And that's exactly why there's frequent releases. Giving the users tools to manage pinning better would be far more useful and reach beyond just a single build backend. |
That’s a distinction without a difference. Simply treat all new major versions (X.0.0) of setuptools as “beta”, and it is identical to what you’re describing. |
|
Hi everyone. Just thought I'd check in. I read some of the thread earlier this week, but didn't have the energy to provide a response. I have limited bandwidth, and I still haven't read the full thread. I appreciate the spirited and civil conversation. Let me say first, I hear you. This kind of disruption isn't acceptable and we'll work to avoid it. We have a particularly difficult landscape to manage, with a codebase that dates to 1998 and users that span packagers, consumers, and integrators. We do our best to limit disruption while also aiming to pay down some of the tech debt and reduce the interface surface to improve quality. I personally was under the impression that this change was likely to be low impact and limited scope for disruption, based on our work over the years to develop a more sophisticated deprecation methodology. I expected that projects that were likely to be impacted would have been flagged previously, so worst-case only old packages would be affected. Obviously, this missed the mark, so I'm adjusting my understanding of how much confidence we can put behind the current deprecation process. There may also have been some judgment calls that should be revisited. For example, when I cut the release, I was primarily looking to release a different, stabilizing change and this other change came along for the ride. In the future when cutting such a release, I'll take more care to monitor closely such a potentially destabilizing change (as I often do for other such changes) or ensure someone else is. I'll plan to coordinate with abravalheri for other possible improvements as well as to attempt to ingest the suggestions above and associated issues. Thanks for your patience on this matter. |
This is entirely false. Betas have a strictly defined meaning. They mean that tooling will not install them unless you do something such as pass the --pre flag. Or in distro land, betas may be packaged but they get packaged to the experimental stream, not the testing stream (Debian terminology) or unkeyworded rather than ~arch (Gentoo terminology). Again, those will only be installed to environments that opted in to beta testing. The science ecosystem has a dedicated extra index url you can use to get access to "scientific-python-nightly-wheels". Similar notion. It's fully compatible with any pip workflow even, but requires using an index other than PyPI. Of course, beta releases are typically used by projects that release relatively infrequently, where releases are a big deal to downstream consumers, where projects are developed by whole teams, or some combination thereof. I suppose some people would like to argue that pip releases are a big deal... |
There is no such definition in the relevant PEPs. See here. The tooling supporting Meanwhile Debian experimental isn't called "beta", (it's All that said, let's go back to the idea of development releases, which folks opt-in to in order to get the latest and greatest setuptools, and everyone else just waits for the "fully-tested" versions. What happens is that the most sensitive end users only use the "fully-tested" versions. Thus, their issues still get hit when the change makes it to the "official" release, because getting a critical mass of testers for low-level 'plumbing' code like setuptools is...unlikely to happen in practice. So everyone just waited longer to find out who was impacted, and...can the fixes go in to the new release, or do those go through the "beta" too? Oh, well since that fix is going to the main one and skipping beta, can we get this one other thing in there too...? I get it, lots of folks have CI that uses pip, lots of abandoned packages have old setuptools setups and the story around pinning setuptools for transitive dependencies is less than ideal, and no one likes CI just breaking. Still, the maintainers clearly care and respond quickly -- and are able to do so in no small part because of, and not despite, their model. Ideal or not, maybe it's the best we can realistically get? Or maybe folks can help, with or without a policy change, by researching and doing legwork to better protect the ecosystem. Isn't that the whole point? |
I suspect you will find that the concept of beta software being synonymous with development releases, predates the invention of the python programming language. I don't see what PEPs have to do with it, except that the PEP you quoted specifically states that the term "developmental release" is a specific variant of non-python development release, and is different from the separate group of "alphas, betas, and release candidates", which are denoted as "pre-releases". In a separate part of the document it then violates this distinction, by stating:
That's the "--pre" handling which you claim is specific to "development releases", but unlike your claim, the PEP specifically says that it applies to the word tag "beta". So yeah, you're wrong. The PEP specifically reinforces the software development paradigm of beta releases, just like I said... :)
No idea what PPAs have to do with anything.You're the only one mentioning them. And nowhere did I say that Debian experimental is called "beta", please don't put words in my mouth. I said that betas can be packaged in Debian experimental and that people who opt in to Debian experimental are also thereby opting into betas. I know this is the case because we (Meson) do exactly this. Part of the Meson development team includes the Debian maintainer for the Debian meson package, and part of the release process of Meson is to package pre releases in Debian Experimental. Note that I'm specifically not talking about sid, which isn't experimental, it's "unstable". I mean genuine honest to goodness "experimental". Given we actually do it, I think that counts as pretty solid evidence. Don't see how much better evidence you could ask for.
I already explained this quite clearly. Given a beta release and a couple weeks of lead time I can personally arrange to have thousands of popular as well as unpopular packages tested against setuptools beta releases. This testing will be done without regard to what the developers of ${package_that_uses_setuptools} feels comfortable doing. It will be done using Linux distribution resources, by people who have spent decades being quite willing to do their own testing, bug reporting, and bug triaging, and then come to upstream projects with patches in a significant percentage of cases, and can also do things like report to the setuptools project "this change breaks XXX number of packages in our testing, which seems quite high, so maybe it would be worth reverting? What do you think?" As I'm both a distro person and a developer of upstream software I'm quite familiar with the possibilities and I know the right strings to pull, even with distros that I'm not associated with. It's genuinely mind-blowing the kind of help and support you can get if you just... go out and... ask. And are willing to consider the possibility that people whose whole organizational existence is centered around large scale integration, might be willing to work with you to ensure your own integration issues can be worked out. I'll go one step further and say that when I reached out as an upstream developer of software to distros I'm not affiliated with and asked them if they could run mass testing for me, they were profusely grateful to me and thanked me for helping them. |
|
FYI, currently pip (and I don't think any other installer tool) passes So you would need to set the environmental variable I'm working on a feature to support selecting individual packages to enable or disable prerelease: pypa/pip#13221. But again I hadn't planned this to affect build backends unless you use it as environmental variables. There is a workaround now, if you include the requirements or constraint |
I went in and looked at how it's currently done. The Lines 61 to 70 in 6ead555 That script uses This is used to determine which semver level to change and ultimately produce a new version number: Then, this version number is applied by running Lines 45 to 47 in 6ead555 So, to implement the "prerelease" idea, you'd need to find a spot to apply your prerelease version suffix (maybe a Or, to implement the "separate index" idea, you'd need to adjust the release tox target to push elsewhere for an "experimental" release. I'll go back to what I said earlier about contributing what you want to see, because a PR or two with the above changes and a description of the adjusted workflow you're describing would make it perfectly clear what you're asking for_and_ make it easy for maintainers to adopt. It could likely all be done in a way that could be tried out -- even publishing one prerelease version and then you getting folks to run run their tests on it -- before it has to impact anyone else at all. |
|
@dwink note that most automation changes would have to go through jaraco/skeleton. |
|
Cool, so I I can try to legislate away a social issue with code changes and have my PR closed because it's not even in the right repository. I'll go back to what I said about needing to discuss large workflow changes in issues before doing the trivial bit twiddling to enable them. If even the trivial config changes needed to switch to a drastically different social model are no longer trivial due to requiring social organization "across projects* then I can't fathom how "just submit a PR, bro" can be a valid criticism of people submitting issues. |
Is that what would happen? Have you tried? If that did happen, you’d know where you stand, right?
Okay, okay, you win, I give up. You’ve got some idea of how whatever-this-is must happen, I don’t understand it, and I no longer have any interest in understanding it. Best of luck. |
|
Very long and pretty useless discussion with a lot of opinions and no actionability. What I suggest everyone is to take active part in this PEP https://peps.python.org/pep-0772/ including discourse thread. Randoom issue created because a problem happened is not going to solve long term problems better than deliberate discussion and decision based on PEP how oversight and decisions (but also funding and getting involvement of community) should happen. |
The recent PR #4870, authored by @abravalheri and approved by @jaraco, introduced a breaking change that immediately and broadly disrupted the Python ecosystem by disallowing deprecated dash-separated and uppercase options in setup.cfg. This was merged despite clear acknowledgment that it would break downstream projects:
"if we are feeling brave. But it can break packages that have not addressed the warning."
— @abravalheri
"I'm inclined to say we should do it, even though it will cause some disruption."
— @jaraco
This change effectively broke builds across the ecosystem with no grace period, runtime fallback, or migration tooling—blocking unrelated emergency patches and releases in critical production systems. The follow-up PR #4909 that removed requests from integration tests—because this change broke internal usage—further illustrates how unprepared the project itself was for the consequences of this decision.
Bug #4910 showed just how widespread the issue was.
After widespread backlash, the change was reluctantly reverted in #4911, but when asked if an opt-out would be provided for a future reintroduction, the author responded:
“The option is to pin setuptools via the PIP_CONSTRAINT environment variable... All things considered this is likely a symptom of something bigger, more problematic, and the package you depend upon has been on borrowed time for a while... Organise among other users contributions (or even a fork) for maintenance. Fund the original developers for maintenance. Find a suitable replacement.”
— @abravalheri
This dismissive response shifts responsibility entirely to users and downstream projects without acknowledging the maintainers’ own role in ecosystem stability. It shows a fundamental lack of empathy for maintainers and users who depend on stability and predictable deprecation cycles.
This raises serious questions about:
We urgently request the following:
The Python ecosystem relies heavily on setuptools, and changes of this magnitude cannot be driven by personal opinion or experimentation—especially when knowingly disruptive.
This incident has eroded trust. Rebuilding that trust will require serious introspection and reform.
The text was updated successfully, but these errors were encountered: