GAnarchy support

[SoniEx2]

What's the problem this feature will solve?

With the removal of dependency links, it's now impossible to use pip with GAnarchy. GAnarchy is a decentralized project hub with the goal of enabling "zero-ownership" projects (OT: I need to update that project commit/create a new one), whereas PyPI is a package index requiring "full-ownership" projects. This makes them inherently incompatible, but that shouldn't be a reason to thwart/kill GAnarchy projects.

Describe the solution you'd like

I'd like something that is not dependency links. Instead, you give it a project commit (e.g. 385e734a52e13949a7a5c71827f6de920dbfea43 for GAnarchy itself), a GAnarchy instance's machine-readable index (e.g. https://ganarchy.autistic.space/config.toml) and the version/revision you want (e.g. f84c60e039864af593f6b8d015b11113a6e59f50 for one of many GAnarchy commits). It would then fetch the repos matching the project commit and search them for the version, and give the user the option to pick a different repo if they so desire.

Note that GAnarchy instances generally publish a machine-readable index at their /config.toml, but this can be customized and the recommendation is to let the user specify the path to the config.toml. Also, the user should be allowed to have many of these set-up, and sort them in whatever order they prefer, as not all instances are expected to reference all projects.

I use GAnarchy and I can't split my (own) dependencies into another GAnarchy project because there's no way of including those as dependencies.

Alternative Solutions

One could publish their packages to PyPI. This completely defeats the purpose of GAnarchy.

Bringing back dependency links would also work. However, they were removed for a good reason.

Additional context

I just want my projects to work and I don't wanna downgrade pip.

[pfmoore]

I'm not clear what you're asking for here. Pip follows PEP 440 for defining package versions, so users request something like foo==1.0. Commit IDs don't conform to that definition, so I don't see how you expect them to work (short of proposing a new PEP to extend PEP 440 with a new identification scheme).

On the other hand, direct references from PEP 440 are intended to offer a URL-based way of depending on another project. Do they help your use case?

[RonnyPfannschmidt]

devpi just recently added light mirroring, which enables a devpi to link to the owned resources on pypi instead for self-serving, this might work for GAnarchy as well

[SoniEx2]

@pfmoore oh. I can use that in setup.py? (altho ideally there would be the flexibility of versioning on the commit history but I can live without that for now, I guess.)

[uranusjr]

It would be much easier to recommend alternatives if you could describe more concretely how you used dependency links with GAnarchy. Some concrete commands that people can run would be much helpful (since it is likely we would be able to provide concrete replacements).

[SoniEx2]

honestly I started using pip long after dependency links got removed, but from what I've read they'd suit my needs...

or part of them at least. they'd let me hardcode git repos. altho I think I read somewhere that downstream users can override those? in which case I'd just use semver and stuff with my packages and it would actually mostly suit my needs if I am understanding things correctly.

I'm not sure if PEP 440 VCS links support versioning and overriding like that.

[uranusjr]

Well, in that case, how do you think it’d suit your need?

I think the problem is that no-body here is particularly familiar with GAnarchy, so it’s difficult to know what exactly you plan to use pip with it. Personally I am quite sure there is a way (pip is extremely versatile), but not quite sure what can help you, in what ways.

Maybe it’d be a good idea for you to downgrade pip and figure out a way for dependency links to work first? That way we would have concrete examples to work with.

[SoniEx2]

so, the basic idea is the ability to substitute forks.

if I have, say, https://soniex2.autistic.space/git-repos/ganarchy.git and https://github.com/ganarchy/GAnarchy (git repos that can be cloned), I want the user to be able to substitute in any of them wherever a "ganarchy" dependency is requested. does that make sense?

(ofc, a prerequisite for that is the ability to use git repos as dependencies in the first place. but anyway.)

[pfmoore]

Not really. If project A depends on a GAnarchy URL, how would pip know what other URLs are equivalent? If pip doesn't need to know, then the user can just edit the source code for A and replace the existing URL with their preferred one, and it's not a pip issue.

Basically, a fully manual solution just involves direct URLs and users having to manually edit files. An automatic solution is impossible because there's no way to know which URLs are equivalent. I don't see a middle ground. But equally, I don't see the point of GAnarchy if it's that simple, it's nothing more than some URLs. So I suspect I am missing something critical.

I want the user to be able to substitute in any of them wherever a "ganarchy" dependency is requested

By that do you mean that if the user says pip install ganarchy then pip would (somehow) know which URL to fetch? The only way pip does that is via a package index, and you've already said that the package index model doesn't suit your needs. Or is it just the way PyPI implements a package index that's the issue? Have you tried writing a PEP 503 compliant index that does what you want?

[SoniEx2]

I mean like, the user would "pip install ganarchy @ some-repo", or equivalent, and it'd replace the "ganarchy" dependency on all dependencies if the version requirements are compatible. (ideally it'd also look for a compatible version, but GAnarchy's index is "somewhat" different from your usual package index, so in practice it'd fall on the user to figure out the dependencies (but pip should still warn the user of incompatibilities).)

I don't think one can make a PEP 503 compliant index for use with GAnarchy. an ideal solution would basically have to search every commit and every setup.py in the repo's history for a compatible version. this seems outside the scope of PEP 503.

[SoniEx2]

Basically, this is what it'd do, to try and resolve the dependencies:

for commit in git_log:
    setup = commit.get_file("setup.py")
    if process_setup(setup).check_version(wanted_version):
        return commit

that is, for every commit in the repo:

1. grab "setup.py" as seen on the work tree of that commit
2. check if the metadata of that setup.py is compatible with what we want
3. if so, return the work tree

then install the resulting work tree as per its setup.py.

if your project has some dependencies that depend on different forks of the same project (ideally specified by the GAnarchy project commit), then just filter the git_log to be compatible with all of them. (there are many algorithms for sorting DAGs)

[habnabit]

So, it seems like there's nothing concrete decided on yet. @SoniEx2 I'd assume that you're not opening this issue to ask the pip team to work with you to decide on a specification or write code for you (since they're very busy as is)—maybe you should turn the code in #7591 (comment) into a real proof of concept (i.e. runnable) implementation and close this issue until you have something much more substantial to work from.

(Even then, projects like pip-tools and poetry exist happily outside of pip; writing your own frontend to pip even means that you can support many more versions of pip with the same tool. There's even PEP 518 entry points which have been supported for several versions now. No need to add extra maintenance burden to pip for a niche use-case.)

[SoniEx2]

one thing you could do with dependency links was replace them downstream. and you can't do that with PEP 440.

dependency links were (mostly) a fallback mechanism, not an authority mechanism. you could version dependency links just like any other dependency and downstream users could then replace them with any compatible versions. this would be more than good enough as far as "GAnarchy support" goes.

[habnabit]

@SoniEx2 sure. None of that addresses what I said, if you intended it to.

[SoniEx2]

okay: I need a mechanism to override dependencies. no more, no less. something to workaround the fact that PEP 440 was not designed with something like GAnarchy coming along and being used with python.

ideally it'd support versioning just like dependency links did.

(in other words my solution is "bring dependency links back" tbh. does this count as concrete?)

[altendky]

What aspect of the dependency needs to be overridden? The distribution package name? The acceptable versions? Where to acquire the dependency? There's a lot going on, we need to be very specific. Also, how would someone be expressing the override?

[SoniEx2]

Dependency links allow you to specify where to acquire the dependency. And this "where" can be easily overridden downstream, by simply installing from the alternative location, while still checking acceptable versions and the distribution package name.

PEP 440 removes a lot of that.

[pradyunsg]

(in other words my solution is "bring dependency links back" tbh. does this count as concrete?)

Yes, and it's not going to happen. See #5898 (comment)

I'm gonna close this out since I don't think there's anything actionable here.