Description
UPDATE 15.12.2020 6pm CET:
After releasing PIP 20.3.3 today we were able to make 2.0 compatible with the new PIP and 1.10.14 almost works (papermill extra is problematic when installing airflow using the new PIP). We will try to address it in case we release 1.10.15 but if you want to install papermill extra, please downgrade pip or use legacy resolver.
While with 2.0 it seems that airflow can be installed with new PIP following our recommended practice, in case you see any installation problem please report them as issues and downgrade to pip 20.2.4 as a workaround.
Thanks again to the PyPI team for the fast resolution (just in time for the 2.0 release).
We leave the issue open for a while but we updated the description and lowered the priority. We will close it once we have observed installations from our users after 2.0 is released and confirm that the problem is solved for our users.
UPDATE 15.12.2020 11am CET:
Seems that with the latest 20.3.3 release and fishing pyarrow dependency we are back in business with 2.0.0rc3.
Once we confirm it and verify 1.10.14 we will be able to close that one!
Thanks to the PYPI team for quick solving it.
I am adding this issue to keep track of the on-going problems with new PIP 20.3 released 30th of November.
There are multiple issues with the new PIP that makes it breaks with Airflow's dependency set.
The first blocking issue is pypa/pip#9203 and pypa/pip#9232.
The latest version of PIP @master is still not usable with Airflow:
- Multiple setuptools and wheels downloaded during installation with PI 20.3.2@master pypa/pip#9297
- PIP install few orders of magnitude slower in PIP 20.3.2@master than 20.2.4 pypa/pip#9298
- "check" does not consider extras and allow environments not possible to re-create with new resolver pypa/pip#9299
Even when those is solved we already know we are affected by a few other problems:
- Installing packages with different egg names fail in pip>=20.3 pypa/pip#9217
- New resolver downloads hundreds of different package versions, without giving reason pypa/pip#9215
- New resolver takes a very long time to complete pypa/pip#9187
- pip 20.3 https://pypi.org SSLError WRONG_VERSION_NUMBER pypa/pip#9223
We've raised the issue to the PIP team and they struggle with fixing a number of other teething problems.
We keep fingers crossed that they will manage to fix the issues promptly and that they will not be overwhelmed with putting out the fire.
There is no resolution yet, so for the time being downgrading PIP to 20.2.4 version is the best thing you can do.
pip install --upgrade pip==20.2.4
We raised the issue pypa/pip#9231 with the proposal of change to PYPI to add an exclusion list to PyPI and we are waiting for their response.
UPDATE! Tested the current master version of PIP (which has been yesterday announced as candidate to 20.3.2) but it still does not solve installation problems with airflow:
Three new issues created:
- Multiple setuptools and wheels downloaded during installation with PI 20.3.2@master pypa/pip#9297
- PIP install few orders of magnitude slower in PIP 20.3.2@master than 20.2.4 pypa/pip#9298
- "check" does not consider extras and allow environments not possible to re-create with new resolver pypa/pip#9299