Improve UX and Performance of Install step

[notatallshaw]

What's the problem this feature will solve?

At the moment when the final install step starts pip gives no output what it is doing, in some real world cases (e.g. large pytorch installations or airflow installs) this steps can take over 30 seconds on fast machines, so minutes on slow machines. The user is left wondering if anything is happening.

Describe the solution you'd like

I would like to see the following improvements:

• 1. Log a message that pip is starting to install packages
• 2. Present a progress bar that tracks the number of packages installed out of the total packages to be installed
• 3. Improve any obvious performance bottlenecks (see follow up post with profile)
• 4. Run installs in parallel (made seperate issue Install packages in parallel #12742)

Alternative Solutions

I think at a bare minimum there should be a log message that lets the user know what's happening.

Additional context

uv runs installs in parallel, and following their issue tracker it does not appear to be problematic, to do this a cli option to control the maximum number would need to be added, the same as how there is a PR for parallel downloads to do this.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

This scenario is artifically constructed to best profile the installer code by removing the need to download, build sdists, or resolve:

1. python3.12 -m venv .venv
2. source .venv/bin/activate
3. <install latest/dev pip>
4. wget https://raw.githubusercontent.com/apache/airflow/constraints-2.9.1/constraints-3.12.txt
5. python -m pip download -d downloads -r constraints-3.12.txt
6. cd downloads
7. for file in $(ls *.tar.gz); do pip wheel --no-deps "$file" && mv "$file" "$file".built ; done
8. for file in $(ls *.zip); do pip wheel --no-deps "$file" && mv "$file" "$file".built ; done
9. cd -
10. python -m pip install --only-binary=:all: --no-index --ignore-installed --no-deps --find-links file://${PWD}/downloads -r constraints-3.12.txt

I ran with and without --dry-run to see the timing difference:
Dry Run: 32s
Regular Install: 144s

I profiled with and without --dry-run to see the profile difference:

Dry Run Profile

Regular Install Profile

There are some clear hotspots here, I will take a look when I have time if there are some easy ways to reduce those hotspots if no one else does.

[ichard26]

The get_dist_name() hot spot should be vastly improved by #12656 FWIW. I scheduled the PR for 24.2 as it feels a bit risky to ship in 24.1 final. Please say something if anyone feels differently.

[pfmoore]

I see no issues with the UI proposal, but I'd want parallel installs to be a separate feature. I can imagine pathological cases where things could break when installing in parallel, and while the experience of uv is encouraging (as is the fact that normal cases are clearly safe) my instinct is that every pathological case is being exercised by some user of pip, somewhere. So we should isolate the risk here by making it a separate feature.

[notatallshaw]

The get_dist_name() hot spot should be vastly improved by #12656 FWIW. I scheduled the PR for 24.2 as it feels a bit risky to ship in 24.1 final. Please say something if anyone feels differently.

Great, I'll reprofile with this PR. I personally wasn't imagining any of these ideas would land for 24.1.

I'd want parallel installs to be a separate feature

Agree, I'll make a seperate issue for that.

Honestly, the others I feel like I could make PRs that safely improve pip, I'm unsure about parralel installs, I think it would at a minimum carefully need to look at what current multiple installs tests there are and potentially expanding them to have a good matrix of different possibilities.

[notatallshaw]

Log a message that pip is starting to install packages

Btw, I was looking at this recently because I noticed pip does tell you it's installing packages. The specific scenario I was seeing was the following:

1. You install a large number of packages
2. You then install a large number of semi-overlapping packages

On step two this produces the following behavior:

1. Packages are resolved and pip tells you what packages it is going to install
2. Pip then quickly uninstalls old packages, filling up the screen
3. There is a long wait with no update on the screen while pip is installing
4. Pip then lists all packages it installed

The real world situaiton this happens is installing large machine learning packages, particularly because you install a bunch of packages from the pytorch index, and then install a bunch of packages from pypi.

I think there are a couple of possible solutions:

1. Re-order or add additional messages, e.g. move or add and "install" message after the uninstalls have completed
2. Add progress bars to both uninstalling packages and installing packages, so it's clear pip is doing things

I will take a look at PRs when I have a chance.

[ichard26]

Caching the result of utils.compatibility_tags.get_supported() in the resolver factory should be another easy win¹ (~3% or 4s in the example above)

pip/src/pip/_internal/resolution/resolvelib/factory.py

Lines 608 to 612 in 86b8b23

	return self._wheel_cache.get_cache_entry(
	link=link,
	package_name=name,
	supported_tags=get_supported(),
	)

I'll submit a PR when I get the chance.

Footnotes

1. I strongly suspect that get_supported() is only "slow" (as in, 1-5ms) on Linux due to the large amount of supported tags per system. ↩

[ichard26]

While taking a look at #12601, I was curious to how easy it would be to add an installation progress bar. The progress bar was pretty trivial to add by extending the pre-existing progress logic... However, it did not play nicely with the logging stack, so any intervening logs would break the progress bar. To fix this, I had to redo how rich was initialized in the logging stack which took a bit 🙂

Anyway, here's a demo:

Screencast.from.2024-07-15.22-31-35.webm

What do you think @notatallshaw?

• I'll note that your mental model for how pip installs packages is wrong. The uninstalls occur "on-demand" right before its replacement package is about to be installed (i.e. the uninstalls/installs are interwoven), so an uninstallation progress bar doesn't really make sense.
• I also chose to include the package currently being installed in the progress bar. Yes, in most situations, the per-package installation time is so low that most packages are never shown to the user (like in the demo), but there are exceptions. If we're installing some massive package, it'd be nice to let the user know we're stuck on $package. I don't feel strongly about this though so I'm fine dropping it.
• I chose to hide the progress bar after the installation step since the "successfully installed ..." printout is going to be emitted anyway.
• The progress bar does feel a bit choppy. This is intentional. I set the progress bar refresh rate to 5 times per second to minimize the added overhead. I don't want a fancy progress bar to add a ton of overhead, haha.¹

Footnotes

1. Ideally, the presentation logic would simply disable the progress bar outright when writing to a non-TTY, but that's a future thing to think about. ↩

[ichard26]

Hmm, it would definitely look less rough if I left-justified the package name. Here's another demo, but the package name is justified to the longest name length seen so far (as doing it properly feels like going against the API contract of pips' progress bars).

Screencast.from.2024-07-15.23-06-07.webm

It does kinda look weird. Perhaps after the bar?

Screencast.from.2024-07-15.23-11-40.webm

I think this looks the best out of all of them :)

[notatallshaw]

I'll note that your mental model for how pip installs packages is wrong. The uninstalls occur "on-demand" right before its replacement package is about to be installed (i.e. the uninstalls/installs are interwoven), so an uninstallation progress bar doesn't really make sense.

Ah, I see, my confusion is that is how pip's current logging displays what is happening. It logs all uninstalls, and then logs what packages it has installed, sometimes there can be a significant time between the last uninstall message and the install message, giving this impression.

I also chose to include the package currently being installed in the progress bar. Yes, in most situations, the per-package installation time is so low that most packages are never shown to the user (like in the demo), but there are exceptions. If we're installing some massive package, it'd be nice to let the user know we're stuck on $package. I don't feel strongly about this though so I'm fine dropping it.

I agree, if you want to try large packages installation where individual packages will be noticable you can do: pip install torch torchvision torchaudio

I think this looks the best out of all of them :)

Yes, I think anything left of the progress bar should be fixed width, and ideally not updating at all. At least in left to right English having the left hand side update feels like I need to keep rereading the whole line, but the right hand side updating just feels like I need to look at the right hand side to check updates.

Once you have a PR I'm happy to throw some difficult scenarios against it.

[notatallshaw]

Okay, since I opened this issue there's been a lot of improvement to install performance of a lot of wheels, here is my synthetic test:

1. python3.12 -m venv .venv
2. source .venv/bin/activate
3. <install latest/dev pip>
4. wget https://raw.githubusercontent.com/apache/airflow/constraints-2.9.1/constraints-3.12.txt
5. python -m pip wheel -w wheels -r constraints-3.12.txt
6. time python -m pip install --only-binary=:all: --no-index --ignore-installed --no-deps --find-links file://${PWD}/wheels -r constraints-3.12.txt

On pip 24.1.2:

real 2m23.338s
user 2m8.488s
sys 0m12.523s

On pip main (effectively 24.2 right now):

real 1m23.565s
user 1m11.482s
sys 0m9.681s

Here is the new call graph:

In this synthetic example ~50% of them time is now spent on O(n²⁺) issues in resolution and ~50% of the time is spent doing wheel specific stuff. It feels like both have algorithmic or caching opportunities. When I get a chance I will take a look.

[notatallshaw]

In my synthetic test I notice ~30% of the time is spent on compile_file. I notice that the standard library compile_dir will create a process pool when it can to speed things up, I wonder if it makes sense to use compile_dir on each root package directory installed, and then verify the pyc files are created as expected?

[notatallshaw]

FYI, I beleive most other installers "optimize" this step by not compiling by default.

[ichard26]

Nope. My free time is rather limited. I'll get around to it some point.

[tyler-suard-parker]

@ichard26 I would like to build a status bar for the installation step, because I want to be a part of such an important project, even in a small way. Before I start working though, would my changes get pushed? Is this needed?

[notatallshaw]

@ichard26 I would like to build a status bar for the installation step, because I want to be a part of such an important project, even in a small way. Before I start working though, would my changes get pushed? Is this needed?

Pip is a volunteer project, so good quality PRs can be accepted from anyone. However someone from the pip maintainer team will need to review and approve it, and there is fairly limited resources to do that.

So, if you want to develop a high quality low complexity PR that forwards the aims of this issue I'm quite sure it would be appreciated, but please be patient at reviews capacity tends to be quite low, and pip is a critical open source project so there is a tendency to be conservative in acceptance.

[ichard26]

As an interim solution before we (ever?) implement parallelized installs, more experiments with parallelizing bytecode compilation should be done.

While @notatallshaw's simple attempt to leverage compileall.compile_dir()'s built-in parallelization (#12907) did result in some speed-up, it isn't an ideal solution. Each call to compile_dir() creates a new multiprocessing pool, meaning a cycle of subprocess initialization and finalization occurs for each package (AFAIU). This is slow, and doubly so on Windows. Actually, on my Linux box, the PR appears to be slower than no parallelization, too.

I propose that we manage the parallelization ourselves, using multiprocessing to spawn a set of compilation subservers (subprocesses) that grab files to compile from a queue. They'd be created at the start of the install step to avoid paying the subprocess creation/finalization cost over and over again.

There are likely a number of fiddly details to figure out, so it's possible this won't be feasible¹, but I'd like to experiment with this approach since bytecode compilation is often the bigger contributor to install time.²

Footnotes

1. Even this may not be enough to significantly speed up Windows which seems to have truly massive subprocess initialization overhead :( ↩

2. In addition, this would be a good way to test the waters with parallelized installs. Parallelized bytecode compilation is (hopefully) unlikely to cause problems even in odd scenarios, but it would still expose any general problems with parallelization. If this lands and causes the sky to fall, then we know that parallelized installs themselves are probably infeasible. ↩

[ichard26]

I have a prototype based on my earlier proposal: https://github.com/pypa/pip/compare/main...ichard26:pip:perf/parallel-compile?expand=1

• It's based off Installation progress bar ✨ #13220 so the improvement in install time is easier to feel
• To disable parallelization, set PIP_SERIAL envvar to 1
• To modify how many workers it uses, set the PIP_WORKERS ennvar (it defaults to 4 or system logical CPU count, whatever is lower)
• In addition to the hacks listed above, I also added a cyan log that reports the total installation time (it's this number I'm reporting below)

It reduces the installation time of numpy and matplotlib and of the demo in #13220 by ~roughly half. I'm on an Ubuntu 24.04 machine, with CPython 3.12.4 and 16 logical CPU cores (Ryzen 5800HS).

Command: pip install matplotlib numpy --find-links temp/wheelhouse/ --no-index --ignore-installed

• Before: 3.106s
• After: 1.486s

Command: pip install -r tests/requirements.txt -r ../mypy/test-requirements.txt --find-links temp/wheelhouse/ --no-index --ignore-installed

• Before: 1.865s
• After: 1.016s

@pfmoore could you try this branch and report whether this makes installation faster on your Windows machine? You can install the branch via:

pip install https://github.com/ichard26/pip/archive/perf/parallel-compile.zip

[pfmoore]

With pip 25.0.1

❯❯ pip list
Package Version
------- -------
pip     25.0.1
❯❯ hyperfine -p "python -m pip uninstall -y numpy" "python -m pip install numpy"
Benchmark 1: python -m pip install numpy
  Time (mean ± σ):      6.467 s ±  0.053 s    [User: 1.434 s, System: 0.934 s]
  Range (min … max):    6.358 s …  6.537 s    10 runs

With your branch:

❯❯ pip list
Package Version
------- ---------
pip     25.1.dev0
❯❯ hyperfine -p "python -m pip uninstall -y numpy" "python -m pip install numpy"
Benchmark 1: python -m pip install numpy
  Time (mean ± σ):      3.972 s ±  0.132 s    [User: 0.981 s, System: 0.763 s]
  Range (min … max):    3.767 s …  4.111 s    10 runs

That's about 60% of the time, so pretty much in line with your results. I did get quite a lot of variation before my "unpatched" test settled down, so there's obviously other factors involved as well, but your patch is a clear win.

[ichard26]

Awesome, thanks @pfmoore. It's good to see that the idea has potential. Now, this is still going to be a non-trivial change:

• pip doesn't have any parallelized logic. It used to¹, but it doesn't today. While I imagine parallelization of bytecode compilation specifically is unlikely to cause issues², parallelization in general is a recipe for odd bugs and edge cases:

  • For example, on Windows, you can't create than 63(?) subprocesses using a Pool. If the system reports having 128 logical cores³, then pip is going to crash when it tries to create a pool with 128 cores
  • The system has X cores, but the user (and thus) pip can only use a portion of those cores. Attempting to use all of the cores would simply result in worse performance. os.process_cpu_count() is the solution here, but that was only added in Python 3.13.
  • The Python install may lack multiprocessing support (e.g. Android or AWS Lambda) or its implementation is subtly broken in a way that's difficult/impossible to detect ahead of time
  • For bytecode specifically, it turns out that bytecode compilation is not deterministic as refcounts are included (Bytecode compilation output depends on order of files compiled python/cpython#129724). Now, AFAIK, pip install is already not deterministic, but it could be that this part of pip install is currently deterministic, so this proposal would break it.
  • Parallelization, despite the tuning I plan on doing (see point 3), could be slower than doing everything serially in certain situations
  • Parallelization runs the risk of resource exhaustion generally, especially with RAM

• We're gonna need an opt-out: for reasons mentioned above, we're going to need an escape hatch so pip doesn't break unnecessarily. I'd love to simply use the standardized PYTHON_CPU_COUNT environment variable which os.cpu_count() and friends respect... but as you may have already guessed, it's another Python 3.13+ feature. Thus, a --workers flag is needed, aka more UI design work 🙃

  • An extension of this is that pip needs to be smart enough to automatically fall back to serial compilation when parallelized compilation is unsupported (when it's possible to check)

• We should strive to avoid slowing down small installs: the fundamental problem with process-based parallelization is that creating new subprocesses is slow.⁴ I'm talking 200+ ms worth of overhead (on Linux, Windows is likely even worse) before the first package file can be installed. This only gets worse with the more subprocesses we spin up (the current maximum is 4, but I'd like to raise it).

  • There has been a lot of effort (including on my end) in making pip more snappy (reducing import-time, deferring expensive logic that be done later) and faster in general. It would be a shame for pip install six (from a local wheel) to take 450ms when it used to take 300ms.

  • The workaround is add a pre-install step that inspects the packages (zipfiles) to be installed and measures how much Python code (by size and by file count) and decide whether to use serial or parallel compilation (and how many cores to use). In a perfect world, this calibration would occur on the end machine, but that sounds like a nightmare to maintain. My plan is to write a script to find the sweet spot where parallelization is a net positive, run that across the variety of platforms I have, and hardcode the cutoffs.⁵

In other words, this is going to take a while. I want to see this through, but I don't want to break the world and/or wake up to a bunch of complaints that this actually made pip slower, so this has to be done right.

Footnotes

1. In pip list --outdated/--uptodate. We removed it when dropping Python 3.7 support. It was brought up in PR review that we should bring it back, but that suggestion was never followed through upon ↩

2. With my current proposal, the packages are still compiled sequentially. It's just for each package, its contents are compiled using a long-lived pool. This should(?) stepside any tricky issues where files are overwritten by multiple packages. ↩

3. Which is not totally unreasonable with the rising popularity of server CPUs with 16/32/64 cores ↩

4. On Linux, where multiprocessing.Pool defaults to the fork method, the overhead is not that bad (20-30ms on my system), but AFAIK, the default is changing to forkserver in Python 3.14 as the fork method tends to break/deadlock. forkserver is more efficient than the spawn method (already the sole option/default on Windows and macOS) at creating a large number of processes, but it still involves a single costly spawn to create the template subprocess for all of the secondary subprocesses. ↩

5. Of course, this is an imperfect science. I plan on erring on the safe side and use the higher-end of the cutoffs I collect as IMO avoiding slowdowns is more important than extracting the most performance for large installs. If people want even more performance, they can use uv. ↩

[ichard26]

Annnnd the hacks keep on stacking. It turns out that multiprocessing imports the main module (__main__) while initializing new subprocesses. This is necessary so any global state in the parent process is also reflected in the subprocesses.

If the pip console script is used, it's the main module that's re-ran. This means pip._internal.cli.main is re-imported. This module unconditionally imports a ton of stuff and is slow to import.

#!/home/ichard26/dev/oss/pip/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

However, if pip is invoked as a package, then pip.__main__ is re-ran, and this time, the costly main entrypoint import is gated by a if __name__ == "__main__" check.

import os
import sys

# Remove '' and current working directory from the first entry
# of sys.path, if present to avoid using current directory
# in pip commands check, freeze, install, list and show,
# when invoked as python -m pip <command>
if sys.path[0] in ("", os.getcwd()):
    sys.path.pop(0)

# If we are running from a wheel, add the wheel to sys.path
# This allows the usage python pip-*.whl/pip install pip-*.whl
if __package__ == "":
    # __file__ is pip-*.whl/pip/__main__.py
    # first dirname call strips of '/__main__.py', second strips off '/pip'
    # Resulting path is the name of the wheel itself
    # Add that to sys.path so we can import pip
    path = os.path.dirname(os.path.dirname(__file__))
    sys.path.insert(0, path)

if __name__ == "__main__":
    from pip._internal.cli.main import main as _main

    sys.exit(_main())

In other words, multiprocessing is faster when pip is invoked via python -m pip vs simply pip. On my system¹, this change results in ~200 less imports during subprocess initialization, saving ~100ms (out of 160ms). This is major discrepancy.

To achieve the same low overhead of python -m pip, we can temporarily replace sys.modules["__main__"] with any random module that does (functionally) nothing.

        # HACK: multiprocessing imports the main module while initializing subprocesses
        # so the global state is retained in the subprocesses. Unfortunately, when pip
        # is run from a console script wrapper, the wrapper unconditionally imports
        # pip._internal.cli.main and everything else it requires. This is *slow*.
        #
        # This module is wholly independent from the rest of the codebase, so we can
        # avoid the costly re-import of pip by replacing sys.modules["__main__"] with
        # any random module that does functionally nothing (e.g., pip.__init__).
        original_main = sys.modules["__main__"]
        sys.modules["__main__"] = sys.modules["pip"]
        try:
            # ctx = multiprocessing.get_context("spawn")
            # self.pool = ctx.Pool(workers)
            self.pool = multiprocessing.Pool(workers)
        finally:
            sys.modules["__main__"] = original_main

An overhead of 60ms on Linux is still too high for me to be okay with using parallelization all of the time, and I still have no idea what the overhead looks like on Windows (or macOS, for what it's worth). The cutoffs can likely be a lot simpler, however.

Footnotes

1. I should note that I'm on Linux using Python 3.12. On this platform, multiprocessing defaults to the fork process creation method which is very fast. However, I am not using these numbers for comparison because Windows/macOS don't use it already, and AFAIK Python 3.14 is going to change the default method to forkserver anyway. ↩

[ichard26]

I'm still working on my parallel bytecode PR, but I've been also thinking about further parallelization we could add (especially as my preferred name for tuning how many bytecode workers pip uses is --install-jobs). I took another look at #12816 and it got me considering parallelizing zip decompression/extraction¹ as well.

That PR does actually result in worse performance on stock Python due to the GIL, however performance does improve significantly on a free-threaded CPython build. There are other problems with that PR though.

• I strongly suspect a large part of the performance uplift on free-threaded Python is due to the parallelization of bytecode compilation, which I'm already addressing separately.
• It takes the entire Requirement.install() method and runs that across multiple threads. This is a lot of code! Code that we haven't been validated or was written with thread-safety in mind.
• Because it's farming out the individual requirements to the thread pool, requirements are going to be installed in a random order. While I'm not sure how deterministic the current install order is, it does seem to be stable across back to back re-runs. I'd like to avoid messing with the install order because that opens up a whole can of worms to deal with overlapping/conflicting packages.
  • There are also potential performance problems with this approach too. If numpy and six are farmed out to two separate threads, the thread processing six will complete in nearly no time while the numpy thread will take a hot minute. This is, evidently, not very efficient.

My approach of spinning up a pool of bytecode compiler workers that is reused across Requirement.install() calls avoids the last three downfalls. In particular, the actual amount of code being parallelized is quite small, reducing the chance of strange errors.

While zip extraction makes up a comparably much smaller portion of the install time, it is still sigificant. We're already going to be paying the startup penalty of spinning up a pool of workers for bytecode compilation, we might as well reuse that pool for zip decompression/extraction too.

Footnotes

1. Decompression is CPU-bound while extraction is a mix of CPU-bound decompression and IO-bound file read/writes. I'm purposefully not choosing one as I haven't done any experiments yet. It may make sense to also handle the IO operations in the pool, although ... thinking about it more, parallelizing I/O does seem more precarious. It's probably best to defer that. ↩

[ichard26]

To be clear, I have no idea whether parallelizing the zip decompression/extraction using a long-lived pool is going to be beneficial or not. Zip decompression does generally complete much faster than bytecode compilation, so any serialization/IPC overhead from submitting decompression jobs to the pool is much more significant.

I do think it's worth looking to though. I'll write a PoC when I get the chance. Once the parallel bytecode PR and potentially a parallel decompression PR are landed, I'd say this issue will be essentially complete. There are likely some smaller micro-optimizations that remain (avoiding redundant packaging.Version construction, optimising distribution metadata lookup, etc.), but I'm not too worried about those.

[ichard26]

(I accidentally posted my comment before it was fully written.)

I spent some time parallelizing zip extraction today. My initial attempts did not produce a measurable performance improvement. The first hurdle is that the ZipBackedFile instances which contain the extraction logic store the open ZipFile ... which is a file object. File objects can't be pickled, and thus the zip file would have to be opened on the worker side. Taking the easy route of opening a fresh ZipFile for every file to be extracted results in much worse performance because the zip metadata/file list is inspected upon ZipFile creation. The solution is to only open the zip once per worker, but that means managing persistent worker state.

In addition, while I'm not a huge fan of the current SerialCompiler() and ParallelCompiler() design, switching the implementation to use a generic worker pool (with a in-process serial fallback) also results in sub-optimal code. The key problem is that the function that may be parallelized needs to live its own module and avoid relying on as little as pip's internals as possible (i.e. it should import very little) so the start-up penalty doesn't blow up. While this is doable for bytecode compilation, it is not as easy for zip extraction (unless I want to refactor how the install code works), and it's probably even less practical for the other things we'd want to parallelize (e.g., wheel building).

Given that zip extraction—in comparison to bytecode compilation—is a smaller contributor to install time, I'm going to defer these problems. I am honestly sick of experimenting and rewriting this branch. I'd quite like to get this clear performance improvement landed sooner than later.

At some point, we should look into parallelizing zip extraction and refactoring the bytecode parallelization code to be more generic/reusable, but that'll be a future us problem.

[ichard26]

Also, if anyone is following my stream of consciousness here, I'll note that my comment (#12712 (comment)) complaining that concurrent.futures.ProcessPoolExecutor is slower is incorrect. While it does have more per-task overhead, it is so small that it won't matter for anything that benefits from parallelization. I'm actually using the concurrent executors as they have better performance characteristics for our workload.

[morotti]

hello, I am seeing the thread pinged my other PR where I added support for parallel extraction #12816

I'm pretty sure all my benchmarks were done without bytecode compilation. I have it disabled in all environments with the env var. it's too slow.

if you want to do a comparative benchmark. the quicker way to install for CI and builds is to disable wheel compilation with PIP_COMPILE=0 then run at the end of the build python -m compileall --workers 8 $VIRTUAL_ENV/lib/python3.8/site-packages/

if you want to run benchmarks, one of the most common and largest packages in terms of files is plotly with around 15000 python files to compile. one of their guys spent months refactoring every function into a single file :D

I spent some time parallelizing zip extraction today. My initial attempts did not produce a measurable performance improvement. The first hurdle is that the ZipBackedFile instances which contain the extraction logic store the open ZipFile ... which is a file object. File objects can't be pickled, and thus the zip file would have to be opened on the worker side. Taking the easy route of opening a fresh ZipFile for every file to be extracted results in much worse performance because the zip metadata/file list is inspected upon ZipFile creation. The solution is to only open the zip once per worker, but that means managing persistent worker state.

I've debugged that in another PR (30% faster extraction). The zipfile class is extremely slow, it's reopening the file and doing small read/seek operations and acquiring thread locks between every operations. #12826

There is no improvement to be made by using multiple threads to extract one zipfile because of the locks. The parallelization needs to be done one level above, having different threads extract different zip files.