"pip freeze" conflates git stdout and stderr

[godlygeek]

Environment

• pip version: 20.1
• Python version: 3.8
• OS: Linux

Description
pip freeze runs git under some circumstances. When it does, winds up calling subprocess.Popen with stderr=subprocess.STDOUT, capturing stderr as well as stdout, which potentially leads to failures to parse the output reported by git on stdout.

Expected behavior
stderr is ignored except as needed to report diagnostic information, and only the output sent to stdout is parsed by pip.

How to Reproduce

python3.8 -m venv venv
. venv/bin/activate
pip install --upgrade pip
mkdir package
cd package
echo "from setuptools import setup; setup(name='mypkg', version='1.0', packages=[])" >setup.py
pip install -e .
git init .
GIT_TRACE=1 pip freeze

Output

$ python3.8 -m venv venv
$ . venv/bin/activate
$ pip install --upgrade pip
<snip>
Successfully installed pip-20.1
$ mkdir package
$ cd package
$ echo "from setuptools import setup; setup(name='mypkg', version='1.0', packages=[])" >setup.py
$ pip install -e .
<snip>
Successfully installed mypkg
$ git init .
Initialized empty Git repository in /home/mwoznisk/package/.git/
$ pip freeze
# Editable Git install with no remote (mypkg==1.0)
-e /home/mwoznisk/package
$ GIT_TRACE=1 pip freeze
ERROR: Command errored out with exit status 128:
 command: git rev-parse HEAD
     cwd: /home/mwoznisk/package
Complete output (5 lines):
16:59:41.297650 git.c:439               trace: built-in: git rev-parse HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
HEAD
----------------------------------------
WARNING: Error when trying to get requirement for VCS system Command errored out with exit status 128: git rev-parse HEAD Check the logs for full command output., falling back to uneditable format
WARNING: Could not determine repository location of /home/mwoznisk/package
## !! Could not determine repository location
mypkg==1.0

$ git add setup.py
$ git commit -m initial
[master (root-commit) c9e7373] initial
 1 file changed, 1 insertion(+)
 create mode 100644 setup.py
$ GIT_TRACE=1 pip freeze
-e git+git.c:439@17:21:25.131888 git.c:439               trace: built-in: git rev-parse HEAD
c9e7373d99756fb1abd2c626876205969af19448#egg=mypkg

Where we encountered this bug, it wasn't because of the GIT_TRACE debugging environment variable - it was because of a warning being printed by the dynamic linker to warn us that the version of libz.so being picked up by the git binary didn't match the version it was built against (roughly this). GIT_TRACE just makes for a convenient way to reproduce the issue; diagnostics printed to stderr are being confused with command output and mistakenly parsed.

Some historical context: it looks like call_subprocess was updated to pass stderr=subprocess.STDOUT about 6 years ago in 0cb2991, in order to allow filtering tracebacks out of the reported error stream. That traceback filtering was removed about 4 years ago in f9429b0, but stderr was left redirected to stdout after that change.

[sbidoul]

Is this a variant of #7545?

[godlygeek]

Looks like it is; I didn't catch that one while searching.

[uranusjr]

Let’s use #7545 as the canonical tracking issue, since it contains more discussion on this issue. I’ve also edited its title to reflect this.