Pip install --upgrade pip breaks pip when installed with Homebrew

This post is close to Homebrew’s pip3 shortcut via pip has hard-coded version dependency causing errors.

Environment

  • pip version: pip 19.2.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
  • Python version: Python 3.7.4
  • OS: MacOS Mojave 10.14.5 (18F132)

I use the Homebrew packet manager: Homebrew 2.1.89-07-23

Description

In the original brew install python installation, /usr/local/bin/pip3 is a symlink. It eventually reaches its target file: /usr/local/Cellar/python/3.7.4/bin/pip3 .

The brew info python help recommends to use unversioned symlinks at /usr/local/opt/python/libexec/bin : /usr/local/opt/python/libexec/bin/pip is a symlink to /usr/local/Cellar/python/3.7.4/bin/pip3 .

pip install --upgrade pip creates a new file /usr/local/bin/pip instead of of updating the target of the symlink, that is, /usr/local/Cellar/python/3.7.4/bin/pip3 . Moreover, this latter file is not updated.

As the PATH finds /usr/local/opt/python/libexec/bin/pip before /usr/local/bin/pip , everything fails…

Expected behavior

On updating pip , /usr/local/bin/pip should be a symlink to /usr/local/opt/python/libexec/bin/pip .

How to Reproduce

  1. Clean off all the pip 's
    find -E /usr/local/ -type f -iregex '.*/pip[23]?' -print -delete

  2. Reinstall python with Homebrew
    brew reinstall python

  3. Check the status of /usr/local/bin/pip*, etc. and look for the similar i-nodes.
    NB: Note there is no /usr/local/bin/pip.
    ls -li /usr/local/bin/pip*
    ls -li /usr/local/opt/python/libexec/bin/pip
    ls -li /usr/local/opt/python/libexec/bin/../../bin/pip3
    ls -li /usr/local/bin/../Cellar/python/3.7.4/bin/pip3

  4. Upgrade pip with pip
    /usr/local/opt/python/libexec/bin/pip install --upgrade pip

  5. Redo Step 3 above.

  6. Try out
    /usr/local/bin/pip --version

  7. Try out
    /usr/local/opt/python/libexec/bin/pip --version

Output

pommeau:~ bouge$ find -E /usr/local/ -type f -iregex '.*/pip[23]?' -print -delete

/usr/local//bin/pip3
/usr/local//bin/pip
find: /usr/local//etc/FileWaveInstallers/FlatPackage-Microsoft_Office_2019_VL_Serializer.pkg: Permission denied
find: /usr/local//etc/FileWaveInstallers/FlatPackage-Microsoft_AutoUpdate_4.3.18090901_Updater.pkg: Permission denied
find: /usr/local//etc/FileWaveInstallers/FlatPackage-Microsoft_Office_16.17.18090901_Installer.pkg: Permission denied
/usr/local//Cellar/python/3.7.4/bin/pip3

pommeau:~ bouge$ brew reinstall python

==> Reinstalling python 
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.4.mojave.bottle.
Already downloaded: /Users/bouge/Library/Caches/Homebrew/downloads/ffcfe300e2ad205387a2bdd27185d98d64621578d26d2f6ccbbcd091ff799fc8--python-3.7.4.mojave.bottle.tar.gz
==> Pouring python-3.7.4.mojave.bottle.tar.gz
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run
  brew install python@2

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python/3.7.4: 3,869 files, 60MB

pommeau:~ bouge$ ls -li /usr/local/bin/pip*

17870036 lrwxr-xr-x  1 bouge  wheel    34B  4 mar 14:11 /usr/local/bin/pip2@ -> ../Cellar/python@2/2.7.16/bin/pip2
17870037 lrwxr-xr-x  1 bouge  wheel    36B  4 mar 14:11 /usr/local/bin/pip2.7@ -> ../Cellar/python@2/2.7.16/bin/pip2.7
36689213 lrwxr-xr-x  1 bouge  wheel    31B 24 jul 15:54 /usr/local/bin/pip3@ -> ../Cellar/python/3.7.4/bin/pip3
36689214 lrwxr-xr-x  1 bouge  wheel    33B 24 jul 15:54 /usr/local/bin/pip3.7@ -> ../Cellar/python/3.7.4/bin/pip3.7
12675428 -rwxr-xr-x@ 1 bouge  wheel   263B 16 jan  2019 /usr/local/bin/pip_upgrade_outdated*

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/pip
36689211 lrwxr-xr-x  1 bouge  staff    14B 24 jul 15:54 /usr/local/opt/python/libexec/bin/pip@ -> ../../bin/pip3

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/../../bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/opt/python/libexec/bin/../../bin/pip3*

pommeau:~ bouge$ ls -li /usr/local/bin/../Cellar/python/3.7.4/bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/bin/../Cellar/python/3.7.4/bin/pip3*

pommeau:~ bouge$ /usr/local/opt/python/libexec/bin/pip install --upgrade pip

Collecting pip
  Using cached https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.1.1
    Uninstalling pip-19.1.1:
      Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.1

pommeau:~ bouge$ ls -li /usr/local/bin/pip*

36690799 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 15:55 /usr/local/bin/pip*
17870036 lrwxr-xr-x  1 bouge  wheel    34B  4 mar 14:11 /usr/local/bin/pip2@ -> ../Cellar/python@2/2.7.16/bin/pip2
17870037 lrwxr-xr-x  1 bouge  wheel    36B  4 mar 14:11 /usr/local/bin/pip2.7@ -> ../Cellar/python@2/2.7.16/bin/pip2.7
36690800 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 15:55 /usr/local/bin/pip3*
36690801 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 15:55 /usr/local/bin/pip3.7*
12675428 -rwxr-xr-x@ 1 bouge  wheel   263B 16 jan  2019 /usr/local/bin/pip_upgrade_outdated*

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/pip
36689211 lrwxr-xr-x  1 bouge  staff    14B 24 jul 15:54 /usr/local/opt/python/libexec/bin/pip@ -> ../../bin/pip3

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/../../bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/opt/python/libexec/bin/../../bin/pip3*

pommeau:~ bouge$ ls -li /usr/local/bin/../Cellar/python/3.7.4/bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/bin/../Cellar/python/3.7.4/bin/pip3*

pommeau:~ bouge$ /usr/local/bin/pip --version

pip 19.2.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

pommeau:~ bouge$ /usr/local/opt/python/libexec/bin/pip --version

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pip 19.2.1 (/usr/local/lib/python3.7/site-packages), Requirement.parse('pip==19.1.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/python/libexec/bin/pip", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip==19.1.1' distribution was not found and is required by the application

I also posted a bug request in the pip site: https://github.com/pypa/pip/issues/6784. They recommended to post at Homebrew, instead. So did I. Luc.

pradyunsg commented [11 minutes ago]

Could you file an issue with Homebrew, linking to this one?

I think they’re in a better position to address this than pip’s maintainers are. We’ll be happy to work with them to resolve this if there’s anything they think upstream pip should do.

I saw exactly the same issue with the same flow of events after pip install --upgrade pip. Thanks for the detailed report.

I am relatively new to Python, so this work-around based on your description of the problem may be naive, but it worked for me:

➜  code git:(master) cp /usr/local/opt/python/libexec/bin/pip /usr/local/opt/python/libexec/bin/pip_ORIG
➜  code git:(master) mv /usr/local/bin/pip /usr/local/opt/python/libexec/bin/pip
➜  code git:(master) ln -s /usr/local/opt/python/libexec/bin/pip /usr/local/bin/pip
➜  code git:(master) pip --version
pip 19.2.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

I opened an issue for this on Homebrew’s GitHub Issues.