Pip3 upgrade overwrites /usr/local/bin/pip


(Anand Buddhdev) #1

Hi folks,

I’ve installed the python@2 and python formulae. My /usr/local/bin/pip symlink points to python2’s pip.

However, if I now run “pip3 install --upgrade pip”, as is suggested by the python formula, it overwrites /usr/local/bin/pip with a wrapper script that calls python3’s pip, because pip itself is unaware of brew and its symlinks. Similarly, upgrading “setuptools” and “wheel” using pip3 overwrites the symlinks in /usr/local/bin. I can’t be the first one to have run into this issue that causes confusion.

Is this a known problem? And if so, what’s the advice? Is it to not upgrade pip, setuptools and wheel?


(Sean Molenaar) #2

It’s generally advisable to stay with one system for version management. Either let homebrew manage setuptools/pip/whatever versions, or do it using pip with only one python version, otherwise it’ll still mess things up.


(Anand Buddhdev) #3

Hi Sean. Thank you for the reply, and it’s what I would do. However, the formula explicitly tells that user:

Blockquote
Pip, setuptools, and wheel have been installed. To update them run
pip3 install --upgrade pip setuptools wheel

Do you think we should remove this advice from the formula?


(Mirko Friedenhagen) #4

I think removing this from the Formula would be good. I always use virtualenv with Python to avoid desasters like this.


(Sean Molenaar) #5

I think that’s a good idea. Can you create a pull request for that?


(Inada Naoki) #6

Does it mean python formulae is updated every time when pip, setuptools, and wheel are updated?


(Anand Buddhdev) #7

No. The pip, setuptools and wheel modules are only installed afresh when the python formula is updated. This means that they can be out of date for a while. But that’s probably okay, because one doesn’t always need the latest versions all the time.


(Inada Naoki) #8

pip shows warning when new version of pip is available.
I think we should update Python formulae when it happens.
Of course, it wouldn’t happen so often.


(Anand Buddhdev) #9

That would probably lead to unnecessary updates of the python formula.

I personally don’t use the pip, setuptools and wheel installed by the formula. The trouble with using brew-installed pip2 and pip3 is that you can’t use them to install the same python package, if that package also provides wrapper scripts for the “bin” directory. As an example, if you first use pip2 to install something like “netaddr” or “ansible”, then those modules will install scripts in /usr/local/bin. Next, if you use pip3 to install the same modules, while their code will be installed into the python3 lib directory, their wrappers will also be installed to /usr/local/bin, and overwrite the ones placed there by pip2. This is a mess.

My strong opinion is that the brew python formulae should not install pip, setuptools or wheel at all, because there is no sane way to install modules using pip2 and pip3 under the current setup.


(Inada Naoki) #11

I use different approach; never use Python 2.

I use venv for projects. But I use direct pip install for tools daily used, not bound to project.


(Anand Buddhdev) #12

You are right, and “ensurepip” is indeed the best of doing this. This is why I don’t like the brew formula suggestion to upgrade pip with “pip install -U pip”. That messes things up, and that’s why I created the pull request to remove that instruction. However, my pull request has not yet been accepted.

Having said this, the python formula also installs setuptools (which provides the easy_install script) and wheel (which provides a wheel script), and these also get overwritten depending on which pip you use.


(Inada Naoki) #13

I’m sorry, I was wrong and I removed my post.
python3 -m ensurepip -U upgrades to “bundled” pip, not latest pip available from pypi.
So python3 -m ensurepip -U can not be replacement for pip install -U pip.


#14

Hi everyone,
I have a related issue. My underlying ‘problem’ seems to share its causality with what has been discussed with regards to @aabdnn 's problem.
I use MacOs Terminal.

tl;dr

before:
$ which python
/usr/bin/python (macos preinstalled)

$ which python3
/usr/local/bin/python3 (homebrew)

$ pip --version
-bash: pip: command not found

$ which pip3
/usr/local/bin/pip3 (from brew python3 formula install)

then I run (as recommended per official brew Python docs)
python3 -m pip install --upgrade pip
note: I used python3 not python as per literal documentation

afterwards
$ which pip
/usr/local/bin/pip

$ which pip3
/usr/local/bin/pip3

both pips are the same version

then I remove the /usr/local/bin/pip file, as it was not there before my pip upgrade and it annoys me.

things go back to as before

Question1: blabla - own answer: I will not upgrade pip manually again, but rather wait for brew python formula update.

Question2: I deleted that /usr/local/bin/pip file that seems to have been created (against my will or expectation) after running the pip upgrade.
My pip and pip3 command seem to work, or not work in the case of pip, as was prior to the upgrade. But was this file removal good clean practice or am I set for trouble later on? Edit: I recall that the file was an executable called pip and it was the newest version.


(Anand Buddhdev) #15

The situation is more complicated. If you install both python2 and python3 from homebrew, then /usr/local/bin/pip points to the pip installed with python2, while pip3 points to the pip installed with python3. If you now upgrade pip using pip3, then it will switch /usr/local/bin/pip from a symlink to the python2 pip, to a symlink to the python3 pip. Just removing it will break your ability to run python2’s pip. So you need to recreate that symlink. There are so many subtleties with the python2 and python3 formulae in Homebrew, that I have decided never to upgrade anything with pip. Instead, I create a virtualenv, and use that to do all my python work. I love python, but the transition from python2 to python3 is a mess, and complicated further by the various packaging systems, each one of which does things differently.


#16

Hi. Ok, so in my case I had no python2 or pip in my usr/local/bin, rather only python3 and pip3. What seems to happen is that when I run python3 -m pip install --upgrade pip somehow creates an executable called pip in usr/local/bin alongside the brewed python3 bootstrapped pip3. Does that make sense?

I understand your problem. I am new to Python, and I admit I spent quite some time now wrapping my head around how I should best go about package management. I also put all my python packages in their own venvs now. Seems like the safest best practice.

I would be interested to know: is there a convention for naming virtual environment directories. From reading online it seems like a standardised name for all my venvs on my machine would be ‘venv/’. However, would it not be better to name them according to the project’s name, so as to avoid confusing which venv is currently activated? How do you name your venvs?


(Anand Buddhdev) #17

Hi, yes normally when you install pip under python3, it creates both a pip3 as well as a pip command. When installing python3 using homebrew, both pip3 and pip are installed, but homebrew does NOT link the unversioned “pip” command into /usr/local/bin. This is because at the moment, the advice from the python community is for unversioned commands to point to python2. If you also install python2 using homebrew, then you get “pip2”, but you also get the unversioned “pip” pointing at pip2.

If you now try to upgrade either of the pips, using just “pip install -U pip”, then they are unaware of homebrew, and they will both install an unversioned “pip” in /usr/local/bin. This is what causes breakage and confusion. Therefore, you should generally NOT upgrade the homebrew-installed pip, setuptools or wheel.

As you can see, there is a lot of subtle stuff here, and I don’t know if there is a smart way of handling this better in homebrew. I’m afraid to criticise the current setup, because the homebrew people take offence at this, and think I’m just criticising for the sake of it. At least they accepted my suggestion to remove the advice in the formula about upgrading pip. But if anyone still does upgrade pip, it ends up with unexpected effects. I’m sorry to say that python2 and python3 co-existence under homebrew is not as good as it could be. I’m also annoyed that homebrew’s “python” formula actually installs python3, whereas everything else is inverted, meaning that the “python” binary points to python2. I have never understood this inverted logic, and I’ve seen many of my colleagues confused by it.