Multiple config scripts and potential Homebrew software installation confusion

I just installed Homebrew onto my Mac and entered brew doctor in Terminal.

Everything seems to be working fine but I am curious about the Warning message that I received:

Warning: “config” scripts exist outside your system or Homebrew directories. ./configure scripts often look for *-config scripts to determine if software packages are installed, and which additional flags to use when compiling and linking.

Having additional scripts in your path can confuse software installed via Homebrew if the config script overrides a system or Homebrew-provided script of the same name. We found the following “config” scripts:

/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7-config
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m-config
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3-config

Although things are fine now, I am concerned that maybe, as warned, a future installation of software via Homebrew might encounter issues due to these additional scripts.

What should I do? Can I or should I erase some of the config files outlined by the warning?
For example, would it be okay for me to just keep:

/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7-config

And delete:

/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m-config
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3-config

Or should I just leave things untouched? Thank you for your advice and guidance. :slightly_smiling_face: :pray:

Remove this from your $PATH.

1 Like

Thank you @MikeMcQuaid! :slightly_smiling_face:

Hi @MikeMcQuaid, thanks again for your help. However, as I am using Atom as my editor, ever since I deleted /Library/Frameworks/Python.framework/Versions/3.7/bin from my$PATH various linters no longer work (no longer on my $PATH). I was wondering if there was another way to do this besides erasing that line from my .bash_profile? Thank you. :pray:

You could install the same python version from homebrew (brew install python) and add that to your path (would probably be there already)

Or, what you can do is:

  1. Put /Library/Frameworks/Python.framework/Versions/3.7/bin back in your PATH;
  2. Type pip freeze > /tmp/my-python-stuff.txt to create a list of what you have installed;
  3. Remove /Library/Frameworks/Python.framework/Versions/3.7/bin from the PATH again;
  4. As @SMillerDev suggests, install Homebrew Python via brew install python
  5. Ensure the Homebrew Python binary is on your PATH (try which python to confirm)
  6. Once Homebrew Python is satisfactorily set up, type: pip install -r /tmp/my-python-stuff.txt to reinstall whatever your earlier Python setup (linters and all) had previously had set up.

Good luck @Kusine !

1 Like

Wow. Thank you @fish2000 for your helpful and detailed step-by-step guidance. Awesome. Thank you also @SMillerDev for your advice.

So when I install Python via homebrew, it will automatically uninstall the current Python which I installed from https://www.python.org ?

Thank you again for your helpfulness. :pray::slightly_smiling_face:

1 Like

@Kusine Naw – it won’t uninstall anything. The Python.org version will sit there until you uninstall it manually, which you probably won’t have to do as long as there are no references to that Python version’s install tree in e.g. your PATH, PKG_CONFIG_PATH, PYTHONPATH (if you have one) or any stray scripts, aliases, and other shell-ish miscellany.

1 Like

Ok, thanks for confirming @fish2000 – shouldn’t I uninstall the python.org version to avoid a second copy that I will not use since I will be using the Python installed from homebrew going forward? In addition to avoiding any potential funkiness that might arise from:

Thanks for your patience as I try to gain a better understanding of things. :slightly_smiling_face:

1 Like

@Kusine your instincts are correct, it is better to remove the unused Python installation. But first, you’ll want to use it to dump out a list of the packages you’re using. So basically what you want to now endeavor to do are these things:
a. First do all of this stuff, and then
b. Remove the offending Python-dot-org package.

Yes!

1 Like

Awesome! Thanks a ton @fish2000!

Final question (if I may :slight_smile: ), is there any particular advantage of having Python installed via homebrew vs. python.org (besides resolving my original issue)? Why would seasoned programmers like yourself choose to install Python through homebrew instead of from the “source” at python.org (I assume they are identical)?

I hope that I haven’t committed a faux-pas by asking this question on the homebrew community site. :smile:

1 Like

For me it’s because I’m very lazy and prefer to install everything from one place with an easy CLI.

1 Like

OK so: there are always personal, professional, stylistic, and inertial reasons to use one version of Python versus another. I’ll tell you briefly about how I personally see and use the macOS system Python, the Python.org Python, the various and sundry Homebrew Pythons, and a few others like Anaconda’s Python.

Fortunately, a given version of python – that is to say, a Python installation with a version number of x.y.z – will work the same as any given Python x.y.z install, provided they are both installed and invoked correctly. At least, as far as the core language and standard library goes… what packages each Python has available to it can and will vary greatly, unless you spend a lot of time ensuring that it doesn’t.

So with that caveat, let me explain. In the beginning, when I started programming in Python around 2005, I used the macOS system Python, which at the time was called the MacOS system Python (note the capitalization difference). The version was either 2.3 or 2.5, I can’t recall. The pip installer was not yet a thing – Python packaging was a terrific topic under which one could start a fantastic array of flamewars on listservs, or early Reddit or Digg (as StackOverflow was also not yet bethingified, and Hacker News was a small core cadre of sycophants). The executable was located at /usr/bin/python, and packages were installed to /Library/Python/2.5/site-packages. I didn’t know anything about Python stuff – I didn’t know why .egg files were terrible, or how native extensions were built and linked, or what the deal was with distutils and setuptools… or, most egregiously why you never ever want to mess around with Apple system binaries.

I used this Python because it shipped with PyObjC installed. PyObjC is a bridge between the Python and the Objective-C runtimes. It is very old (it has been in development since Steve Jobs was at NeXT), very thorough, and very vast – it’s a constellation of interdependant Python packages that, at the time, had to be compiled from source to be properly installed. Since I had a vested interest in using PyObjC and no idea how to install it from scratch, I took to using the system Python.

I will spare you all of the details of my subsequent learning-the-hard-way about meddling with Apple system binaries, and also forgo a lengthy digression into how I moved from trashing my MacOS installation to using package managers: first Fink, then MacPorts, and then finally Homebrew – which mercifully was the least insane and most pragmatically malleable of all of these. Between the epoch of MacPorts and the Homebrew era, I actually swore off MacOS package management for a while, and spent a brief lacuna working with the Python.org distribution, the well-known ActiveState releases, my own sub-optimal builds, and many others.

By this time, it was about 2011, and I had learned a few things: how using Homebrew’s Python for development would keep the macOS system Python clean and pristine; how virtualenv and virtualenvwrapper could sequester a project further, making it independent of the Python configuration; how testing and deployment with the Vagrant virtual-machine manager would keep everything in its right place – I have OCD, so these things all appealed to my innate organizational sense as well as my technical inclinations.

And the tools had grown as I had: pip had established itself as the de-facto Python package-management standard. A victor emerged from the packaging wars: setuptools was now being actively developed, instead of bickered over endlessly online. GitHub and git were more and more central every day to my development workflow, as well as that of countless others. Homebrew used git, and while it was written in Ruby, it was being developed by people who appreciated clean and legible code – if one could read the Homebrew source, one could hack on it.

This, then, is where I have come to stand on the various and sundry versions of Python that are available:

  • The macOS system Python – the Python install that ships with macOS

    • version: 2.7.10 (as of Mojave 10.14.5)
    • binary: /usr/bin/python
    • packages: /Library/Python/2.7/site-packages
    • verdict: DO NOT USE. This Python is an old and out-of-date version from the endangered 2.x lineage; even if you were somehow willing to deal with being locked into an old and un-upgraeable programming environment, you’d have to also take care not to disturb this Python installation, such that any macOS updates or hotfixes would be able to count on it working exactly as Apple assumes it would if it were pristine. That means you also can’t upgrade the bundled PyObjC packages, which at the time of writing are an appalingly ancient version, 2.5.1 (by contrast, I have PyObjC version 5.2b1 installed into my Homebrew Python’s package library). In fact, any upgrades, changes, tweaks, modifications, or ∆-minutiae you impose could screw things up later when some lazily-written Apple package script runs on the system Python and acts slightly different than it did in Cupertino. So yeah I repeat: DO NOT USE.
  • The official Python.org Python package distribution

    • version: latest stable, latest development RCs, and archives of prior releases
    • framework: /Library/Frameworks/Python.framework
    • binary: $framework + /Versions/3.7/bin
    • packages: $framework + /Versions/3.7/lib/python3.7/site-packages
    • verdict: Are you, or would you like to be, a core Python developer? Is contributing to the Python language and its central components – the parser, the interpreter, the bytecode generator, the peephole optimizer, the standard library, the “Argument Clinic” preprocessor, the Unicode algorithms, the POSIX wrappers – if you are still reading this and are specifically not bored, this is the Python package for you to use daily. Not to develop all of those non-boring things, no – you’ll of course want to have multiple git repositories of the Python source compiling and running in VMs orchestrated with Docker and whatnot for all that – but you, being a so-called “language lawyer,” will nonetheless want to work with the most official Python you can.
  • The Anaconda Python distribution – from the conda package manager

    • version: 3.7.3 and 2.7.16, respectively, at the time of writing
    • binary: /usr/local/conda/bin/python
    • packages: /usr/local/lib/python3.7/site-packages
    • verdict: Are you a data scientist? Or is that something you say you are when introducing yourself at meetups and conventions? Either way, if you love data and hate all those icky compiler flags and configuration files and settings directory paths and all that other nonsense, you might just be in the market for Anaconda. Anaconda, through its conda package-management CLU, is a whole ecosystem of data-science-y software – all of it easily and painlessly installable with just a few commands (like e.g. conda install numpy, conda install dlib)… Software originating from diverse ecosystems, with complex configurations and lengthy build-times, can all be installed easily in pre-tested, ready-to-run binary form through conda. Anaconda’s Python distribution is one of the fundamental components of this system, and it will always be built with the most reasonable options and delivered within a minimal lag from its official release. You, the data scientist par excellence, would clearly rather spend your time (and your processor’s time) training unsupervised models instead of compiling stuff – and that is why you, like Sir Mixalot, choose Anaconda!
  • The Homebrew Python distribution – from the python and python@2 formulae

    • version: 3.7.3 and 2.7.16, respectively, at the time of writing
    • framework:
      • Python 3: /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework
      • Python 2: /usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework
    • binary:
      • Python 3: $py3framework + /Versions/3.7/bin/python3.7
        • symlinked as /usr/local/bin/python3
      • Python 2: $py2framework + /Versions/2.7/bin/python2.7
        • symlinked as /usr/local/bin/python2
    • packages:
      • Python 3: /usr/local/lib/python3.7/site-packages
      • Python 2: /usr/local/lib/python2.7/site-packages
    • verdict: Use this Python for Python 3, and likely for Python 2. Homebrew has come a long way from its humble beginnings, as an @mxcl side-hustle that compiled everything you asked for and promiscuously linked with anything on your system – from SuperEnv isolation to bottle binary distribution to the stats-driven winnowing of superfluous options, Homebrew is pretty robust and not at all challenging to use. I use the Homebrew Python 3 installation as both my development Python and my workhorse Python – meaning it’s what I use to get miscellaneous tasks done around my system. While some might divide these two roles amongst two Pythons, I find that the one-off tools I write for workhorse tasks can benefit from some of my library development work, and vice versa. Homebrew Python installations have a great feature that installed packages like Python.org Pythons often lack: upgrading the Homebrew Python package will not clobber the associated site-packages directory, keeping your installed packages installed. In addition to Homebrew’s Python 3, I keep its Python 2 formula installed as well, mainly to satisfy the myriad requirements of the formulae that currently demand a Python 2 interpreter. I also occasionally fire up the Python 2 REPL to examine some quirk of the old environment, but I have to do that less and less frequently these days… I am champing at the bit for 2020, when Python 3 will be the Python!
  • The Homebrew PyPy distributions – from the pypy and pypy3 formulae

    • version: 7.1.1 and 7.0.0 – equivalent to 2.7.13 and 3.6.1, respectively
    • binary: /usr/local/bin/pypy and /usr/local/bin/pypy3
    • packages:
      • PyPy2: /usr/local/Cellar/pypy/7.1.1/libexec/site-packages
      • PyPy3: /usr/local/Cellar/pypy3/7.0.0/libexec/site-packages
    • verdict: Have you heard of PyPy? It’s kind of a crazy thing to describe it to those who haven’t: it’s a JIT-optimizing GIL-free reimplementation of the Python interpreter, written in a subset of Python 2 called “RPython”. PyPy is extremely fast, ridiculously internally convoluted, frustratingly counterintuitive to deal with, outrageously slow to compile (or “translate” as it is known in PyPy-land) and highly piquantly interesting. Luckily, you can install the bottled binaries from Homebrew and play around with it without tying up your computer for six or so hours (really). Internally, concepts implemented by PyPy include pickleable stack frames (from Stackless Python), a method JIT and a tracing JIT (from the JVM), a fair generational (non-refcounted) garbage-collection scheme (from everyone who ever tried to patch Python to remove the GIL), coroutines that are not hacked-up generator objects (from Go and Greenlet) and the async stuff from Python 3.4-plus-plus. Did you get all that? No? Don’t bother with PyPy, in that case – it will confuse the living daylights out of you. If you did get all that, and you have a codebase with no C-API extension dependencies, out of which you need to squeeze some performance gains before your next quarterly review so you can afford that time-share in Cleveland – then by all means, go for it, get yourself some PyPy and see what good can come of it. Otherwise DO NOT BOTHER unless you need to dispatch some serious free time and you don’t mind being flummoxed by a thing that deceptively looks like another thing with which you are familiar, but then isn’t. Yep.

Does that answer your question, @Kusine ? I do hope so, indeed!

3 Likes

Thanks @SMillerDev and another big wow @fish2000 ! :grinning: Thank you for a well written and an incredibly thorough, interesting and helpful read. I am going to copy and paste (if that’s okay) what you wrote into my notes for quick referral in the future – what you wrote should be archived/pinned/starred somewhere! :pray:

So in summary:
DO NOT USE unless you are looking for death or pain:

  • The macOS system Python
    thankfully I have never touched this.
  • PyPy distributions
    have no intention of ever using this after reading your verdict.

NEUTRAL:

  • The official python.org package distribution
    what I’ve been using so far. Note: I am very new to Python and just started learning 3 months ago.

RECOMMENDED depending on use:

  • The Anaconda Python distribution
    I am considering using this distribution one day once I am more proficient with Python and am able to hopefully achieve my long term hopes. I think it’s too early for me to install this right now, please correct me if I am wrong.
  • The Homebrew Python distribution
    considering changing to this over the official Python distribution package after reading your explanations and verdicts on the various options.

All in all you have answered my question and a big thank you to you @fish2000!

2 Likes

Of course there’s a relevant XKCD:

and less relevant: https://www.xkcd.com/353/

2 Likes

Hehe :smile: thanks @SMillerDev :+1:

1 Like

@Kusine you are welcome, I’m glad my circumlocutions on the history of my time with Python could at all be useful. Yes!

1 Like

Both a nice read and useful @fish2000, many thanks for your guidance! :+1:

Hi @fish2000, so I have done steps 1-5 as you outlined previously.

However when I do which python to confirm whether the Homebrew Python binary is on my path, I get the following: /usr/bin/python. This points me to, if I am not mistaken, the MacOS system Python version 2.7.10 (python --version).

If I do which python3, I get:/usr/local/bin/python3.

I have made sure that in my .bash_profile I have:

  1. Removed /Library/Frameworks/Python.framework/Versions/3.7/bin from the PATH again.
  2. Added export PATH="/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/bin/python3.7:$PATH";
    I also have export PATH="/usr/local/bin:$PATH"; and export PATH="/usr/local/sbin:$PATH"; – is this correct?

Should which python have pointed to the Homebrew Python binary as you mentioned in Step 5?

Finally, in preparation for Step 7, can you please provide me with help in properly uninstalling the python.org Python package? I have checked on python.org and found a little section about this but I do not wish to make any mistakes and inadvertently erase/tamper with the MacOS system Python during the uninstallation process. I have taken your warning seriously:

Whenever you might have a moment, I would therefore greatly appreciate your guidance with the correct and appropriate steps to uninstalling.

Thank you @fish2000 for your help. :pray::slightly_smiling_face: