Python 3.8 and Large Python (or other large groups of dependencies)

This is not meant as a “why has it not been updated yet” thread! There is already one of those here:

The PR for update is starting to have a discussion that belongs here not on Github. Posts to the PR asking why it hasn’t been updated and making suggestions for how to update it get sent to everyone actually working on the update. Rather than bombard them with more traffic in the PR. Let’s discuss it all over here.

One of the observations I have is that the update is being held up until all (maybe most) of the dependent packages build properly. I think perhaps a different approach is needed which follows the example of how this same problem is solved in FreeBSD’s ports tree.

Python 3.8 does not contain many dependencies but many other formulas are dépendant on Python 3. It would be helpful if the Python 3.x formulae could be updated rapidly after release without breaking all the formulas are dépendant on Python 3.

Would it be possible to add an exception list to the Python 3.8 formula which lists those other formulas that are not yet compatible with 3.8 and still need 3.7 instead? That way all the action items live in a single file and all contributors can see at-a-glance what todos remain. If none of my project’s dependencies are listed in that formula, then I have a green light to upgrade to 3.8.

And I thought everyone here developed with a py virtual environment. I guess I was wrong

Can someone explain why it looks like just 1 person is working on the migration of all the tightly coupled python formulas? Is there no way to distribute the work load so we can get python 3.8 shipped?

Because you don’t necessarily see communication in the private homebrew slack and also because it’s not the people that are bottlenecking this. It’s CI time and resources. Every build takes the better part of a day and Jenkins is very error prone. There has been work to move CI to github actions/Azure to make it better but that simply isn’t done yet (because homebrew has rather unique CI problems that need to be supported first).

Thank you for the response and clarity…

As I have stated in a couple PR’s, I am here to help. Just need to be told how I can do that without clogging up CI anymore then it already is.

Is there a document on how to upgrade?

  • I have the python formula installed already (3.7.5) and brew update; brew upgrade does not upgrade python 3.7.5
  • Better to brew uninstall python before brew install 'python@3.8'
  • Should I be posting these questions some where else?

Thanks.

Spurious files in python@3.8?

brew install 'python@3.8'
==> Downloading https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz
<snip>
==> Summary
🍺  /usr/local/Cellar/python@3.8/3.8.0: 8,788 files, 124.1MB, built in 3 minutes 21 seconds

What are the files in the ‘xxxx’ for?

$ find /usr/local/ -name "python*" | grep -i bin
/usr/local//bin/python-build
/usr/local//bin/xxxx/bin/python
/usr/local//bin/xxxx/bin/python-config
/usr/local//bin/xxxx/bin/python2
/usr/local//bin/xxxx/bin/python2.7
/usr/local//bin/xxxx/include/python2.7
/usr/local//bin/xxxx/lib/python2.7

I don’t know what the files in the xxxx are for but I think they’re installed by you since I don’t have them.
For now python@3.8 is a separate formula but eventually it’ll automatically be upgraded with brew upgrade.

Just wanted to circle back to this thread.

I may not have the process described exactly, so please forgive me. In FreeBSD if a particular port that depends on another port does not build, the release of the parent is not held up. Packages that don’t build for one reason or another are the responsibility of the maintainer of that package. If it doesn’t build, and nobody fixes it, that package is just marked as such and eventually removed.

Perhaps something like this system could be adopted so that packages that depend on Python don’t hold up Python itself being released.

I may not have the process described exactly, so please forgive me. In FreeBSD if a particular port that depends on another port does not build, the release of the parent is not held up. Packages that don’t build for one reason or another are the responsibility of the maintainer of that package. If it doesn’t build, and nobody fixes it, that package is just marked as such and eventually removed.

Perhaps something like this system could be adopted so that packages that depend on Python don’t hold up Python itself being released.

We discussed this at FOSDEM and this is what we would like to introduce. We would like to mark packages as “broken/deprecated”, and disable CI for them.

We need to implement this, but we should be able to have something equivalent soon.