Python 2 EOL 2020

(Michka Popoff) #1

Hi

I would like to gather some opinions for the Python 2 EOL in January 2020. We maybe need to prepare a little bit, so that we can: announce our decision upfront to our users, fix what needs to be fixed, and have a smooth Christmas 2019 period.

I want this discussion to be public, so that everybody can chime in. I’ll try to synthesise the different opinions when we have enough feedback.

Some documentation:

This discussion is triggered as upstream still seems to be unsure about what to recommend after 2020:
https://lwn.net/ml/python-dev/9e69c6dc-07cd-2265-b4b8-b9f7e9f81b00@gmail.com/

This discussion is about the choices that need to be made for mac.
I’ll have a word on the Linux situation at the end of this post.

In the hypothesis we delete the python@2 formula in january 2020, we need to check what happens with the formulae that needed it. There is also an option to keep python@2 forever, but in general we remove deprecated packages (we can have a grace period though).

We have 108 formula that explicitely depend on python@2. Around 5 of them also depend on python 3, so dropping python@2 for these 5 is probably not an issue (for example numpy, opencv, scipy…). I made a list below, with the state of the repo on 19/04/2019.

We also need to think about the reverse dependencies. The list below is just a part of the picture: some other packages may depend on the list below.

The following point need to be discussed:

  • What happens with the formulae that depend explicitely on python@2 after the removal? It seems to me that if we explicitely used python@2 for them, we do not want to use system python 2 (for whatever reasons). Should we remove them?
  • What happens with formulae that implicitely depend on system python 2? Should we disallow formulae to use this deprecated library, and depend on our Python 3 library everywhere?

Please note that the python formula will continue to install a “python3” binary, and there is no plan to install a “python” alias. “python” will continue to point to Python 2 on mac until upstream sorts out what they want to do with PEP394. We can open a separate discussion for this though if needed. This topic is not in the scope of this discussion right now.

Note for Linux: we do not allow to use any system python (2 or 3). We are explicitely declaring the dependency where needed. Right now we have around 40 formulae that need to taken care of. This is a separate thing I’ll handle in Linuxbrew. There is a risk that some packages may not build anymore.

Here is the list of packages that may be at risk for 2020 on mac:

aap.rb depends_on “python@2” # does not support Python 3
afflib.rb depends_on “python@2” # does not support Python 3
alot.rb depends_on “python@2”
ansible@1.9.rb depends_on “python@2” # does not support Python 3
ansible@2.0.rb depends_on “python@2” # does not support Python 3
apm-server.rb depends_on “python@2” => :build
appscale-tools.rb depends_on “python@2”
archivemail.rb depends_on “python@2” # does not support Python 3
audacious.rb depends_on “python@2”
auditbeat.rb depends_on “python@2” => :build
blastem.rb depends_on “python@2”
boost-python@1.59.rb depends_on “python@2”
bubbros.rb depends_on “python@2” # does not support Python 3
bup.rb depends_on “python@2” # does not support Python 3
bzr-fastimport.rb depends_on “python@2”
cassandra@2.1.rb depends_on “python@2” # does not support Python 3
cassandra@2.2.rb depends_on “python@2” # does not support Python 3.7
charm-tools.rb depends_on “python@2”
closure-linter.rb depends_on “python@2” # does not support Python 3
curlish.rb depends_on “python@2” # does not support Python 3
cvs2svn.rb depends_on “python@2” # does not support Python 3
davix.rb depends_on “python@2” => :build
ddar.rb depends_on “python@2” # does not support Python 3
dnsviz.rb depends_on “python@2”
docker-cloud.rb depends_on “python@2” # does not support Python 3
duplicity.rb depends_on “python@2” # does not support Python 3
dxpy.rb depends_on “python@2” # does not support Python 3
emscripten.rb depends_on “python@2”
euca2ools.rb depends_on “python@2” # does not support Python 3
filebeat.rb depends_on “python@2” => :build
fontforge.rb depends_on “python@2”
fpp.rb depends_on “python@2”
gdal.rb depends_on “python@2”
git-plus.rb depends_on “python@2” # does not support Python 3
git-remote-hg.rb depends_on “python@2” # does not support Python 3
gnuradio.rb depends_on “python@2”
gwyddion.rb depends_on “python@2”
handbrake.rb depends_on “python@2” => :build
headphones.rb depends_on “python@2” # does not support Python 3
heartbeat.rb depends_on “python@2” => :build
hg-flow.rb depends_on “python@2”
ino.rb depends_on “python@2” # does not support Python 3
ipython@5 depends_on “python@2”
joplin.rb depends_on “python@2” => :build
joshua.rb depends_on “python@2” => :build
ledger.rb depends_on “python@2”
libdnet.rb depends_on “python@2” # does not support Python 3
llnode.rb depends_on “python@2” => :build
logentries.rb depends_on “python@2” # does not support Python 3
mercurial.rb depends_on “python@2” # does not support Python 3
mesa.rb depends_on “python@2” => :build
mesos.rb depends_on “python@2”
metricbeat.rb depends_on “python@2” => :build
mkvtomp4.rb depends_on “python@2” # does not support Python 3
molecule.rb depends_on “python@2”
mongodb.rb depends_on “python@2”
mongodb@3.6.rb depends_on “python@2”
muttils.rb depends_on “python@2” # does not support Python 3
mysql-utilities.rb depends_on “python@2” # does not support Python 3
nanopb-generator.rb depends_on “python@2”
nicotine-plus.rb depends_on “python@2” # does not support Python 3
node.rbdepends_on “python@2” => :build
node@10.rb depends_on “python@2” => :build
node@6.rb depends_on “python@2” => :build
node@8.rb depends_on “python@2” => :build
notmuch.rb depends_on “python@2”
numpy.rb depends_on “python@2”
offlineimap.rb depends_on “python@2” # does not support Python 3
ola.rb depends_on “python@2” # protobuf@3.1 does not support Python 3
ooniprobe.rb depends_on “python@2”
opencolorio.rb depends_on “python@2”
opencv.rb depends_on “python@2”
opencv@2.rb depends_on “python@2” # does not support Python 3
opencv@3.rb depends_on “python@2”
osc.rb depends_on “python@2”
osquery.rb depends_on “python@2” => :build
packetbeat.rb depends_on “python@2” => :build
pdf-redact-tools.rb depends_on “python@2” # does not support Python 3
platformio.rb depends_on “python@2” # does not support Python 3
protobuf.rb depends_on “python@2” => [:build, :test]
protobuf@3.1.rb depends_on “python@2” # does not support Python 3
protobuf@3.6.rb depends_on “python@2”
pssh.rb depends_on “python@2” # does not support Python 3
pwntools.rb depends_on “python@2” # does not support Python 3
py2cairo.rb depends_on “python@2”
pygobject.rb depends_on “python@2”
pygobject3.rb depends_on “python@2”
pyqt.rb depends_on “python@2”
pyside.rb depends_on “python@2”
pyvim.rb depends_on “python@2”
qscintilla2.rb depends_on “python@2”
recon-ng.rb depends_on “python@2” # does not support Python 3
scipy.rb depends_on “python@2”
shogun.rb depends_on “python@2”
sip.rb depends_on “python@2”
ssh-audit.rb depends_on “python@2”
sslyze.rb depends_on “python@2”
subliminal.rb depends_on “python@2” # does not support Python 3.7
supervisor.rb depends_on “python@2” # does not support Python 3
terminator.rb depends_on “python@2”
tnote.rb depends_on “python@2” # does not support Python 3
uwsgi.rb depends_on “python@2”
v8.rb depends_on “python@2” => :build # depot_tools/GN require Python 2.7+
volatility.rb depends_on “python@2”
vte.rb depends_on “python@2”
weechat.rb depends_on “python@2”
wxpython.rb depends_on “python@2”
zurl.rb depends_on “python@2” => :test

3 Likes
(Sean Molenaar) #2

Personally I’m in favor of removing anything that doesn’t build with python3 and/or system python by the due date. There’s not really a good reason to keep around unsuported software, we can easily readd things.

3 Likes
(FX) #3

I’ve been the one adding does not support Python 3 comments on many formulas, when it’s explicitly declared that they don’t support Python 3 (or when I found out after investigating the code that it was not supported). But I didn’t finish, because… life.

It sounds to me the first step is triaging: investigate the remaining formulas that depend on Python 2 but don’t have this comment. Either they can be migrated to Python 3 (taking care of dependencies and reverse dependencies), and this needs to be done. Or the comment can be added to them.

Then, when EOL hits, it is going to be hard to justify removing Python 2. I’d love to, but there are some widely-used formulas that depend on it, including node.

(Jonathan Chang) #4

I was curious what the Linux distributions were planning and found this LWN article:

I think it would be best if we mirrored Ubuntu on this: Python 2 support gets dropped in April 2020. If macOS is still shipping Python 2, then formula needing Python 2 must use system Python or be removed from Homebrew/core.

3 Likes
(Mike McQuaid) #5

I think we should drop Python 2 whenever it is EOL (which sounds like January). Ubuntu have the desire/resources to e.g. backport security patches after that date which we do not.

1 Like
(Michka Popoff) #6

I am glad you did that, it helps indirectly for Linuxbrew. And it allowed to create this first list.

One question: how do you know that a specific formula depends on Python 2. If it depends on system Python 2, you will not notice.

And regarding node (and others): the process will probably be painful.

(Paul Hoffman) #7

A “mee too” for drop support Brew Python2, and allow packages to risk it with Mac Python2 if they need it.

1 Like
(Mirko Friedenhagen) #8

I would go for dropping python2 support as well and removing dependencing formulas. Probably homebrew needed more build resources to test this thoroughly though. Having a kind of Darwin container/jail system would help as well, just remove both homebrew and system python2 inside the jail and build along.

(ccc) #9

It will be funny to see how the Node.js community will react when brew install node fails because even the soon-to-be-released Node 12 requires legacy Python.

1 Like
(Michka Popoff) #10

This is an issue for the node community, but I think we should try to take this into account. We are in April now. We could write on their issue tracker during the summer, telling them that they have 6 months to fix it.

Right now nothing is decided, I want to gather more opinions. The only thing I see here is that most of the people would like to get rid of Python 2 support. Not sure we should discriminate between Brewed Python 2 support and Apple Python 2 support.

(Ben Lackey) #11

I think dropping 2 is going to cause a word of pain. A well behaved shell today has python resolving to 2 and python3 resolving to 3. So many systems depend on that and that’s at the user/cli level. Digging a little deeper there are numerous dependencies sprinkled everywhere. I’m having flashbacks to the attempt to kill perl 4 as 5 became a thing. It never really transitioned, instead the language fragmented. This is, to be fair, a python issue, not a brew issue. I just suspect brew is going to be a lot more usable with ongoing 2 support even as the community attempts (and I predict fails) to EOL it. All this is true for Linux but even more so for Mac OS given the less coupled nature to various open source projects.

(Izaak Beekman) #12

I think we should take a cue from Apple when the next macOS beta comes out. If they are still shipping python2 it would be nice to allow formulae to use system python2.

One big question I have (mostly due to my incomplete understanding of brewed python packages and the internals of homebrew/brew) is:

Will vendored modules/packages still work with system python2? Will we need to migrate/revision bump formulae with vendored python2 dependencies to reinstall them with system python?

I have a distant memory of the painful situation of figuring out where system python pip installs stuff, and dealing with bad permissions… I’ve been using brewed python for so long that I have no idea whether any of this is still applicable.

1 Like
(Michka Popoff) #13

Yes. It will be painful. But it should not be painful for the brew maintainers. This is why we are having this discussion right now. It is a Python issue, and how the thing was handled by the Python maintainers/community.

Removing Python 2 from brew will probably break some people’s workflows. But it will not break the underlying OS. At least we are on the safe side here.

We can not take the responsibility to ship a know unmaintained and deprecated library. Especially one as critical as Python, which may live on with unmatched security issues. This is why we need to drop it.

Even if they release the next mac OS with Python 3 this year, we still support the two previous mac OS versions. Apple is 3 years too late for us. And we need to find a solution that fits well for the 3 last versions of mac OS. Not 1 solution for the latest mac OS, and another solution for the 2 previous ones.

It depends on what we choose here. If we say that Apple’s system Python 2 is fine, why not.

(Michka Popoff) #14

I posted a comment on the node issue tracker: https://github.com/nodejs/node-gyp/issues/1337#issuecomment-486087002

Just to let them know of the situation. Because this is one of the biggest packages in brew.

We should take their situation into account, but not block the whole deprecation process only because node is not ready.

I still want to hear some more ideas/opinions before summarising the options we have here.

1 Like
(Izaak Beekman) #15

Yes I agree we need one solution. If the next release of macOS still ships with Python2, however, then that is an indication of some level of continued support from apple. So in that case I’m more inclined to allow formulae to have dependencies on system python2.

If, however, they ship with Python3, then I would be inclined to drop all Python2 support at Python2 EOL.

(Mike McQuaid) #16

From following macOS releases for 10 years my best guesses would be (in descending probability of happening):

  1. macOS 10.15 ships just Python 2
  2. macOS 10.15 ships both Python 2 and Python 3
  3. macOS 10.15 ships Python 3
(Michka Popoff) #17

Looks like the discussion has started at openSuse too: https://lists.opensuse.org/opensuse-factory/2019-04/msg00229.html

(Steve Peters) #18

I just noticed that mercurial 5 has been released with beta support for python3. I’ve made a mercurial-python3 formula in a tap for developers who want to test it (they say 1% of tests fail, so they aren’t ready to distribute it with python3 support):

1 Like