Python Formula - Using Wheels


(Micah Culpepper) #1

Hello Homebrew Community!

I’m writing my first brew formula, for an internal tool at my workplace that’s written in Python. Following the advice in https://github.com/Homebrew/brew/blob/master/docs/Formula-Cookbook.md and https://github.com/Homebrew/brew/blob/master/docs/Python-for-Formula-Authors.md, my formula defines resource stanzas for all of the Python packages that we depend on. My question is about the types of Python distributions that may be used.

Python packages can be distributed as source distributions in .tar.gz format, or built/wheel distributions in .whl (basically .zip) format. The docs for Python formula authors seem to indicate that source distributions are preferable, and indeed, the venv.pip_install command will by default refuse to install .whl distributions.

I have noticed that when I use 100% source distributions, my formula takes about 10 minutes to install, because it has to compile a bunch of C. However, if I skip the formula and use pip in a clean virtual environment to install my package from requirement specifiers, allowing it to select .whl distributions when available, the install process takes about 2 minutes. This is a huge performance difference, so I’m sure my users would prefer the formula to use .whl distributions when available.

Also, the docs are somewhat vague on this requirement; while it is clear that the url in a formula should download a source distribution, should all of the resources in a formula refer to source distributions? Why is venv.pip_install programmed to refuse to install .whls? Will puppies and kittens really die if I have binary dependencies?

I’d appreciate any advice or insight.


(Sean Molenaar) #2

For an internal tool this might be less of an issue, but in general homebrew only accepts formula that compile from source (see Acceptable Formulae).
Homebrew core solves this issue by providing precompiled binaries from the source in the formula (bottles). If you have your formula hooked up to a CI system you could maybe do the same for extra speed gain?


(Micah Culpepper) #3

Looks like bottling is exactly what I need. Thanks @SMillerDev!


(Paweł Adamczak) #4

Sorry to dig up an old thread, but did you ever figure out how to use wheels inside the Formula? Installing my package takes 3 minutes 11 seconds (~30 seconds via pip, with wheels where available) because of what you mentioned originally, which makes it very painful to sell to my colleagues.


(Sean Molenaar) #5

I think the easiest option here is to make a bottle or a cask.


(Paweł Adamczak) #6

Thanks for the quick reply @SMillerDev, I appreciate it a lot.

I tried looking into bottles a bit but couldn’t find any examples that I could replicate. And correct me if I’m wrong, but it requires a separate CD server to build them, right?

I haven’t thought of Cask because I associate it with GUI apps, but will look into it, thanks for the tip. I’m guessing it has worse integration with Python / venv, but we’ll see.

The funny part - all I need Homebrew to do is run pip install . for my package inside a virtualenv and link the bin entry points, that’s it : )