Maintaining a tap to provide options for a formula

Anything coming from the homebrew organization on Github would still suggest support. Thats a large part of the issue and also the reason why only usage is being removed, not the feature in general. Homebrew is not a company but is volunteer run and just the effort that goes into explaining people why we don’t want to support their options tells me this was a good call to reduce the support burden.

I’m glad the feature is sticking around. I was under the misapprehension that it was going away entirely. That makes more sense. Probably don’t even need code for “child formulae” a custom tap and some sort of lambda that reacts to the main repo changing and applies patches would do the trick.

@saagarjha are you up for it? I think the problem still becomes knowing how to adequately and securely manage those patches.

To put it bluntly, and with all due respect and acknowledgment that you’re doing a great job, but that’s a BS perspective.

In my world, telling a maintainer that their perspective is bullshit is the exact opposite of respectful behaviour.

I’m thankful for all the work the core devs put in here

Thank you. I appreciate user feedback, be it positive or negative, even though we may disagree on many points.

communicating this out to users should have been priority number 1.

While you have a very good point in that we could do a better job in communication, I still feel we have done our best in prioritizing. Sometimes, an org needs to prioritize staying functional and avoiding burnout over anything else.

1 Like

“a BS perspective” is not respectful. Please reread our Code of Conduct adjust your future communication accordingly.

Communication, like many part of Homebrew, could be improved. Who is going to do that, though? What you (and others) seem to ignore about requests for additional communication is that you’re actually asking that more work be done by the individuals who are already overstretched (i.e. me) about a change that you fundamentally reject anyway.

We’re not willing to ship things we don’t support, sorry. I do not consider it acceptable to ship software that we are unable to test and we know will break. Even if I did consider that acceptable I’d be unwilling to accept the additional support burden that the maintainers have to take on from people submitting issues when they are clearly told not to (something that still happens often).

In short, read through https://github.com/Homebrew/homebrew-core/issues/31510 and if you have suggested to solve the problems that are mentioned there (e.g. maintainer and CI resources, inability to bottle options, inability to test options) then I’m all ears. With all due respect, though, you’ve contributed some pull requests (which I genuinely appreciate) and I’ve been maintaining this package manager for 10 years. I don’t make up problems and in the last year there’s been several occasions where this project has run a real chance of dying completely; changes like this are how we are going to be able to keep it alive while being run in the spare time of volunteers.

1 Like

Please consider reading the issues linked from the release notes if you are going to complain about them which would have explained this. Somewhat relatedly, it’s a little ironic to complain about poor communication on our part and not read the communication we have made…

To be clear: like brew extract we want to make it as easy as possible to maintain these formulae in taps. We are open to Homebrew/brew changes in order to make that process easier.

(I apologize in advance if my post seems a bit disjoint; I had started writing this when @SMillerDev replied to my comment and then I stepped away, so it’s kind of in a weird state where I tried to take my original comments and merge in the new developments.)

I don’t agree with the way @joshka said it, but I do agree with the point he brought up: the communication around this has quite lacking. When contributors to Homebrew (a group which I am part of, if only barely) know nothing about this change until it’s already being implemented, I think there’s at least some issue here. I’m reminded of this exchange from The Hitchhikers Guide to the Galaxy:

Mr Prosser: But, Mr Dent, the plans have been available in the local planning office for the last nine months.
Arthur: Oh yes, well as soon as I heard I went straight round to see them, yesterday afternoon. You hadn’t exactly gone out of your way to call attention to them had you? I mean like actually telling anybody or anything.
Mr Prosser: But the plans were on display…
Arthur: On display? I eventually had to go down to the cellar to find them.
Mr Prosser: That’s the display department.
Arthur: With a torch.
Mr Prosser: The lights had probably gone out.
Arthur: So had the stairs.
Mr Prosser: But look, you found the notice, didn’t you?
Arthur: Yes yes I did. It was on display at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying beware of the leopard.

Sure, I technically had a chance to comment, but it was in a place that I had no reason to know the location of, and even now had to dig through GitHub to find. Putting the discussion in a public place (actually, that’s not quite true: apparently some of this happened in Slack?) and expecting to get feedback doesn’t seem like it would really work. I’m actually surprised you got feedback at all, actually, but then again most of the conversation came from the maintainers and a couple of contributors…

I don’t see where you getting that number. Unless I am not understanding the install analytics correctly, for the examples I posted, this ratio is over 10%, going up to 70% for ed --with-default-names. I am still not seeing the case for these use of options being uncommon; it seems like any way I look at it the ratio is signficantly more than 1%.

There is a huge difference between a popular program already in homebrew/core like moreutils, but installed in way so that it doesn’t conflict with another popular package, and a toy shell script that I want to share with my friends who use Homebrew or really any other package not found in homebrew/core. There are difference in number of users is multiple orders of magnitude, for one, and it’s confusing to find the same package in the official tap but also a popular “fork” in another random place. I really do think that these two should be treated differently; not as a “special case”, per se, but still be recognized as something that people use.

I think this is lacking the context in which I wrote it. The person doing the work here is not me or the Homebrew maintainers, but the author/interested user of the project the formula installs. Previously, they would have a centralized place where they could make sure their project was available on macOS, but now there is no reason they would go to saagarjha/tap-with-options in addition to homebrew/core. The work that was being done before in a central place no longer gets applied everywhere else.

Again, I’d like to reiterate that I appreciate the work that the Homebrew maintainers put into the project (having a PR already open to fix a formula so I don’t need to look at it, which has happened to me multiple times, is really great), and, considering that I use Homebrew, I have a vested interest in keeping it alive and working as well as possible. While I was not entirely aware of the recent issues Homebrew has run into, I can empathize with your viewpoints, having maintained or participated significantly in the development of other popular open-source projects I am keenly aware of the work that maintainers must do, and having to tell people that we don’t have the time or energy to implement something that they wanted. That being said…

…this makes me a bit sad. I am not trying to exasperate you with calls of “bring back the options”, and I hope I have not come off that way with my responses here. My intent was to bring light on points that I felt were not adequately touched upon during the discussion of the issue, try to get a better understanding of the reasons why it was implemented, and lament the fact that it was quite difficult for me to provide input in the hopes that more avenues for communication will open up in the future; namely, notifying affected users to comment on the discussion and have a longer “deprecation” period, and keeping sources close to the conversation open and being clearer in indicating a place to provide feedback–when I tried to extend discussion on GitHub, I was prevented from doing so on the original thread, nor was I comfortable opening a new issue and link back to the old one due to the frankly scary issue template.

That being said, I still think that the current solution is going too far in the other way and ends up having more of a negative effect than it should, and I think that there is a compromise solution somewhere that we just haven’t gotten to yet that would end up being much nicer for everyone. Would you mind qualifying your issues a bit more, specifically with regards to the maintenance burden of supporting options, as well as what you consider to be “acceptable” for Homebrew and minimizes support requirements? As I see it, the benefits we are losing by going this route are mainly focused around convenience and customizability for users, at the cost of a higher maintenance burden. I have come up with a couple ideas in this thread, and was wondering how feasible they would be considering the constraints you have; it seems like any taps under homebrew/ is off limits because it would cause people to open issues, and Homebrew does not have a good way of preventing people from filing issues on things marked as unsupported? How much CI was this using up, and can Homebrew spare any at all? AFAIK Homebrew tests every binary bottle, to what extent does this coverage go, and are builds from source also included? How many combinations are considered reasonable to test, and will anything outside of this be removed at some point? How much complexity was being added by this change (I’m not great with Ruby and as such haven’t touched the core of Homebrew, so I’m not unfortunately not very aware of this)? What was the support burden?

Would you accept functionality for Homebrew to do anything like distribute the source code of the formula (which it is doing anyways, since --HEAD seems to be a thing that will continue to exist), and allow me to compile my own package with relevant options? The benefits I get is that I the basic auto-updating portion is done by Homebrew itself, but whenever I build I run the “patch” (which I am imagining is something like me selectively copying the options I care about from an old commit to homebrew/core and telling Homebrew “here, apply this”) to customize it for my environment (transparently; the “patch” is applied each time a new update comes and brew update is run). Is it reasonable to expect that people working at this level of Homebrew will know enough to not file issues for the issues they encounter?

Again, I appreciate the feedback that you have given me, and I hope that it’s possible to improve the situation we have currently to benefit everyone. Finally,

If we end up with something here that I could reasonably implement, I’d be happy to contribute, as long as I can understand the Ruby :wink:

3 Likes

When we remove options we adjust the defaults to best fit how people are using the application. In many cases this results in the defaults being changed to include things that were options.

Yes. This is because the people who do the bulk of the work to run the central place are not willing to ship options any more.

You’ve been able to provide input here. When it comes to “better understanding the reasons” unfortunately that’s another request for time from already busy maintainers.

There is not. I’ve searched long and hard for one and solicited many suggestions and found nothing.

We do not support people who wish to build from source, no.

In terms of the other numbers/details requested: you can browse around but I’m afraid I/we can’t justify the resources to gather that information. It’s a non-trivial endeavour.

Unfortunately it is not reasonable. People have been warned in the past and continue to do so.

Something like this seems like the best fit. Right now I think you might be able to subclass an existing formula and override the install method to handle any options you use.

1 Like

I’m assuming that the general guideline for this would be to pick the option that the majority of users use, but this still leaves out a significant number of users if the “option/no option” ratio is near 1:1, as is the case with many packages. What is the plan for formulae like these? Will everything with >1% usage end up being split into two (or more, in particularly divided cases): formula and formula-with-options-applied?

This response is starting to frustrate me, since every time I bring up “centralized work” the response seems to always be “we are doing this work for you and therefore don’t have to listen to you”, which is a true statement but is also a misunderstanding of what I am trying to say. My point is that the Homebrew maintainers and collaborators are already doing the work to keep packages up to date, and they are doing so in a way that is central: homebrew/core. I do not see it controversial that they will continue to do this job, since without it Homebrew would not function at all.

Your argument seems to be that options were causing an unduly burden on maintainers which they did not want to deal with, and I am willing to accept that this as true. However, I do not see why you not wanting to have to deal with options means that I, as a downstream user of your software, can’t use the work already being done to keep formulae up-to-date and extend it locally (specifically: in a way that should not affect homebrew/core’s day-to-day development) to simulate options. The reasons why I’m forced to bother you about it here is that it doesn’t look like Homebrew currently supports the “hooks” into the build process that I would need to simulate this feature locally, so I am trying to argue that these should be added so I could go and write my own “patches” without having to get anyone from homebrew/core to continue to support it.

I only asked for this information because I assumed that you must have gathered it already to measure the benefits of removing this feature, because as far as I can tell one of the arguments to remove them was because they kept breaking due to a lack of CI resources to test options.

With all due respect, I am finding it difficult to believe that you have not been able to find any solutions for this. You are saying that it is fundamentally impossible to make something that is advanced enough that it will prevent people who are willing to waste your time repeatedly filing issues about formulae breaking in unsupported configurations when told not to, which to me seems quite far-fetched: are you anticipating to be overwhelmed with users that say “I tried brew install formula on my Mac OS X 10.3 system but it didn’t work so I used Homebrew to get the source and and dug into Homebrew’s obscure compilation modification feature and used it to coerce the linker into producing executables for PowerPC, but I messed it up and now I’m going to repeatedly forget how to read and ignore half a dozen warnings to not file an issue for this and waste your time”? The people using this feature are going to be somewhat advanced users, even more so the more we push this on users to implement themselves. They are competent enough to know that they shouldn’t be filing issues for problems that are clearly “on their own” to fix.

Again, does this allow me to only have to deal with writing options, and not have to handle updates? Because I fundamentally cannot keep track of the release schedule of every package I use; the main reason I use a package manager is so that I can reuse the work of the dozens of people who contribute to the main repository. I am totally OK with writing a custom install step, options, whatever, as long as autoupdating is not something I need to reimplement.

2 Likes

With all due respect, I am finding it difficult to believe that you have not been able to find any solutions for this. You are saying that it is fundamentally impossible to make something that is advanced enough that it will prevent people who are willing to waste your time repeatedly filing issues about formulae breaking in unsupported configurations when told not to, which to me seems quite far-fetched: are you anticipating to be overwhelmed with users that say “I tried brew install formula on my Mac OS X 10.3 system but it didn’t work so I used Homebrew to get the source and and dug into Homebrew’s obscure compilation modification feature and used it to coerce the linker into producing executables for PowerPC, but I messed it up and now I’m going to repeatedly forget how to read and ignore half a dozen warnings to not file an issue for this and waste your time”? The people using this feature are going to be somewhat advanced users, even more so the more we push this on users to implement themselves. They are competent enough to know that they shouldn’t be filing issues for problems that are clearly “on their own” to fix.

You’d be surprised how often this would happen. The first people might be advanced users but then someone posts it on stackoverflow and suddenly everyone is an expert and files issues for it.

In the end you just have 2 options, deal with the lack of options and have support from homebrew-core or make your own version and don’t have the support. Subclassing something in the ruby files might work with keeping the updates but in the end, none of the maintainers would provide active support for such a hackaround.

2 Likes

We will not be creating duplicate formulae unless we’ve needed them with contradictory options (see curl-openssl). People will need to maintain less widely used options in taps.

Because if we continue to keep the options in formulae we need to continue to deal with the support issues that result (and we’re not willing to do that).

I am open to adding hooks to make it easier to maintain formulae with options in taps.

It’s not respectful to (indirectly) accuse of me of having found solutions but lying about having done so, sorry. Regardless, you don’t need to believe me because it doesn’t really matter either way and the end result will not change.

They should be but they aren’t so: here we are

I have no idea, I haven’t tried it.

If you wish to reuse their work you will have to accept (such as in this case) that their decisions on how the package should be built will differ from yours and you then have a choice of doing the work yourself or accepting their versions. Note this is how pretty much every other package manager works.


So where we’re at: we’re removing all options for formulae in Homebrew/homebrew-core in the near future. Options will need to be maintained in taps. I will review and help with (to an extent) PRs that make that process easier for you and for others to maintain those taps but I will not be doing most of that work by myself.

1 Like

Likewise, I don’t think it’s particularly nice to misconstrue what I wrote into something where I’m calling you a liar; trying to hunt for oblique rudeness does nothing other than lower the overall quality of conversation and drag it into personal attacks, which is a place that I would very much rather not go. I’d appreciate it if you took the time to consider my comments in good faith, and you’ll find that in this case the quote you pulled, out-of-context, makes a lot more sense when you consider the circumstances: you have laid out certain points that you claim to refuse to compromise over (namely, building from source/allowing customization and preventing people from filing issues), ruling out the suggestions I make (as well as those brought up before), but seemingly accept the current policy which has its own issues in these areas.

OK, so I’m going to have to ask for clarification here: from the responses I’m getting, it seems like homebrew/core doesn’t want to deal with any issues involving unsupported configuations at all. However, this is clearly unfeasible: there is nothing stopping someone from reporting an issue with my tap to homebrew/core, nor is there a lack of people currently posting issues about their formulae not working because of this change (most of which end up getting a response like “wontfix, look at #31510”). Clearly, there is a number of issues that you are OK with handling, and this number is greater than zero. So my question is what this number is, to allow me to come up with a solution that generates fewer issues than this amount. Usually, this manifests itself as having users follow more steps to get to a state where they can mess up; as a core maintainer of a quite popular project I have found that the more advanced features get significantly fewer bug reports filed against them, regardless of the existence of Stack Overflow or any other support forum.

No, this just isn’t true at all; basically every other package manager that I’m aware of differs from Homebrew in this regard. Off the top of my head, MacPorts and APT both

  • provide different versions of packages
  • provide variants for packages
  • allow for compiling packages from source locally
  • allow the user to control how to compile their packages, offering options like passing in custom environment variables

Homebrew has been moving away from offering this functionality (though, it still does allow for building formulae from source, just not in a way that is customizable) and towards being more like a binary distribution service, so it’s basically alone here.

I’m glad to hear it. If we can figure out something that provides measurable improvement in the area of allowing users to customize their formula installs in a way that is acceptable to Homebrew’s maintainers, I could try my hand at implementing this.

3 Likes

Alternatively, I found something you said to be rude and somehow it’s justifiable to blame me for that rather than apologising. Ok then!

Sorry, no. This thread has had way more clarification and explanation than is necessary and, at this point, a good use of time.

It is very far from alone but: yes, we’re a binary distribution now and that’s intentional.

Please do. Until then, I think we’ve exhausted this topic, thanks.

1 Like

Reopening due to rekindled interest in possible PRs as outlined in https://github.com/Homebrew/homebrew-core/issues/31510#issuecomment-457870984.

The communication here from the maintainers has been severely disappointing from the start. The lack of honest representation of the way these changes would effect end users is concerning. I am one of those option users that had to sift through the homebrew/core repository to find issue #31510 as a link to a PR weeks ago that only starting effecting me now that a new version of a tool I use is out.

The change to a formula’s build behavior without a revision bump is an oversight and one I expect has been a cause for why more option users have not run into issues yet. At the very least this drastic breaking change in behavior should have had a disclaimer during update as has been stated previously in this thread.

Taking a survey over my installed tools, only 7 of 160 make use of options. Looking into those I see that they each have had options removed without a revision of the formula. I was able to then reduce those 7 formula down to only 2 with essential options and 1 of those was for a tool I no longer need for the work I do. Of those 2 the option was not a behavior change, but an additional library linkage. Are these class of options candidates for a GitHub issue or PR to enable by default?

I want to thank saagarjha for starting this conversation and trudging through this experience to voice concerns that I share.

4 Likes

Thanks for your feedback. What has been said before still holds. We hear your concerns. I believe we’ve been doing a much better job at communication than we did a decade ago. The fact that we’ve failed this time doesn’t mean we’ve stopped learning altogether.

Getting better won’t happen overnight though. As a Homebrew maintainer, I’m struggling to carve time out of my already busy life. If we want to do a better job at keeping users happy, we’re going to have to take things off our plates. That’s why I believe removing options from homebrew-core was a healthy thing to do. It gives us resources to get better, and have happier users in the long run.

It’s not clear to me how a revision bump would have helped here. A revision bump is meant for fixes that are urgent enough to be rolled out to everyone right now.

At the very least this drastic breaking change in behavior should have had a disclaimer during update as has been stated previously in this thread.

A caveat makes no sense unless it gives the user something actionable at the same time. Sorry but there’s no way we could have found the resources to craft individual per-option caveats, and more resources to deprecate and remove those caveats again later.

Are these class of options candidates for a GitHub issue or PR to enable by default?

Whether such a PR is going to be merged or not depends on many factors.
Why not give it a try? The worst thing that can happen is for your PR to be refused, and for you to go :woman_shrugging: and move the affected formula to your own tap.

Final reminder:

  • This thread is about maintaining custom taps, and providing tap authors with the resources they need.

  • Please do not discuss the removal of options from homebrew-core. That ship has sailed.

  • For other topics, open another thread.

There have been several reminders already. Please consider this a final warning.

1 Like

@claui Thank you for re-opening this discussion. I won’t belabor the points already made. To @MikeMcQuaid 's credit, he had mentioned this day of reckoning was coming over many years although a BIG FLASHING WARNING with a date would have been appreciated :slight_smile: The changes obviously impact a subset of users who care deeply about changing their workflow.

Some projects have sprung up namely to address our concerns. Namely, https://github.com/portage-brew.

I understand the HB maintainers not wanting to point users toward a particular direction as it may be perceived as tacit support. However, we can use some help in collecting different ideas to build something that supports the needs of this user subset.

Ideally we’d leverage the work of homebrew rather than altogether do something separate. Ideally we’d have a centralized tap to lessen fragmentation.

If you care enough to voice you concerns about these changes, I hope you’ll care enough to contribute to a go forward solution… at the project above or another.

3 Likes