Brew test-bot - should I rebuild formula when nothing has changed?

Hi there!

My name is Ladislas, I’m one of the maintainer of the osx-cross/avr tap used to build the AVR toolchain.

I’ve been playing recently with brew test-bot in order to create bottle of the different formulae and save a lot of compilation time for our users.

I’ve managed to make in work locally with brew test-bot, brew test-bot --ci-upload and finally brew pull --bottle and I’m now going to implement that for Azure Pipeline.

The goal is to obviously automate the process of building bottle from PRs but also to have a cron job to test if dependencies have changed and if a bottle should be rebuilt.

My first question is:

How would one test if a rebuild is needed or not?

My second question is:

When running brew test-bot formulae_name twice locally, I end up with two different sha512 – is this expected behavior? Am I doing something wrong?

Thanks a lot for your kind help!

When I have everything figured out, I’ll write a tutorial for all of this! :slight_smile:

Ladislas

How would one test if a rebuild is needed or not?

In core we run the test of everything that uses the changed formula. This will indicate what still works with the changes and everything that doesn’t needs a manual revision bump.

When running brew test-bot formulae_name twice locally, I end up with two different sha512 – is this expected behavior? Am I doing something wrong?

Homebrew doesn’t have reproducible builds so yeah, that happens.

brew test $YOUR_FORMULA && brew linkage $YOUR_FORMULA

This is expected (although undesirable) as our builds are not yet reproducible.

Thanks a lot @SMillerDev & @MikeMcQuaid for the quick answers!

I understand “undesirable” expectedness of the non reproducibility of the builds. That makes sense.

Do you mean ... && brew linkage --test $YOUR_FORMULA?

Another question that I have:

It sometimes seems like the brew pull -v --bottle --bintray-org=ladislas --test-bot-user=ladislas https://github.com/ladislas/homebrew-avr/pull/7 command does not publish the file on bintray (although it was uploaded with --ci-upload). When it’s the case, I don’t have any errors. Am I missing something?

Yes, sorry.

What is the output in that case?

The output is just:

→ brew pull -v  --bottle --bintray-org=ladislas --test-bot-user=ladislas https://github.com/ladislas/homebrew-avr/pull/7
==> Fetching patch
Patch: https://github.com/ladislas/homebrew-avr/pull/7.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145    0   145    0     0    592      0 --:--:-- --:--:-- --:--:--   591
100  1004    0  1004    0     0    932      0 --:--:--  0:00:01 --:--:--   932
==> Applying patch
git am --whitespace=fix -3 /Users/ladislas/Library/Caches/Homebrew/7.patch
Applying: update readme
Using index info to reconstruct a base tree...
M	README.md
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
==> Patch closes issue #7
git commit --amend --signoff --allow-empty -q -m avarice: add 2.13 bottle.


Closes #7.
==> Patch changed:
git diff-tree -r --stat 2ef2815 HEAD

Looks like in that case it’s not detecting or publishing a new bottle.

Thanks @MikeMcQuaid!

I’ve kept reading the documentation, the source code and the tutorials available and I think I’m doing something wrong, not sure what exactly but I have some ideas to investigate.

I’m starting from scratch with a very simple project and I’ll see where it gets me :slight_smile:

If you add --warn-on-publish-failure it will check bintray to ensure that the bottle was published.

I haven’t dug through the testbot code, but if you’re rebottling something, you might need a formula revision to ensure the new bottle is downloaded… it might be possible that the upload failed or was skipped without a formula revision, or there is logic to prevent uploading or deploying without a revision… but, TBH, this is pretty close to pure speculation on my part, YMMV.

Thanks for the input @zbeekman! I’ve tried the option but with no success…

I’ve created a new post which explains the issue I’m facing in greater details.

Hi there! --

cc @MikeMcQuaid @SMillerDev @sjackman

I kept working on the topic and I’m close to presenting a complete solution.

One part I haven’t touched yet is the brew test $YOUR_FORMULA && brew linkage $YOUR_FORMULA.

My goal is to run a cron job every night to check if my formulae’s bottles still work and if not, create new bottles, upload them and publish them and change the formula’s bottle info.

My understanding of brew test && brew linkage (and please correct me if I’m wrong) is that:

  • this will test the formula to see if the current status of its dependencies allows it to work
  • if not, I can brew test-bot --root-url to rebuild a bottle and brew test-bot --ci-upload to upload it to Bintray

Now, how do I publish the bottle automatically? There is no PR in this case so I can’t run brew pull --bottle.

Should my CI script create a PR?

Thanks a lot for your help!

Best,
-- Ladislas

(I’d use brew linkage --test instead)

Yup!

Yup!

You may need to modify the brew pull code or just call the API directly. There’s been talk about separating the brew pull publication logic into a separate command that could be called by brew pull (CC @iMichka); that might be worth doing?

Personally I actually like the “bots create PRs and self-merge them” workflow; it provides visibility for both you and anyone watching your project.

Thanks a lot @MikeMcQuaid! :slight_smile:

(I’d use brew linkage --test instead)

Yes that’s what I meant!

You may need to modify the brew pull code or just call the API directly. There’s been talk about separating the brew pull publication logic into a separate command that could be called by brew pull (CC @iMichka); that might be worth doing?

Yes that’s what I thought as well, separating both upload and pull could be nice. Is it related to #5758? (@iMichka)

Personally I actually like the “bots create PRs and self-merge them” workflow; it provides visibility for both you and anyone watching your project.

Me too! When you mean PR, is it an actual Github PR or a local git checkout -b updated-formulae and then git checkout master && git merge updated-formulae ?

Yup! Would probably need to get that merged first.

Yeh, an actual PR! Means people can see on the repo if they are watching.

Yes, the GitHub PR should bump the revision to trigger a rebuild of the bottles.

1 Like

Is there a brew cmd that does that automagically?

To be explicit: bottles will be rebuilt regardless and can be used but a revision bump will force anyone with the current bottle instead to show it as outdated and upgrade with brew upgrade whereas a bottle rebuild increase will not.

brew bump-revision.

1 Like

Thanks all for your feedback! :slight_smile:

To summarize and make I’m not missing anything, here’s what I’ll do as pseudo-codish bash script that will run once a day:

  • (setup git config with MyTapTestBot username and email)
  • git checkout -b new-cron-test-job-branch
  • for each formula in Formula, run brew test formula && brew linkage --test formula
  • if it fails, call brew bump-revision formula
  • when all formulae have been tested, commit the bump-revision changes
  • push the new branch and create a pull request [1]

Now that the pull request is created, the usual brew test-bot --root-url && brew test-bot --ci-upload will run. [2]

Now for [1], how would one do that from the command line? bump-formula does that but not bump-revision (@MikeMcQuaid I’d be happy to add this ability as well to this command with a PR).

With [2], I’ll have to manually brew pull --bottle or I could read the branch name or the commit message and determine that it’s a revision bump but it seems a little far fetched.

Maybe I’m still missing something in the process :slight_smile:

We probably wouldn’t accept a PR for this but can use the hub command to do this from the CLI.

This is a limitation of our current tooling: we provide no way to publish bottles on bintray automatically. You would likely need to do this with any mechanism that also merged the PR.

Thanks for asking these questions; it helps us think how to better support these workflows.