Repositories aren't generated anymore

[pulkomandy]

Since 3rd of january, there was no update of the "haiku" package repo. The buildbots are apparently not triggered

• ​https://buildbot.haiku-os.org/builders/haiku-repository-x86_64
• ​https://buildbot.haiku-os.org/builders/haiku-repository-x86_gcc2_hybrid

Maybe related to the switch to Gerrit? Did we forgot to plug some hook?

[kallisti5]

odd... the only thing I know of at the moment is git hooks not running to trigger builds on commit. Maybe the nightly images have some kind of "change" dependency in them?

I've looked through the web of cron scripts on vmrepo and don't see anything to trigger nightlies. Let me see if manually calling the commit hook makes a nightly happen.

[pulkomandy]

IIRC they are nightlies, so triggered every night (not from a git hook). I think the buildbot install triggers that by itself, but I seem to have lost my password to get there and check.

(according to previous builds: "The Nightly scheduler named 'nightly-haiku-master' triggered this build")

I would assume it somehow fails to poll the repo for changes or something similar?

[korli]

The post commit hook of the git repo did check the list of packages to add/remove, then sent a job to regenerate the repository (I suppose including the haiku packages).

[korli]

I did push the backlog of commits to the buildbot, so that the repository is up to date as of hrev51791. Still needs a proper fix though.

[korli]

ping

[kallisti5]

should be fixed now. Had to compile and install a bunch of twisted python modules required for the silly-simple hook script to tell buildbot a new commit was pushed :-|

[korli]

Seems broken as of hrev51857.

[kallisti5]

Yeah, we ran out of space on baron which started it... ​https://download.haiku-os.org/haiku-repositories/master/x86_64/r1~alpha4_pm_hrev51856/ looks like the last repo to generate.

I thought it was fixed after freeing some space... but looks like something is broken. This stuff is held together with hopes, dreams, and a little python.

I'll see if I can figure it out today.

[kallisti5]

actually... that looks like the issue: ​https://buildbot.haiku-os.org/builders/haiku-nightlies-x86_64/builds/1380/steps/upload%20nightlies/logs/stdio

I wonder why buildbot didn't detect that as a failure :-|

[korli]

Also no emails are sent and the hrev tags aren't applied anymore on new commits after hrev51857.

[kallisti5]

Yup. This has been corrected.

I recently moved the git hooks into github. I also applied some security updates to maui (as well as bumping the version of docker-compose we used to a /usr/local/bin version to support "scale")

I logged into the gerrit container, and the hook scripts were missing for some reason from /var/gerrit/review_site/hooks. So it looks like the mount didn't work for some reason when bringing up the containers.

I rebooted to apply the security patches (which included docker + kernel), and this issue seems to be resolved (the hooks are now mounted where they should be)

I'll start checking these hooks by hand occasionally when I docker up/down to make sure it doesn't happen again.

Thanks for the heads up so quickly... We missed one tag, but it shouldn't be the end of the world.

[korli]

I'm reopening, there are still problems with tagging, plus I pushed a commit which doesn't appear in cgit, buildbot, github, or the commits mailing list.

[kallisti5]

grumble. So a little history:

• We had to rewrite the git hooks for Gerrit since it uses a non-standard git hook design.
• The hooks seemed to be working for a while.
• I put the "generic/common" hooks into the git repo and mounted it within the container a few days ago.
• Stuff seemed to silently stop working again (even though dry-runs worked fine)
• Logging from gerrit on the hook execution is shite

Now! I'm a bit confused. The hook triggering the buildbot repo generation *seemed* to work on the previous commit (unless someone did it manually? Not sure)

I just tried to run the hook by hand... and see that a lockfile is stuck.

GIT_DIR=/var/gerrit/review_site/git/haiku.git/ ./ref-updated --project haiku --refname refs/heads/master --submitter "Jérôme Duval <jerome.duval@xxxxx.com>" --oldrev 8e27fff0df12a06981ffc62b72184c4faf844f01 --newrev 823466613fe977a0dde894db7a1f496090c1c8e3

DEBUG: THE REFNAME IS: refs/origin/masters
DEBUG: THE SUBMITTER IS Jérôme Duval <jerome.duval@xxxxx.com>
DEBUG: THE OLDREV is 8e27fff0df12a06981ffc62b72184c4faf844f01
DEBUG: THE NEWREV is 823466613fe977a0dde894db7a1f496090c1c8e3
Obtaining lock...
lockfile: Sorry, giving up on "/var/gerrit/review_site/git/haiku.git/receive.lock"

So.. I cleared the receive.lock file and re-ran. Seems to have worked this time.

GIT_DIR=/var/gerrit/review_site/git/haiku.git/ ./ref-updated --project haiku --refname refs/heads/master --submitter "Jérôme Duval <jerome.duval@xxxxx.com>" --oldrev 8e27fff0df12a06981ffc62b72184c4faf844f01 --newrev 823466613fe977a0dde894db7a1f496090c1c8e3

DEBUG: THE REFNAME IS: refs/heads/master
DEBUG: THE PROJECT IS: haiku
DEBUG: THE SUBMITTER IS Jérôme Duval <jerome.duval@xxxxx.com>
DEBUG: THE OLDREV is 8e27fff0df12a06981ffc62b72184c4faf844f01
DEBUG: THE NEWREV is 823466613fe977a0dde894db7a1f496090c1c8e3
Obtaining lock...
master -> hrev51859
Notifying buildbot of 823466613fe977a0dde894db7a1f496090c1c8e3 @ refs/heads/master
Release lock...

So.. the big question is why that lockfile got stuck :-|

[korli]

Replying to Alexander von Gluck:

Now! I'm a bit confused. The hook triggering the buildbot repo generation *seemed* to work on the previous commit (unless someone did it manually? Not sure)

I did push a few commits to the buildbot it yesterday to trigger the repository generation. Couldn't be done for the last non tagged commit though.

[kallisti5]

We saw two commits and it seems like things are indeed working now. (tagged + buildbot notified)

so... the question is still why the lockfile got stuck :-|

Things seem to be working... lets monitor for the moment.

[korli]

hrev51861 didn't get to the buildbot, not OK.

[kallisti5]

Actually it did.

​https://buildbot.haiku-os.org/builders/haiku-nightlies-x86_64/builds/1388

The build failed on geist bot 4.

[korli]

No, this one is a nightly build. I speak of ​https://buildbot.haiku-os.org/builders/haiku-repository-x86_64

[korli]

Same for hrev51863.

[kallisti5]

Same for what? hrev51863 was pushed exactly when you made this comment. These builds do take a little time to begin / complete. I see the reps through hrev51862 built. (and hrev51863 queued) . I didn't queue these unless someone else did?

[korli]

I queued the todays builds because new revisions don't get to the buildbot.

[korli]

Seems fixed. Can be reopened if needed.

[korli]

Reopening. ​https://buildbot.haiku-os.org/builders/haiku-repository-x86_64 not triggered since a week.

[kallisti5]

Fixed. We enabled logging in the hooks which ended up breaking the buildbot notification

script due to permission issues :-|

We finally figured out Gerrit logging (and spawned several documentation fixes with them) and can troubleshoot hooks now.