Opened 11 months ago

Closed 7 months ago

#13937 closed task (fixed)

Repositories aren't generated anymore

Reported by: pulkomandy Owned by: haiku-sysadmin
Priority: blocker Milestone: R1/beta1
Component: Sys-Admin Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description (last modified by pulkomandy)

Since 3rd of january, there was no update of the "haiku" package repo. The buildbots are apparently not triggered

Maybe related to the switch to Gerrit? Did we forgot to plug some hook?

Change History (27)

comment:1 Changed 11 months ago by pulkomandy

Description: modified (diff)

comment:2 Changed 11 months ago by kallisti5

odd... the only thing I know of at the moment is git hooks not running to trigger builds on commit. Maybe the nightly images have some kind of "change" dependency in them?

I've looked through the web of cron scripts on vmrepo and don't see anything to trigger nightlies. Let me see if manually calling the commit hook makes a nightly happen.

comment:3 Changed 11 months ago by pulkomandy

IIRC they are nightlies, so triggered every night (not from a git hook). I think the buildbot install triggers that by itself, but I seem to have lost my password to get there and check.

(according to previous builds: "The Nightly scheduler named 'nightly-haiku-master' triggered this build")

I would assume it somehow fails to poll the repo for changes or something similar?

comment:4 Changed 11 months ago by korli

The post commit hook of the git repo did check the list of packages to add/remove, then sent a job to regenerate the repository (I suppose including the haiku packages).

comment:5 Changed 10 months ago by korli

I did push the backlog of commits to the buildbot, so that the repository is up to date as of hrev51791. Still needs a proper fix though.

comment:6 Changed 10 months ago by korli

ping

comment:7 Changed 10 months ago by kallisti5

Resolution: fixed
Status: newclosed

should be fixed now. Had to compile and install a bunch of twisted python modules required for the silly-simple hook script to tell buildbot a new commit was pushed :-|

Last edited 10 months ago by kallisti5 (previous) (diff)

comment:8 Changed 9 months ago by korli

Resolution: fixed
Status: closedreopened

Seems broken as of hrev51857.

comment:9 Changed 9 months ago by kallisti5

Yeah, we ran out of space on baron which started it... https://download.haiku-os.org/haiku-repositories/master/x86_64/r1~alpha4_pm_hrev51856/ looks like the last repo to generate.

I thought it was fixed after freeing some space... but looks like something is broken. This stuff is held together with hopes, dreams, and a little python.

I'll see if I can figure it out today.

comment:10 Changed 9 months ago by kallisti5

actually... that looks like the issue: https://buildbot.haiku-os.org/builders/haiku-nightlies-x86_64/builds/1380/steps/upload%20nightlies/logs/stdio

I wonder why buildbot didn't detect that as a failure :-|

comment:11 Changed 9 months ago by korli

Also no emails are sent and the hrev tags aren't applied anymore on new commits after hrev51857.

Last edited 9 months ago by korli (previous) (diff)

comment:12 Changed 9 months ago by kallisti5

Yup. This has been corrected.

I recently moved the git hooks into github. I also applied some security updates to maui (as well as bumping the version of docker-compose we used to a /usr/local/bin version to support "scale")

I logged into the gerrit container, and the hook scripts were missing for some reason from /var/gerrit/review_site/hooks. So it looks like the mount didn't work for some reason when bringing up the containers.

I rebooted to apply the security patches (which included docker + kernel), and this issue seems to be resolved (the hooks are now mounted where they should be)

I'll start checking these hooks by hand occasionally when I docker up/down to make sure it doesn't happen again.

Thanks for the heads up so quickly... We missed one tag, but it shouldn't be the end of the world.

comment:13 Changed 9 months ago by kallisti5

Resolution: fixed
Status: reopenedclosed

comment:14 Changed 9 months ago by korli

I'm reopening, there are still problems with tagging, plus I pushed a commit which doesn't appear in cgit, buildbot, github, or the commits mailing list.

comment:15 Changed 9 months ago by korli

Resolution: fixed
Status: closedreopened

comment:16 Changed 9 months ago by kallisti5

grumble. So a little history:

  • We had to rewrite the git hooks for Gerrit since it uses a non-standard git hook design.
  • The hooks seemed to be working for a while.
  • I put the "generic/common" hooks into the git repo and mounted it within the container a few days ago.
  • Stuff seemed to silently stop working again (even though dry-runs worked fine)
  • Logging from gerrit on the hook execution is shite

Now! I'm a bit confused. The hook triggering the buildbot repo generation *seemed* to work on the previous commit (unless someone did it manually? Not sure)

I just tried to run the hook by hand... and see that a lockfile is stuck.

GIT_DIR=/var/gerrit/review_site/git/haiku.git/ ./ref-updated --project haiku --refname refs/heads/master --submitter "Jérôme Duval <jerome.duval@xxxxx.com>" --oldrev 8e27fff0df12a06981ffc62b72184c4faf844f01 --newrev 823466613fe977a0dde894db7a1f496090c1c8e3

DEBUG: THE REFNAME IS: refs/origin/masters
DEBUG: THE SUBMITTER IS Jérôme Duval <jerome.duval@xxxxx.com>
DEBUG: THE OLDREV is 8e27fff0df12a06981ffc62b72184c4faf844f01
DEBUG: THE NEWREV is 823466613fe977a0dde894db7a1f496090c1c8e3
Obtaining lock...
lockfile: Sorry, giving up on "/var/gerrit/review_site/git/haiku.git/receive.lock"

So.. I cleared the receive.lock file and re-ran. Seems to have worked this time.

GIT_DIR=/var/gerrit/review_site/git/haiku.git/ ./ref-updated --project haiku --refname refs/heads/master --submitter "Jérôme Duval <jerome.duval@xxxxx.com>" --oldrev 8e27fff0df12a06981ffc62b72184c4faf844f01 --newrev 823466613fe977a0dde894db7a1f496090c1c8e3

DEBUG: THE REFNAME IS: refs/heads/master
DEBUG: THE PROJECT IS: haiku
DEBUG: THE SUBMITTER IS Jérôme Duval <jerome.duval@xxxxx.com>
DEBUG: THE OLDREV is 8e27fff0df12a06981ffc62b72184c4faf844f01
DEBUG: THE NEWREV is 823466613fe977a0dde894db7a1f496090c1c8e3
Obtaining lock...
master -> hrev51859
Notifying buildbot of 823466613fe977a0dde894db7a1f496090c1c8e3 @ refs/heads/master
Release lock...

So.. the big question is why that lockfile got stuck :-|

Last edited 9 months ago by korli (previous) (diff)

comment:17 in reply to:  16 Changed 9 months ago by korli

Replying to Alexander von Gluck:

Now! I'm a bit confused. The hook triggering the buildbot repo generation *seemed* to work on the previous commit (unless someone did it manually? Not sure)

I did push a few commits to the buildbot it yesterday to trigger the repository generation. Couldn't be done for the last non tagged commit though.

comment:18 Changed 9 months ago by kallisti5

We saw two commits and it seems like things are indeed working now. (tagged + buildbot notified)

so... the question is still why the lockfile got stuck :-|

Things seem to be working... lets monitor for the moment.

comment:19 Changed 9 months ago by korli

hrev51861 didn't get to the buildbot, not OK.

comment:20 Changed 9 months ago by kallisti5

Actually it did.

https://buildbot.haiku-os.org/builders/haiku-nightlies-x86_64/builds/1388

The build failed on geist bot 4.

comment:21 Changed 9 months ago by korli

No, this one is a nightly build. I speak of https://buildbot.haiku-os.org/builders/haiku-repository-x86_64

comment:22 Changed 8 months ago by korli

Same for hrev51863.

comment:23 Changed 8 months ago by kallisti5

Same for what? hrev51863 was pushed exactly when you made this comment. These builds do take a little time to begin / complete. I see the repos through hrev51862 built. (and hrev51863 queued) . I didn't queue these unless someone else did?

Last edited 8 months ago by kallisti5 (previous) (diff)

comment:24 Changed 8 months ago by korli

I queued the todays builds because new revisions don't get to the buildbot.

comment:25 Changed 8 months ago by korli

Resolution: fixed
Status: reopenedclosed

Seems fixed. Can be reopened if needed.

comment:26 Changed 7 months ago by korli

Resolution: fixed
Status: closedreopened

comment:27 Changed 7 months ago by kallisti5

Resolution: fixed
Status: reopenedclosed

Fixed. We enabled logging in the hooks which ended up breaking the buildbot notification

script due to permission issues :-|

We finally figured out Gerrit logging (and spawned several documentation fixes with them) and can troubleshoot hooks now.

Note: See TracTickets for help on using tickets.