Opened 7 years ago
Closed 6 years ago
#13937 closed task (fixed)
Repositories aren't generated anymore
Reported by: | pulkomandy | Owned by: | haiku-sysadmin |
---|---|---|---|
Priority: | blocker | Milestone: | R1/beta1 |
Component: | Sys-Admin | Version: | R1/Development |
Keywords: | Cc: | ||
Blocked By: | Blocking: | ||
Platform: | All |
Description (last modified by )
Since 3rd of january, there was no update of the "haiku" package repo. The buildbots are apparently not triggered
- https://buildbot.haiku-os.org/builders/haiku-repository-x86_64
- https://buildbot.haiku-os.org/builders/haiku-repository-x86_gcc2_hybrid
Maybe related to the switch to Gerrit? Did we forgot to plug some hook?
Change History (27)
comment:1 by , 7 years ago
Description: | modified (diff) |
---|
comment:2 by , 7 years ago
comment:3 by , 7 years ago
IIRC they are nightlies, so triggered every night (not from a git hook). I think the buildbot install triggers that by itself, but I seem to have lost my password to get there and check.
(according to previous builds: "The Nightly scheduler named 'nightly-haiku-master' triggered this build")
I would assume it somehow fails to poll the repo for changes or something similar?
comment:4 by , 7 years ago
The post commit hook of the git repo did check the list of packages to add/remove, then sent a job to regenerate the repository (I suppose including the haiku packages).
comment:5 by , 7 years ago
I did push the backlog of commits to the buildbot, so that the repository is up to date as of hrev51791. Still needs a proper fix though.
comment:7 by , 7 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
should be fixed now. Had to compile and install a bunch of twisted python modules required for the silly-simple hook script to tell buildbot a new commit was pushed :-|
comment:8 by , 7 years ago
Resolution: | fixed |
---|---|
Status: | closed → reopened |
Seems broken as of hrev51857.
comment:9 by , 7 years ago
Yeah, we ran out of space on baron which started it... https://download.haiku-os.org/haiku-repositories/master/x86_64/r1~alpha4_pm_hrev51856/ looks like the last repo to generate.
I thought it was fixed after freeing some space... but looks like something is broken. This stuff is held together with hopes, dreams, and a little python.
I'll see if I can figure it out today.
comment:10 by , 7 years ago
actually... that looks like the issue: https://buildbot.haiku-os.org/builders/haiku-nightlies-x86_64/builds/1380/steps/upload%20nightlies/logs/stdio
I wonder why buildbot didn't detect that as a failure :-|
comment:11 by , 7 years ago
Also no emails are sent and the hrev tags aren't applied anymore on new commits after hrev51857.
comment:12 by , 7 years ago
Yup. This has been corrected.
I recently moved the git hooks into github. I also applied some security updates to maui (as well as bumping the version of docker-compose we used to a /usr/local/bin version to support "scale")
I logged into the gerrit container, and the hook scripts were missing for some reason from /var/gerrit/review_site/hooks. So it looks like the mount didn't work for some reason when bringing up the containers.
I rebooted to apply the security patches (which included docker + kernel), and this issue seems to be resolved (the hooks are now mounted where they should be)
I'll start checking these hooks by hand occasionally when I docker up/down to make sure it doesn't happen again.
Thanks for the heads up so quickly... We missed one tag, but it shouldn't be the end of the world.
comment:13 by , 7 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
comment:14 by , 7 years ago
I'm reopening, there are still problems with tagging, plus I pushed a commit which doesn't appear in cgit, buildbot, github, or the commits mailing list.
comment:15 by , 7 years ago
Resolution: | fixed |
---|---|
Status: | closed → reopened |
follow-up: 17 comment:16 by , 7 years ago
grumble. So a little history:
- We had to rewrite the git hooks for Gerrit since it uses a non-standard git hook design.
- The hooks seemed to be working for a while.
- I put the "generic/common" hooks into the git repo and mounted it within the container a few days ago.
- Stuff seemed to silently stop working again (even though dry-runs worked fine)
- Logging from gerrit on the hook execution is shite
Now! I'm a bit confused. The hook triggering the buildbot repo generation *seemed* to work on the previous commit (unless someone did it manually? Not sure)
I just tried to run the hook by hand... and see that a lockfile is stuck.
GIT_DIR=/var/gerrit/review_site/git/haiku.git/ ./ref-updated --project haiku --refname refs/heads/master --submitter "Jérôme Duval <jerome.duval@xxxxx.com>" --oldrev 8e27fff0df12a06981ffc62b72184c4faf844f01 --newrev 823466613fe977a0dde894db7a1f496090c1c8e3 DEBUG: THE REFNAME IS: refs/origin/masters DEBUG: THE SUBMITTER IS Jérôme Duval <jerome.duval@xxxxx.com> DEBUG: THE OLDREV is 8e27fff0df12a06981ffc62b72184c4faf844f01 DEBUG: THE NEWREV is 823466613fe977a0dde894db7a1f496090c1c8e3 Obtaining lock... lockfile: Sorry, giving up on "/var/gerrit/review_site/git/haiku.git/receive.lock"
So.. I cleared the receive.lock file and re-ran. Seems to have worked this time.
GIT_DIR=/var/gerrit/review_site/git/haiku.git/ ./ref-updated --project haiku --refname refs/heads/master --submitter "Jérôme Duval <jerome.duval@xxxxx.com>" --oldrev 8e27fff0df12a06981ffc62b72184c4faf844f01 --newrev 823466613fe977a0dde894db7a1f496090c1c8e3 DEBUG: THE REFNAME IS: refs/heads/master DEBUG: THE PROJECT IS: haiku DEBUG: THE SUBMITTER IS Jérôme Duval <jerome.duval@xxxxx.com> DEBUG: THE OLDREV is 8e27fff0df12a06981ffc62b72184c4faf844f01 DEBUG: THE NEWREV is 823466613fe977a0dde894db7a1f496090c1c8e3 Obtaining lock... master -> hrev51859 Notifying buildbot of 823466613fe977a0dde894db7a1f496090c1c8e3 @ refs/heads/master Release lock...
So.. the big question is why that lockfile got stuck :-|
comment:17 by , 7 years ago
Replying to Alexander von Gluck:
Now! I'm a bit confused. The hook triggering the buildbot repo generation *seemed* to work on the previous commit (unless someone did it manually? Not sure)
I did push a few commits to the buildbot it yesterday to trigger the repository generation. Couldn't be done for the last non tagged commit though.
comment:18 by , 7 years ago
We saw two commits and it seems like things are indeed working now. (tagged + buildbot notified)
so... the question is still why the lockfile got stuck :-|
Things seem to be working... lets monitor for the moment.
comment:20 by , 7 years ago
Actually it did.
https://buildbot.haiku-os.org/builders/haiku-nightlies-x86_64/builds/1388
The build failed on geist bot 4.
comment:21 by , 7 years ago
No, this one is a nightly build. I speak of https://buildbot.haiku-os.org/builders/haiku-repository-x86_64
comment:23 by , 7 years ago
comment:24 by , 7 years ago
I queued the todays builds because new revisions don't get to the buildbot.
comment:25 by , 7 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
Seems fixed. Can be reopened if needed.
comment:26 by , 6 years ago
Resolution: | fixed |
---|---|
Status: | closed → reopened |
Reopening. https://buildbot.haiku-os.org/builders/haiku-repository-x86_64 not triggered since a week.
comment:27 by , 6 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
Fixed. We enabled logging in the hooks which ended up breaking the buildbot notification
script due to permission issues :-|
We finally figured out Gerrit logging (and spawned several documentation fixes with them) and can troubleshoot hooks now.
odd... the only thing I know of at the moment is git hooks not running to trigger builds on commit. Maybe the nightly images have some kind of "change" dependency in them?
I've looked through the web of cron scripts on vmrepo and don't see anything to trigger nightlies. Let me see if manually calling the commit hook makes a nightly happen.