Opened 21 months ago

Closed 3 months ago

Last modified 2 months ago

#14717 closed bug (fixed)

HaikuDepot Server doesn't get new packages

Reported by: CodeforEvolution Owned by: kallisti5
Priority: normal Milestone: R1/beta2
Component: Website/HaikuDepotServer Version:
Keywords: Cc: kallisti5
Blocked By: Blocking: #14727, #15265
Platform: All

Description

I’ve found despite new packages being added and updated on my system, such new packages and updates haven’t been represented in the packages tab on the main Haiku website as normal.

Attachments (1)

12-29-2019.log (238.4 KB ) - added by kallisti5 8 months ago.

Download all attachments as: .zip

Change History (26)

comment:1 by waddlesplash, 21 months ago

Cc: kallisti5 added
Component: Website/www.haiku-os.orgWebsite
Owner: changed from waddlesplash to apl-haiku
Status: newassigned

comment:2 by waddlesplash, 21 months ago

Blocking: 14727 added

comment:3 by waddlesplash, 21 months ago

Summary: Packages Tab on Haiku Website Doesn’t UpdateHaikuDepot Server doesn't get new packages

comment:4 by apl-haiku, 21 months ago

It would be easier if we keep operational queries on the mailing list in the first instance. That would be helpful because the problem here is most likely related to the HDS deployment environment rather than a software bug. As I no longer have access to the deployment environment, I cannot diagnose this issue.

Last edited 21 months ago by apl-haiku (previous) (diff)

comment:5 by apl-haiku, 21 months ago

Owner: changed from apl-haiku to kallisti5

comment:6 by exlmotodev, 21 months ago

Any news?

comment:7 by kallisti5, 20 months ago

There aren't many details in this ticket, but the original description was the biggest give-away "Packages Tab on Haiku Website Doesn’t Update"

I never added the cron jobs per https://github.com/haiku/haikudepotserver/issues/152. I went ahead and configured the import job for haikuports.

I think I need to add additional cron jobs for the other configured repositories?

  • BeSly
  • Clasqm
  • FatElk

I really wish 152 above would get some love :-| Cron jobs work, but are kind of a hack.

comment:8 by apl-haiku, 20 months ago

Hello Alex; You do not need to setup cron tasks for each repository. You only need to setup the following cron tasks;

32 * * * * curl ...../__maintenance/hourly
45 3 * * * curl ...../__maintenance/daily

In particular, the daily maintenance task scheduled with cron above will, amongst other things, refresh the data for all active repositories in the HDS system.

As part of a given repository's presumably scripted (Python etc...) build/update process it is advisable but not necessary for the repository build script (note: not cron) to invoke a URL such as...

/__repository/.../import

...when it (eg: Python script) finishes building/updating the repository. The documentation for this HDS endpoint are described at [1]. By having the repository build/update script do this, HDS will import the repository data faster without having to wait for periodic polls performed during the HDS daily maintenance task.

I am happy to help anybody with questions about this on the HDS mailing list.

I will take a look to see if I can setup task-scheduling inside HDS using Quartz scheduler, but am presently (slowly) busy with changes in the C++ code of the desktop application.

[1] https://depot.haiku-os.org/__docs/ch13.html#api-importrepositorydata

comment:9 by waddlesplash, 12 months ago

Blocking: 15265 added

comment:10 by diver, 12 months ago

Resolution: fixed
Status: assignedclosed

Seems to be fixed now.

comment:11 by diver, 11 months ago

Component: WebsiteWebsite/HaikuDepotServer

And now it doesn't get new packages again.

comment:12 by apl-haiku, 11 months ago

The cron based scheduling discussed above was replaced with an internal-application-server scheduler in commit 656292be79c087dc267418ad0f5fb38df3c3b7ba (2018-12-18). This was then deployed in 1.0.108 sometime after 2019-01-25.

comment:13 by diver, 8 months ago

Resolution: fixed
Status: closedreopened

comment:14 by kallisti5, 8 months ago

Definitely something wrong with HaikuDepot here. The last update date on the repos is September. Navigating into the individual repos then shows a later last update date.

<arm chair debugging> The scheduling doesn't seem to be applying at a high enough level? </arm chair debugging>

comment:15 by apl-haiku, 8 months ago

Here is a log trace from my locally running HDS to help you obtain some log lines to search for.

2019-12-17 18:30:00,112 / [job-run-3...] INFO  o.h.h.job.model.JobService - js <3451..> (repositoryhpkringress); start
2019-12-17 18:30:00,126 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]]
2019-12-17 18:30:00,186 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo.info) to temporary file
2019-12-17 18:30:01,558 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo) to temporary file
2019-12-17 18:30:06,026 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1224361 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo) to temporary file
2019-12-17 18:30:06,037 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64
2019-12-17 18:30:42,804 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did process data for repository hpkr RepositorySource[code=haikuports_x86_64] in 36767ms
2019-12-17 18:30:42,843 / [job-run-3...] INFO  o.h.h.job.model.JobService - js <3451..> (repositoryhpkringress); finish

It appears that the data at this URL curl "https://depot.haiku-os.org/__repository/haikuports/source/haikuports_x86_64/import" is not changing?

If you want to manually trigger a repository synchronization to see if the import process is working then you can do this by using the following API call;

curl "https://depot.haiku-os.org/__repository/haikuports/source/haikuports_x86_64/import"

You will be able to obtain other codes for repositories or repository sources using the HDS web user interface.

Aside from this one-off API call there is a scheduled task that should run daily at 12:17 GMT+0 for importing all of the repositories. If this is working then you should see a log line;

did trigger daily maintenance

Can you please;

  • check that you see the 'maintenance' log line to confirm that the scheduled maintenance is working
  • invoke the one-off API call to update a repository and check you see log trace to confirm it is importing
  • establish why the HPKR data at the URL in the logs is not changing if this is the problem

by kallisti5, 8 months ago

Attachment: 12-29-2019.log added

comment:16 by kallisti5, 8 months ago

I've attached the logs from 12-29-2019. I see a lot of java backtraces and exceptions around the repository imports.

comment:17 by kallisti5, 8 months ago

it looks like the importer hits a HTTP 301 at besly.. and then the import process breaks preventing the rest of the repos from importing? Haiku repos now do a HTTP 303.. so HDS really needs to follow redirects :-)

comment:18 by apl-haiku, 8 months ago

Release 1.0.114 of HDS will switch from the legacy java HTTP client to the new one introduced in Java 10. This supports redirects which have been enabled for both fetching repository data (HPKR and info.repo driver files) as well as for getting the size of repository objects (HPKG files). Once this is deployed, hopefully the problem will be solved.

comment:19 by cocobean, 8 months ago

@apl-haiku - HDS 1.0.114

BeSly, Clasqm, FatElk - imported as of 12/31/2019.
HaikuPorts - No new packages importing yet. Last import: 2019-09-04
Last edited 8 months ago by cocobean (previous) (diff)

comment:20 by apl-haiku, 8 months ago

@kallisi5 -- yes it appears that it is still not working properly in the deployed environment. Can you please again get me some updated logs to review?

comment:21 by apl-haiku, 6 months ago

(copied from mailing list so there is a trace in one place)

On Thu, 2 Jan 2020, at 17:11, kallisti5@… wrote:

We're back to issues doing hair-pin Nat in docker swarm. (The original

fix isn't working anymore, but sure why)

On 2018-11-05 3:03 p.m., Alexander von Gluck IV wrote:

...Pretty much you can't go out the ingress and come back in the same ingress.

Assuming that you have internal names for the containers, you can set a "Forced Internal Base URL" for a Repository Source that is used by the server to obtain the data. Authenticate in the UI, navigate to the Repository Source and edit it. In this way you can refer to the Docker network name for the HTTP server serving the repository data rather than the world-visible hostname.

Could this be a solution?

comment:22 by kallisti5, 6 months ago

Fixedish in #15503. Waiting on more confirms.

comment:23 by CodeforEvolution, 3 months ago

I'm pretty certain this can be closed, the Haiku website has been consistent in staying up to date with new package builds.

comment:24 by waddlesplash, 3 months ago

Resolution: fixed
Status: reopenedclosed

comment:25 by nielx, 2 months ago

Milestone: R1/beta2

Assign tickets with status=closed and resolution=fixed within the R1/beta2 development window to the R1/beta2 Milestone

(final time)

Note: See TracTickets for help on using tickets.