HaikuDepot Server doesn't get new packages

[CodeforEvolution]

I’ve found despite new packages being added and updated on my system, such new packages and updates haven’t been represented in the packages tab on the main Haiku website as normal.

[apl-haiku]

It would be easier if we keep operational queries on the mailing list in the first instance. That would be helpful because the problem here is most likely related to the HDS deployment environment rather than a software bug. As I no longer have access to the deployment environment so I cannot diagnose this issue.

[exlmotodev]

Any news?

[kallisti5]

There aren't many details in this ticket, but the original description was the biggest give-away "Packages Tab on Haiku Website Doesn’t Update"

I never added the cron jobs per ​https://github.com/haiku/haikudepotserver/issues/152. I went ahead and configured the import job for haikuports.

I think I need to add additional cron jobs for the other configured repositories?

• BeSly
• Clasqm
• FatElk

I really wish 152 above would get some love :-| Cron jobs work, but are kind of a hack.

[apl-haiku]

Hello Alex; You do not need to setup cron tasks for each repository. You only need to setup the following cron tasks;

32 * * * * curl ...../__maintenance/hourly
45 3 * * * curl ...../__maintenance/daily

In particular, the daily maintenance task scheduled with cron above will, amongst other things, refresh the data for all active repositories in the HDS system.

As part of a given repository's presumably scripted (Python etc...) build/update process it is advisable but not necessary for the repository build script (note: not cron) to invoke a URL such as...

/__repository/.../import

...when it (eg: Python script) finishes building/updating the repository. The documentation for this HDS endpoint are described at [1]. By having the repository build/update script do this, HDS will import the repository data faster without having to wait for periodic polls performed during the HDS daily maintenance task.

I am happy to help anybody with questions about this on the HDS mailing list.

I will take a look to see if I can setup task-scheduling inside HDS using Quartz scheduler, but am presently (slowly) busy with changes in the C++ code of the desktop application.

[1] ​https://depot.haiku-os.org/__docs/ch13.html#api-importrepositorydata

[diver]

Seems to be fixed now.

[diver]

And now it doesn't get new packages again.

[apl-haiku]

The cron based scheduling discussed above was replaced with an internal-application-server scheduler in commit 656292be79c087dc267418ad0f5fb38df3c3b7ba (2018-12-18). This was then deployed in 1.0.108 sometime after 2019-01-25.

[kallisti5]

Definitely something wrong with HaikuDepot here. The last update date on the repos is September. Navigating into the individual repos then shows a later last update date.

<arm chair debugging> The scheduling doesn't seem to be applying at a high enough level? </arm chair debugging>

[apl-haiku]

Here is a log trace from my locally running HDS to help you obtain some log lines to search for.

2019-12-17 18:30:00,112 / [job-run-3...] INFO  o.h.h.job.model.JobService - js <3451..> (repositoryhpkringress); start
2019-12-17 18:30:00,126 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]]
2019-12-17 18:30:00,186 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo.info) to temporary file
2019-12-17 18:30:01,558 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo) to temporary file
2019-12-17 18:30:06,026 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1224361 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo) to temporary file
2019-12-17 18:30:06,037 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64
2019-12-17 18:30:42,804 / [job-run-3...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did process data for repository hpkr RepositorySource[code=haikuports_x86_64] in 36767ms
2019-12-17 18:30:42,843 / [job-run-3...] INFO  o.h.h.job.model.JobService - js <3451..> (repositoryhpkringress); finish

It appears that the data at this URL curl "https://depot.haiku-os.org/__repository/haikuports/source/haikuports_x86_64/import" is not changing?

If you want to manually trigger a repository synchronization to see if the import process is working then you can do this by using the following API call;

curl "https://depot.haiku-os.org/__repository/haikuports/source/haikuports_x86_64/import"

You will be able to obtain other codes for repositories or repository sources using the HDS web user interface.

Aside from this one-off API call there is a scheduled task that should run daily at 12:17 GMT+0 for importing all of the repositories. If this is working then you should see a log line;

did trigger daily maintenance

Can you please;

• check that you see the 'maintenance' log line to confirm that the scheduled maintenance is working
• invoke the one-off API call to update a repository and check you see log trace to confirm it is importing
• establish why the HPKR data at the URL in the logs is not changing if this is the problem

[kallisti5]

I've attached the logs from 12-29-2019. I see a lot of java backtraces and exceptions around the repository imports.

[kallisti5]

it looks like the importer hits a HTTP 301 at besly.. and then the import process breaks preventing the rest of the repos from importing? Haiku repos now do a HTTP 303.. so HDS really needs to follow redirects :-)

[apl-haiku]

Release 1.0.114 of HDS will switch from the legacy java HTTP client to the new one introduced in Java 10. This supports redirects which have been enabled for both fetching repository data (HPKR and info.repo driver files) as well as for getting the size of repository objects (HPKG files). Once this is deployed, hopefully the problem will be solved.

[cocobean]

@apl-haiku - HDS 1.0.114

BeSly, Clasqm, FatElk - imported as of 12/31/2019.
HaikuPorts - No new packages importing yet. Last import: 2019-09-04

[apl-haiku]

@kallisi5 -- yes it appears that it is still not working properly in the deployed environment. Can you please again get me some updated logs to review?

[apl-haiku]

(copied from mailing list so there is a trace in one place)

On Thu, 2 Jan 2020, at 17:11, kallisti5@… wrote:

We're back to issues doing hair-pin Nat in docker swarm. (The original

fix isn't working anymore, but sure why)

On 2018-11-05 3:03 p.m., Alexander von Gluck IV wrote:

...Pretty much you can't go out the ingress and come back in the same ingress.

Assuming that you have internal names for the containers, you can set a "Forced Internal Base URL" for a Repository Source that is used by the server to obtain the data. Authenticate in the UI, navigate to the Repository Source and edit it. In this way you can refer to the Docker network name for the HTTP server serving the repository data rather than the world-visible hostname.

Could this be a solution?

[kallisti5]

Fixedish in #15503. Waiting on more confirms.

[CodeforEvolution]

I'm pretty certain this can be closed, the Haiku website has been consistent in staying up to date with new package builds.

[nielx]

Assign tickets with status=closed and resolution=fixed within the R1/beta2 development window to the R1/beta2 Milestone

(final time)