Recent packages list is not updating

[cocobean]

Recent packages list is not updating on Haiku Website. Ref: ​https://www.haiku-os.org/#pkgs

Last updated: 06-Sep-2019

[apl-haiku]

This happens because "latest" is only considering material newer than 90 days. Because the import of new packages into HDS from the repositories has been failing for about 4-5 months there are only old packages in there and therefore nothing is found that matches. If somebody is able to get the package import working then this issue should be resolved. See #14717.

[kallisti5]

Tickets are more visible! :-) . Looking.

[kallisti5]

Should be resolved, but HDS showing an error.

2020-02-03 19:09:48,321 / [job-run-a...] INFO  o.h.h.job.model.JobService - js <a01b..> (repositoryhpkringress); start
2020-02-03 19:09:48,325 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]]
2020-02-03 19:09:48,328 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo.info) to temporary file
2020-02-03 19:09:49,063 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-03 19:09:49,147 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1253761 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-03 19:09:49,155 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64
2020-02-03 19:09:49,355 / [job-run-a...] INFO  o.h.h.pkg.PkgImportServiceImpl - the version [1.0.3..3] of package [zopfli_source] did not exist; will create
2020-02-03 19:09:50,448 / [job-run-a...] ERROR o.h.h.pkg.PkgImportServiceImpl - unable to get the payload length for; PkgVersion[versionCoordinates=1.0.3-3]
java.net.ConnectException: No route to host

[diver]

New packages are there now but they are shown in the wrong order.

[cocobean]

Confirmed. Looks like HDS updated to latest HP pkg build, but noticed games like 0ad and Kobo do not show up in 'Categories->Games' queries - only the 'Alphabetical', 'Featured', and 'Recent Updates' queries.

[diver]

This is because categories weren't set for them in HDS. I've just added them to Games.

[cocobean]

Ok. Checked: zopfli-1.0.3

No issues with gcc2 build.

Might have lost connectivity altogether to host during import....

[apl-haiku]

Replying to diver:

New packages are there now but they are shown in the wrong order.

Can you please explain what you mean by this.

[apl-haiku]

Replying to kallisti5:

Should be resolved, but HDS showing an error.

...
java.net.ConnectException: No route to host

Can you please deploy version 1.0.115 of HDS and see if this problem is resolved. I think what is happening is that the retrieval of the packages' HPKG file-sizes is using the external URL base rather than the internal URL base. Hopefully this latest version will resolve this problem.

[kallisti5]

ADD failed: Get ​http://central.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar: dial tcp: lookup central.maven.org on 1.1.1.1:53: no such host

[kallisti5]

Looks like Maven changed or something... A quick search on maven.org showed the new URL as:

​https://search.maven.org/remotecontent?filepath=org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar . ?

[kallisti5]

fixes for HDS

[kallisti5]

1.0.115 . deployed.. Now seeing a Java Heap Space error:

` 2020-02-04 14:47:18,531 / [job-run-a...] ERROR o.h.h.pkg.PkgImportServiceImpl - unable to get the payload length for; PkgVersion[versionCoordinates=3.0.2-1] java.io.IOException: Java heap space

at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:567) at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119) at org.haiku.haikudepotserver.support.URLHelper.payloadLengthHttp(URLHelper.java:84) at org.haiku.haikudepotserver.support.URLHelper.payloadLength(URLHelper.java:55) at org.haiku.haikudepotserver.pkg.PkgImportServiceImpl.populatePayloadLength(PkgImportServiceImpl.java:261) at org.haiku.haikudepotserver.pkg.PkgImportServiceImpl.importFrom(PkgImportServiceImpl.java:155) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.runImportHpkrForRepositorySource(RepositoryHpkrIngressJobRunner.java:218) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.runForRepositorySource(RepositoryHpkrIngressJobRunner.java:109) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.lambda$run$1(RepositoryHpkrIngressJobRunner.java:90) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87) at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40) at org.apache.cayenne.configuration.server.ServerRuntime.performInTransaction(ServerRuntime.java:143) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.lambda$run$2(RepositoryHpkrIngressJobRunner.java:88) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.run(RepositoryHpkrIngressJobRunner.java:87) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.run(RepositoryHpkrIngressJobRunner.java:49) at org.haiku.haikudepotserver.job.LocalJobServiceImpl.runSpecificationInCurrentThread(LocalJobServiceImpl.java:298) at org.haiku.haikudepotserver.job.LocalJobServiceImpl.lambda$createInternalJobBySubmittingToExecutor$8(LocalJobServiceImpl.java:256) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.lang.OutOfMemoryError: Java heap space

at java.net.http/jdk.internal.net.http.ResponseSubscribers$ByteArraySubscriber.join(ResponseSubscribers.java:265) at java.net.http/jdk.internal.net.http.ResponseSubscribers$ByteArraySubscriber.onComplete(ResponseSubscribers.java:278) at java.net.http/jdk.internal.net.http.Http1Response$Http1BodySubscriber.complete(Http1Response.java:324) at java.net.http/jdk.internal.net.http.Http1Response$Http1BodySubscriber.onComplete(Http1Response.java:374) at java.net.http/jdk.internal.net.http.ResponseContent$FixedLengthBodyParser.accept(ResponseContent.java:555) at java.net.http/jdk.internal.net.http.ResponseContent$FixedLengthBodyParser.accept(ResponseContent.java:476) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.handle(Http1Response.java:771) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.handle(Http1Response.java:701) at java.net.http/jdk.internal.net.http.Http1Response$Receiver.accept(Http1Response.java:593) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.tryAsyncReceive(Http1Response.java:731) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:228) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$$Lambda$497/0x00000008407f2840.run(Unknown Source) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) ... 3 common frames omitted

`

[kallisti5]

I'm bumping the heap to 320/512 in 1.0.115 and pushing to HDS master.

[kallisti5]

@apl Heap was bumped to 320/512. Same error persists.

[apl-haiku]

Hello @kallisti5; sorry I must have introduced a bug. I will investigate, but can't for another 12h or so. Can you please deploy the previous version (1.0.114) again in the interim?

[kallisti5]

@apl, no worries. To be honest though.. the 1.0.115 seems to work "a bit better" than the 1.0.114 version. The whole app doesn't go down or anything... just 1 of 10 of the query updates.

Small side note.. I've been looking through all of the code for the interactions between HaikuDepot and HaikuDepotServer. Have you looked into GraphQL? It looks to have great java support and seems to be doing what the HDS interaction is doing, but with less restrictive data structures.

[leavengood]

I have been doing GraphQL at work and I really like it. It provides type safety and some great tooling. You should be able to generate server side and client side code from the same GraphQL schema.

I am happy to help or provide advise on this. I don't yet know the HDS API but I plan to start learning it.

[apl-haiku]

Have you looked into GraphQL? It looks to have great java support and seems to be doing what the HDS interaction is doing, but with less restrictive data structures.

I have also looked into GraphQL before, but I cannot see how this would help here to be honest. The existing system as well as storage layer is working very well, is stable and getting to this point has been a large piece of work. It seems like completely changing the architecture of it would be a very large volume of work with nothing to gain.

Available development effort would be better spent on increasing the performance of the load of the data in the desktop application itself. The download and cache of the data should be sound, but I suspect the loading could be improved. For example;

• is it possible to optimize the un-tar of the icons payload?
• could icons be loaded on-demand? (#15370)
• can it stop re-loading the data after an install? (#15617)
• can the parsed data from HDS be loaded and married-up with the local HPKR data faster?
• splitting download and parse (...Process classes) may help do more of the initial load in parallel?

Maybe we would be better to take this discussion over the HDS mailing list?

In any case I will sort out that memory bug and let you know when there is a 1.0.116 ready -- I'll try to test it a bit better this time! Thank you for deploying 1.0.115 anyway.

[apl-haiku]

Hello @kallisi5; 1.0.116 is ready to be deployed when you have a moment. It is now using HEAD to get the length of the packages' payloads and should work better. Hopefully that is working properly. Let me know how you get on.

[diver]

Replying to apl-haiku:

Replying to diver:

New packages are there now but they are shown in the wrong order.

Can you please explain what you mean by this.

For example QEMU is now on the second page in the Recent updates list. While it was really last updated on Nov 19, 2019 (​https://github.com/haikuports/haikuports/commits/master/app-emulation/qemu).

[kallisti5]

@apl-haiku; 1.0.116 has been deployed. The heap errors are gone, however things seem a lot slower (see timestamps below)

EDIT: Maybe not... it might be just less verbose.

2020-02-08 15:20:47,390 / [job-run-e...] INFO  o.h.h.job.model.JobService - js <e62d..> (repositoryhpkringress); start
2020-02-08 15:20:47,399 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]]
2020-02-08 15:20:47,401 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo.info) to temporary file
2020-02-08 15:20:47,653 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-08 15:20:47,740 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1256780 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-08 15:20:47,748 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64
2020-02-08 15:20:51,467 -/saf [qtp405662939-35] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:20:58,355 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [921366937b] for url [http://www:80/haikuports/master/x86_64/current/packages/0ad_data-0.0.23b~alpha-1-any.hpkg]
2020-02-08 15:20:58,363 -/saf [qtp405662939-38] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:06,364 -/saf [qtp405662939-35] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:09,582 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [731517310b] for url [http://www:80/haikuports/master/x86_64/current/packages/qt5_source-5.14.0-1-source.hpkg]
2020-02-08 15:21:11,067 -/saf [qtp405662939-12] INFO  o.h.h.pkg.PkgServiceImpl - did increment the view counter for 'Pkg[name=devilutionx]'
2020-02-08 15:21:12,789 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [247698223b] for url [http://www:80/haikuports/master/x86_64/current/packages/krita_source-4.2.8.2-1-source.hpkg]
2020-02-08 15:21:16,842 kallisti5/chr [qtp405662939-38] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:18,042 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [353674230b] for url [http://www:80/haikuports/master/x86_64/current/packages/digikam_source-6.3.0-1-source.hpkg]
2020-02-08 15:21:23,148 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [426740791b] for url [http://www:80/haikuports/master/x86_64/current/packages/wesnoth_source-1.12.6-3-source.hpkg]
2020-02-08 15:21:32,205 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [395427629b] for url [http://www:80/haikuports/master/x86_64/current/packages/wesnoth-1.12.6-3-x86_64.hpkg]
2020-02-08 15:21:34,353 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [524258576b] for url [http://www:80/haikuports/master/x86_64/current/packages/libreoffice_source-6.3.3.2-1-source.hpkg]
2020-02-08 15:21:38,693 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did process data for repository hpkr RepositorySource[code=haikuports_x86_64] in 50945ms
2020-02-08 15:21:38,697 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_gcc2]]
2020-02-08 15:21:38,699 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo.info) to temporary file
2020-02-08 15:21:38,736 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo) to temporary file
2020-02-08 15:21:38,772 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1476665 bytes for repository hpkr [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo) to temporary file
2020-02-08 15:21:38,773 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_gcc2
2020-02-08 15:21:45,556 -/saf [qtp405662939-11] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:47,320 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [524272996b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/libreoffice_x86_source-6.3.3.2-1-source.hpkg]
2020-02-08 15:21:48,384 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp_python36 which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:21:48,384 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2]
2020-02-08 15:21:49,858 -/saf [qtp405662939-41] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:53,227 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko_python3 which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:21:53,227 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=2.6.0-1]
2020-02-08 15:21:55,914 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [247721958b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/krita_x86_source-4.2.8.2-1-source.hpkg]
2020-02-08 15:22:07,939 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp_python3 which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:22:07,939 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2]
2020-02-08 15:22:09,033 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:22:09,033 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2]
2020-02-08 15:22:11,134 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [731497482b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/qt5_x86_source-5.14.0-1-source.hpkg]
2020-02-08 15:22:15,207 -/saf [qtp405662939-15] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:22:22,026 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko_python which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:22:22,026 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=2.6.0-1]
2020-02-08 15:22:22,308 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest

[cocobean]

@kallisti5 - Seems resolved for 1.0.123. Please close this ticket.

[kallisti5]

agree. this one seems stale.