Opened 5 years ago
Closed 4 years ago
#15503 closed bug (fixed)
Recent packages list is not updating
Reported by: | cocobean | Owned by: | kallisti5 |
---|---|---|---|
Priority: | normal | Milestone: | Unscheduled |
Component: | Website/www.haiku-os.org | Version: | R1/Development |
Keywords: | pkgs | Cc: | |
Blocked By: | Blocking: | ||
Platform: | All |
Description
Recent packages list is not updating on Haiku Website. Ref: https://www.haiku-os.org/#pkgs
Last updated: 06-Sep-2019
Attachments (1)
Change History (26)
comment:1 by , 5 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:2 by , 5 years ago
comment:3 by , 5 years ago
Owner: | changed from | to
---|
follow-up: 11 comment:5 by , 5 years ago
Should be resolved, but HDS showing an error.
2020-02-03 19:09:48,321 / [job-run-a...] INFO o.h.h.job.model.JobService - js <a01b..> (repositoryhpkringress); start 2020-02-03 19:09:48,325 / [job-run-a...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]] 2020-02-03 19:09:48,328 / [job-run-a...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo.info) to temporary file 2020-02-03 19:09:49,063 / [job-run-a...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file 2020-02-03 19:09:49,147 / [job-run-a...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1253761 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file 2020-02-03 19:09:49,155 / [job-run-a...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64 2020-02-03 19:09:49,355 / [job-run-a...] INFO o.h.h.pkg.PkgImportServiceImpl - the version [1.0.3..3] of package [zopfli_source] did not exist; will create 2020-02-03 19:09:50,448 / [job-run-a...] ERROR o.h.h.pkg.PkgImportServiceImpl - unable to get the payload length for; PkgVersion[versionCoordinates=1.0.3-3] java.net.ConnectException: No route to host
follow-up: 10 comment:6 by , 5 years ago
New packages are there now but they are shown in the wrong order.
comment:7 by , 5 years ago
Confirmed. Looks like HDS updated to latest HP pkg build, but noticed games like 0ad and Kobo do not show up in 'Categories->Games' queries - only the 'Alphabetical', 'Featured', and 'Recent Updates' queries.
comment:8 by , 5 years ago
This is because categories weren't set for them in HDS. I've just added them to Games.
comment:9 by , 5 years ago
Ok. Checked: zopfli-1.0.3
No issues with gcc2 build.
Might have lost connectivity altogether to host during import....
follow-up: 22 comment:10 by , 5 years ago
Replying to diver:
New packages are there now but they are shown in the wrong order.
Can you please explain what you mean by this.
comment:11 by , 5 years ago
Replying to kallisti5:
Should be resolved, but HDS showing an error.
... java.net.ConnectException: No route to host
Can you please deploy version 1.0.115 of HDS and see if this problem is resolved. I think what is happening is that the retrieval of the packages' HPKG file-sizes is using the external URL base rather than the internal URL base. Hopefully this latest version will resolve this problem.
comment:12 by , 5 years ago
ADD failed: Get http://central.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar: dial tcp: lookup central.maven.org on 1.1.1.1:53: no such host
comment:13 by , 5 years ago
Looks like Maven changed or something... A quick search on maven.org showed the new URL as:
comment:14 by , 5 years ago
1.0.115 . deployed.. Now seeing a Java Heap Space error:
`
2020-02-04 14:47:18,531 / [job-run-a...] ERROR o.h.h.pkg.PkgImportServiceImpl - unable to get the payload length for; PkgVersion[versionCoordinates=3.0.2-1]
java.io.IOException: Java heap space
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:567) at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119) at org.haiku.haikudepotserver.support.URLHelper.payloadLengthHttp(URLHelper.java:84) at org.haiku.haikudepotserver.support.URLHelper.payloadLength(URLHelper.java:55) at org.haiku.haikudepotserver.pkg.PkgImportServiceImpl.populatePayloadLength(PkgImportServiceImpl.java:261) at org.haiku.haikudepotserver.pkg.PkgImportServiceImpl.importFrom(PkgImportServiceImpl.java:155) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.runImportHpkrForRepositorySource(RepositoryHpkrIngressJobRunner.java:218) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.runForRepositorySource(RepositoryHpkrIngressJobRunner.java:109) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.lambda$run$1(RepositoryHpkrIngressJobRunner.java:90) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87) at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40) at org.apache.cayenne.configuration.server.ServerRuntime.performInTransaction(ServerRuntime.java:143) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.lambda$run$2(RepositoryHpkrIngressJobRunner.java:88) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.run(RepositoryHpkrIngressJobRunner.java:87) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.run(RepositoryHpkrIngressJobRunner.java:49) at org.haiku.haikudepotserver.job.LocalJobServiceImpl.runSpecificationInCurrentThread(LocalJobServiceImpl.java:298) at org.haiku.haikudepotserver.job.LocalJobServiceImpl.lambda$createInternalJobBySubmittingToExecutor$8(LocalJobServiceImpl.java:256) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.net.http/jdk.internal.net.http.ResponseSubscribers$ByteArraySubscriber.join(ResponseSubscribers.java:265) at java.net.http/jdk.internal.net.http.ResponseSubscribers$ByteArraySubscriber.onComplete(ResponseSubscribers.java:278) at java.net.http/jdk.internal.net.http.Http1Response$Http1BodySubscriber.complete(Http1Response.java:324) at java.net.http/jdk.internal.net.http.Http1Response$Http1BodySubscriber.onComplete(Http1Response.java:374) at java.net.http/jdk.internal.net.http.ResponseContent$FixedLengthBodyParser.accept(ResponseContent.java:555) at java.net.http/jdk.internal.net.http.ResponseContent$FixedLengthBodyParser.accept(ResponseContent.java:476) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.handle(Http1Response.java:771) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.handle(Http1Response.java:701) at java.net.http/jdk.internal.net.http.Http1Response$Receiver.accept(Http1Response.java:593) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.tryAsyncReceive(Http1Response.java:731) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:228) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$$Lambda$497/0x00000008407f2840.run(Unknown Source) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) ... 3 common frames omitted
`
comment:17 by , 5 years ago
Hello @kallisti5; sorry I must have introduced a bug. I will investigate, but can't for another 12h or so. Can you please deploy the previous version (1.0.114) again in the interim?
comment:18 by , 5 years ago
@apl, no worries. To be honest though.. the 1.0.115 seems to work "a bit better" than the 1.0.114 version. The whole app doesn't go down or anything... just 1 of 10 of the query updates.
Small side note.. I've been looking through all of the code for the interactions between HaikuDepot and HaikuDepotServer. Have you looked into GraphQL? It looks to have great java support and seems to be doing what the HDS interaction is doing, but with less restrictive data structures.
comment:19 by , 5 years ago
I have been doing GraphQL at work and I really like it. It provides type safety and some great tooling. You should be able to generate server side and client side code from the same GraphQL schema.
I am happy to help or provide advise on this. I don't yet know the HDS API but I plan to start learning it.
comment:20 by , 5 years ago
Have you looked into GraphQL? It looks to have great java support and seems to be doing what the HDS interaction is doing, but with less restrictive data structures.
I have also looked into GraphQL before, but I cannot see how this would help here in HDS to be honest. The existing server system as well as storage layer is working very well, is stable and getting to this point has been a large piece of work. It seems like completely changing the architecture of it would be a very large volume of work with nothing to gain.
Available development effort would be better spent on increasing the performance of the load of the data in the desktop application itself. The download and cache of the data should be sound, but I suspect the loading could be improved. For example;
- is it possible to optimize the un-tar of the icons payload?
- could icons be loaded on-demand? (#15370)
- can it stop re-loading the data after an install? (#15617)
- can the parsed data from HDS be loaded and married-up with the local HPKR data faster?
- splitting download and parse (
...Process
classes) may help do more of the initial load in parallel?
Maybe we would be better to take this discussion over the HDS mailing list?
In any case I will sort out that memory bug and let you know when there is a 1.0.116 ready -- I'll try to test it a bit better this time! Thank you for deploying 1.0.115 anyway.
comment:21 by , 5 years ago
Hello @kallisi5; 1.0.116 is ready to be deployed when you have a moment. It is now using HEAD
to get the length of the packages' payloads and should work better. Hopefully that is working properly. Let me know how you get on.
comment:22 by , 5 years ago
Replying to apl-haiku:
Replying to diver:
New packages are there now but they are shown in the wrong order.
Can you please explain what you mean by this.
For example QEMU is now on the second page in the Recent updates list. While it was really last updated on Nov 19, 2019 (https://github.com/haikuports/haikuports/commits/master/app-emulation/qemu).
comment:23 by , 5 years ago
@apl-haiku; 1.0.116 has been deployed. The heap errors are gone, however things seem a lot slower (see timestamps below)
EDIT: Maybe not... it might be just less verbose.
2020-02-08 15:20:47,390 / [job-run-e...] INFO o.h.h.job.model.JobService - js <e62d..> (repositoryhpkringress); start 2020-02-08 15:20:47,399 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]] 2020-02-08 15:20:47,401 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo.info) to temporary file 2020-02-08 15:20:47,653 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file 2020-02-08 15:20:47,740 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1256780 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file 2020-02-08 15:20:47,748 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64 2020-02-08 15:20:51,467 -/saf [qtp405662939-35] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:20:58,355 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [921366937b] for url [http://www:80/haikuports/master/x86_64/current/packages/0ad_data-0.0.23b~alpha-1-any.hpkg] 2020-02-08 15:20:58,363 -/saf [qtp405662939-38] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:21:06,364 -/saf [qtp405662939-35] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:21:09,582 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [731517310b] for url [http://www:80/haikuports/master/x86_64/current/packages/qt5_source-5.14.0-1-source.hpkg] 2020-02-08 15:21:11,067 -/saf [qtp405662939-12] INFO o.h.h.pkg.PkgServiceImpl - did increment the view counter for 'Pkg[name=devilutionx]' 2020-02-08 15:21:12,789 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [247698223b] for url [http://www:80/haikuports/master/x86_64/current/packages/krita_source-4.2.8.2-1-source.hpkg] 2020-02-08 15:21:16,842 kallisti5/chr [qtp405662939-38] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:21:18,042 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [353674230b] for url [http://www:80/haikuports/master/x86_64/current/packages/digikam_source-6.3.0-1-source.hpkg] 2020-02-08 15:21:23,148 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [426740791b] for url [http://www:80/haikuports/master/x86_64/current/packages/wesnoth_source-1.12.6-3-source.hpkg] 2020-02-08 15:21:32,205 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [395427629b] for url [http://www:80/haikuports/master/x86_64/current/packages/wesnoth-1.12.6-3-x86_64.hpkg] 2020-02-08 15:21:34,353 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [524258576b] for url [http://www:80/haikuports/master/x86_64/current/packages/libreoffice_source-6.3.3.2-1-source.hpkg] 2020-02-08 15:21:38,693 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - did process data for repository hpkr RepositorySource[code=haikuports_x86_64] in 50945ms 2020-02-08 15:21:38,697 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_gcc2]] 2020-02-08 15:21:38,699 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo.info) to temporary file 2020-02-08 15:21:38,736 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo) to temporary file 2020-02-08 15:21:38,772 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1476665 bytes for repository hpkr [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo) to temporary file 2020-02-08 15:21:38,773 / [job-run-e...] INFO o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_gcc2 2020-02-08 15:21:45,556 -/saf [qtp405662939-11] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:21:47,320 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [524272996b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/libreoffice_x86_source-6.3.3.2-1-source.hpkg] 2020-02-08 15:21:48,384 / [job-run-e...] WARN o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp_python36 which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest 2020-02-08 15:21:48,384 / [job-run-e...] INFO o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2] 2020-02-08 15:21:49,858 -/saf [qtp405662939-41] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:21:53,227 / [job-run-e...] WARN o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko_python3 which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest 2020-02-08 15:21:53,227 / [job-run-e...] INFO o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=2.6.0-1] 2020-02-08 15:21:55,914 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [247721958b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/krita_x86_source-4.2.8.2-1-source.hpkg] 2020-02-08 15:22:07,939 / [job-run-e...] WARN o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp_python3 which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest 2020-02-08 15:22:07,939 / [job-run-e...] INFO o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2] 2020-02-08 15:22:09,033 / [job-run-e...] WARN o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest 2020-02-08 15:22:09,033 / [job-run-e...] INFO o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2] 2020-02-08 15:22:11,134 / [job-run-e...] INFO o.h.h.support.URLHelperService - did obtain length [731497482b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/qt5_x86_source-5.14.0-1-source.hpkg] 2020-02-08 15:22:15,207 -/saf [qtp405662939-15] INFO o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results 2020-02-08 15:22:22,026 / [job-run-e...] WARN o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko_python which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest 2020-02-08 15:22:22,026 / [job-run-e...] INFO o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=2.6.0-1] 2020-02-08 15:22:22,308 / [job-run-e...] WARN o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
comment:25 by , 4 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
agree. this one seems stale.
This happens because "latest" is only considering material newer than 90 days. Because the import of new packages into HDS from the repositories has been failing for about 4-5 months there are only old packages in there and therefore nothing is found that matches. If somebody is able to get the package import working then this issue should be resolved. See #14717.