Opened 5 years ago

Closed 4 years ago

#15503 closed bug (fixed)

Recent packages list is not updating

Reported by: cocobean Owned by: kallisti5
Priority: normal Milestone: Unscheduled
Component: Website/www.haiku-os.org Version: R1/Development
Keywords: pkgs Cc:
Blocked By: Blocking:
Platform: All

Description

Recent packages list is not updating on Haiku Website. Ref: https://www.haiku-os.org/#pkgs

Last updated: 06-Sep-2019

Attachments (1)

hds-fixes.diff (2.0 KB ) - added by kallisti5 5 years ago.
fixes for HDS

Download all attachments as: .zip

Change History (26)

comment:1 by waddlesplash, 5 years ago

Owner: changed from waddlesplash to apl-haiku
Status: newassigned

comment:2 by apl-haiku, 5 years ago

This happens because "latest" is only considering material newer than 90 days. Because the import of new packages into HDS from the repositories has been failing for about 4-5 months there are only old packages in there and therefore nothing is found that matches. If somebody is able to get the package import working then this issue should be resolved. See #14717.

comment:3 by apl-haiku, 5 years ago

Owner: changed from apl-haiku to kallisti5

comment:4 by kallisti5, 5 years ago

Tickets are more visible! :-) . Looking.

comment:5 by kallisti5, 5 years ago

Should be resolved, but HDS showing an error.

2020-02-03 19:09:48,321 / [job-run-a...] INFO  o.h.h.job.model.JobService - js <a01b..> (repositoryhpkringress); start
2020-02-03 19:09:48,325 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]]
2020-02-03 19:09:48,328 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo.info) to temporary file
2020-02-03 19:09:49,063 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-03 19:09:49,147 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1253761 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-03 19:09:49,155 / [job-run-a...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64
2020-02-03 19:09:49,355 / [job-run-a...] INFO  o.h.h.pkg.PkgImportServiceImpl - the version [1.0.3..3] of package [zopfli_source] did not exist; will create
2020-02-03 19:09:50,448 / [job-run-a...] ERROR o.h.h.pkg.PkgImportServiceImpl - unable to get the payload length for; PkgVersion[versionCoordinates=1.0.3-3]
java.net.ConnectException: No route to host

comment:6 by diver, 5 years ago

New packages are there now but they are shown in the wrong order.

comment:7 by cocobean, 5 years ago

Confirmed. Looks like HDS updated to latest HP pkg build, but noticed games like 0ad and Kobo do not show up in 'Categories->Games' queries - only the 'Alphabetical', 'Featured', and 'Recent Updates' queries.

Last edited 5 years ago by cocobean (previous) (diff)

comment:8 by diver, 5 years ago

This is because categories weren't set for them in HDS. I've just added them to Games.

comment:9 by cocobean, 5 years ago

Ok. Checked: zopfli-1.0.3

No issues with gcc2 build.

Might have lost connectivity altogether to host during import....

Last edited 5 years ago by cocobean (previous) (diff)

in reply to:  6 ; comment:10 by apl-haiku, 5 years ago

Replying to diver:

New packages are there now but they are shown in the wrong order.

Can you please explain what you mean by this.

in reply to:  5 comment:11 by apl-haiku, 5 years ago

Replying to kallisti5:

Should be resolved, but HDS showing an error.

...
java.net.ConnectException: No route to host

Can you please deploy version 1.0.115 of HDS and see if this problem is resolved. I think what is happening is that the retrieval of the packages' HPKG file-sizes is using the external URL base rather than the internal URL base. Hopefully this latest version will resolve this problem.

comment:12 by kallisti5, 5 years ago

ADD failed: Get http://central.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar: dial tcp: lookup central.maven.org on 1.1.1.1:53: no such host

comment:13 by kallisti5, 5 years ago

Looks like Maven changed or something... A quick search on maven.org showed the new URL as:

https://search.maven.org/remotecontent?filepath=org/eclipse/jetty/jetty-runner/9.4.20.v20190813/jetty-runner-9.4.20.v20190813.jar . ?

by kallisti5, 5 years ago

Attachment: hds-fixes.diff added

fixes for HDS

comment:14 by kallisti5, 5 years ago

1.0.115 . deployed.. Now seeing a Java Heap Space error:

` 2020-02-04 14:47:18,531 / [job-run-a...] ERROR o.h.h.pkg.PkgImportServiceImpl - unable to get the payload length for; PkgVersion[versionCoordinates=3.0.2-1] java.io.IOException: Java heap space

at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:567) at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119) at org.haiku.haikudepotserver.support.URLHelper.payloadLengthHttp(URLHelper.java:84) at org.haiku.haikudepotserver.support.URLHelper.payloadLength(URLHelper.java:55) at org.haiku.haikudepotserver.pkg.PkgImportServiceImpl.populatePayloadLength(PkgImportServiceImpl.java:261) at org.haiku.haikudepotserver.pkg.PkgImportServiceImpl.importFrom(PkgImportServiceImpl.java:155) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.runImportHpkrForRepositorySource(RepositoryHpkrIngressJobRunner.java:218) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.runForRepositorySource(RepositoryHpkrIngressJobRunner.java:109) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.lambda$run$1(RepositoryHpkrIngressJobRunner.java:90) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:87) at org.apache.cayenne.tx.DefaultTransactionManager.performInLocalTransaction(DefaultTransactionManager.java:59) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:52) at org.apache.cayenne.tx.DefaultTransactionManager.performInTransaction(DefaultTransactionManager.java:40) at org.apache.cayenne.configuration.server.ServerRuntime.performInTransaction(ServerRuntime.java:143) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.lambda$run$2(RepositoryHpkrIngressJobRunner.java:88) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.run(RepositoryHpkrIngressJobRunner.java:87) at org.haiku.haikudepotserver.repository.job.RepositoryHpkrIngressJobRunner.run(RepositoryHpkrIngressJobRunner.java:49) at org.haiku.haikudepotserver.job.LocalJobServiceImpl.runSpecificationInCurrentThread(LocalJobServiceImpl.java:298) at org.haiku.haikudepotserver.job.LocalJobServiceImpl.lambda$createInternalJobBySubmittingToExecutor$8(LocalJobServiceImpl.java:256) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.lang.OutOfMemoryError: Java heap space

at java.net.http/jdk.internal.net.http.ResponseSubscribers$ByteArraySubscriber.join(ResponseSubscribers.java:265) at java.net.http/jdk.internal.net.http.ResponseSubscribers$ByteArraySubscriber.onComplete(ResponseSubscribers.java:278) at java.net.http/jdk.internal.net.http.Http1Response$Http1BodySubscriber.complete(Http1Response.java:324) at java.net.http/jdk.internal.net.http.Http1Response$Http1BodySubscriber.onComplete(Http1Response.java:374) at java.net.http/jdk.internal.net.http.ResponseContent$FixedLengthBodyParser.accept(ResponseContent.java:555) at java.net.http/jdk.internal.net.http.ResponseContent$FixedLengthBodyParser.accept(ResponseContent.java:476) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.handle(Http1Response.java:771) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.handle(Http1Response.java:701) at java.net.http/jdk.internal.net.http.Http1Response$Receiver.accept(Http1Response.java:593) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.tryAsyncReceive(Http1Response.java:731) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:228) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$$Lambda$497/0x00000008407f2840.run(Unknown Source) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) ... 3 common frames omitted

`

comment:15 by kallisti5, 5 years ago

I'm bumping the heap to 320/512 in 1.0.115 and pushing to HDS master.

comment:16 by kallisti5, 5 years ago

@apl Heap was bumped to 320/512. Same error persists.

comment:17 by apl-haiku, 5 years ago

Hello @kallisti5; sorry I must have introduced a bug. I will investigate, but can't for another 12h or so. Can you please deploy the previous version (1.0.114) again in the interim?

comment:18 by kallisti5, 5 years ago

@apl, no worries. To be honest though.. the 1.0.115 seems to work "a bit better" than the 1.0.114 version. The whole app doesn't go down or anything... just 1 of 10 of the query updates.

Small side note.. I've been looking through all of the code for the interactions between HaikuDepot and HaikuDepotServer. Have you looked into GraphQL? It looks to have great java support and seems to be doing what the HDS interaction is doing, but with less restrictive data structures.

comment:19 by leavengood, 5 years ago

I have been doing GraphQL at work and I really like it. It provides type safety and some great tooling. You should be able to generate server side and client side code from the same GraphQL schema.

I am happy to help or provide advise on this. I don't yet know the HDS API but I plan to start learning it.

comment:20 by apl-haiku, 5 years ago

Have you looked into GraphQL? It looks to have great java support and seems to be doing what the HDS interaction is doing, but with less restrictive data structures.

I have also looked into GraphQL before, but I cannot see how this would help here in HDS to be honest. The existing server system as well as storage layer is working very well, is stable and getting to this point has been a large piece of work. It seems like completely changing the architecture of it would be a very large volume of work with nothing to gain.

Available development effort would be better spent on increasing the performance of the load of the data in the desktop application itself. The download and cache of the data should be sound, but I suspect the loading could be improved. For example;

  • is it possible to optimize the un-tar of the icons payload?
  • could icons be loaded on-demand? (#15370)
  • can it stop re-loading the data after an install? (#15617)
  • can the parsed data from HDS be loaded and married-up with the local HPKR data faster?
  • splitting download and parse (...Process classes) may help do more of the initial load in parallel?

Maybe we would be better to take this discussion over the HDS mailing list?

In any case I will sort out that memory bug and let you know when there is a 1.0.116 ready -- I'll try to test it a bit better this time! Thank you for deploying 1.0.115 anyway.

Last edited 5 years ago by apl-haiku (previous) (diff)

comment:21 by apl-haiku, 5 years ago

Hello @kallisi5; 1.0.116 is ready to be deployed when you have a moment. It is now using HEAD to get the length of the packages' payloads and should work better. Hopefully that is working properly. Let me know how you get on.

in reply to:  10 comment:22 by diver, 5 years ago

Replying to apl-haiku:

Replying to diver:

New packages are there now but they are shown in the wrong order.

Can you please explain what you mean by this.

For example QEMU is now on the second page in the Recent updates list. While it was really last updated on Nov 19, 2019 (https://github.com/haikuports/haikuports/commits/master/app-emulation/qemu).

Last edited 5 years ago by diver (previous) (diff)

comment:23 by kallisti5, 5 years ago

@apl-haiku; 1.0.116 has been deployed. The heap errors are gone, however things seem a lot slower (see timestamps below)

EDIT: Maybe not... it might be just less verbose.

2020-02-08 15:20:47,390 / [job-run-e...] INFO  o.h.h.job.model.JobService - js <e62d..> (repositoryhpkringress); start
2020-02-08 15:20:47,399 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_64]]
2020-02-08 15:20:47,401 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo.info) to temporary file
2020-02-08 15:20:47,653 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-08 15:20:47,740 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1256780 bytes for repository hpkr [RepositorySource[code=haikuports_x86_64]] (http://www:80/haikuports/master/x86_64/current/repo) to temporary file
2020-02-08 15:20:47,748 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_64
2020-02-08 15:20:51,467 -/saf [qtp405662939-35] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:20:58,355 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [921366937b] for url [http://www:80/haikuports/master/x86_64/current/packages/0ad_data-0.0.23b~alpha-1-any.hpkg]
2020-02-08 15:20:58,363 -/saf [qtp405662939-38] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:06,364 -/saf [qtp405662939-35] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:09,582 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [731517310b] for url [http://www:80/haikuports/master/x86_64/current/packages/qt5_source-5.14.0-1-source.hpkg]
2020-02-08 15:21:11,067 -/saf [qtp405662939-12] INFO  o.h.h.pkg.PkgServiceImpl - did increment the view counter for 'Pkg[name=devilutionx]'
2020-02-08 15:21:12,789 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [247698223b] for url [http://www:80/haikuports/master/x86_64/current/packages/krita_source-4.2.8.2-1-source.hpkg]
2020-02-08 15:21:16,842 kallisti5/chr [qtp405662939-38] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:18,042 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [353674230b] for url [http://www:80/haikuports/master/x86_64/current/packages/digikam_source-6.3.0-1-source.hpkg]
2020-02-08 15:21:23,148 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [426740791b] for url [http://www:80/haikuports/master/x86_64/current/packages/wesnoth_source-1.12.6-3-source.hpkg]
2020-02-08 15:21:32,205 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [395427629b] for url [http://www:80/haikuports/master/x86_64/current/packages/wesnoth-1.12.6-3-x86_64.hpkg]
2020-02-08 15:21:34,353 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [524258576b] for url [http://www:80/haikuports/master/x86_64/current/packages/libreoffice_source-6.3.3.2-1-source.hpkg]
2020-02-08 15:21:38,693 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did process data for repository hpkr RepositorySource[code=haikuports_x86_64] in 50945ms
2020-02-08 15:21:38,697 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will import for repository source [RepositorySource[code=haikuports_x86_gcc2]]
2020-02-08 15:21:38,699 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy data for repository info [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo.info) to temporary file
2020-02-08 15:21:38,736 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will copy repository hpkr [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo) to temporary file
2020-02-08 15:21:38,772 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - did copy 1476665 bytes for repository hpkr [RepositorySource[code=haikuports_x86_gcc2]] (http://www:80/haikuports/master/x86_gcc2/current/repo) to temporary file
2020-02-08 15:21:38,773 / [job-run-e...] INFO  o.h.h.r.j.RepositoryHpkrIngressJobRunner - will process data for repository hpkr haikuports_x86_gcc2
2020-02-08 15:21:45,556 -/saf [qtp405662939-11] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:47,320 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [524272996b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/libreoffice_x86_source-6.3.3.2-1-source.hpkg]
2020-02-08 15:21:48,384 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp_python36 which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:21:48,384 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2]
2020-02-08 15:21:49,858 -/saf [qtp405662939-41] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:21:53,227 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko_python3 which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:21:53,227 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=2.6.0-1]
2020-02-08 15:21:55,914 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [247721958b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/krita_x86_source-4.2.8.2-1-source.hpkg]
2020-02-08 15:22:07,939 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp_python3 which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:22:07,939 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2]
2020-02-08 15:22:09,033 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.4.2-1 of slixmpp which is older or the same as the existing 1.4.2-2 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:22:09,033 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=1.4.2-2]
2020-02-08 15:22:11,134 / [job-run-e...] INFO  o.h.h.support.URLHelperService - did obtain length [731497482b] for url [http://www:80/haikuports/master/x86_gcc2/current/packages/qt5_x86_source-5.14.0-1-source.hpkg]
2020-02-08 15:22:15,207 -/saf [qtp405662939-15] INFO  o.h.haikudepotserver.api1.PkgApiImpl - search for pkgs found 15 results
2020-02-08 15:22:22,026 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko_python which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
2020-02-08 15:22:22,026 / [job-run-e...] INFO  o.h.h.pkg.PkgImportServiceImpl - deactivated PkgVersion[versionCoordinates=2.6.0-1]
2020-02-08 15:22:22,308 / [job-run-e...] WARN  o.h.h.pkg.PkgImportServiceImpl - imported a package version 1.18.2-1 of paramiko which is older or the same as the existing 2.6.0-1 -- will deactivate 1 pkg versions after the imported one and make the imported one as latest
Last edited 5 years ago by kallisti5 (previous) (diff)

comment:24 by cocobean, 4 years ago

@kallisti5 - Seems resolved for 1.0.123. Please close this ticket.

comment:25 by kallisti5, 4 years ago

Resolution: fixed
Status: assignedclosed

agree. this one seems stale.

Note: See TracTickets for help on using tickets.