'Featured packages' take very long to be populated.

[humdinger]

This is hrev56588, 64bit.

HaikuDepot seems to take much longer "Synchronizing package data from HaikuPorts" than it used to.

While the "All packages" list view gets populated quite quickly, the "Featured packages" icon view is empty for about 20 seconds. When started from Terminal, I see that the output pauses a long time after these 2 lines:

@04200699 {I} <t:18> [Node<ServerRepositoryDataUpdateProcess>] finished process in thread 2.820583 seconds
@04201699 {I} <t:21> [Node<ServerPkgDataUpdateProcess<Haiku>>] finished process in thread 3.000769 seconds

After the pause the icon view is populated and the rest follows quickly:

@04217093 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] will redirect to; https://depot.haiku-os.org/__secured/jobdata/d22670c1-c85b-457d-a3af-1c61d364a959/download
@04217093 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] will stream 'https://depot.haiku-os.org/__secured/jobdata/d22670c1-c85b-457d-a3af-1c61d364a959/download' to [/boot/system/cache/tmp/fileCHeMay]
@04219144 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] did complete streaming data [973067 bytes]
@04219145 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] did fetch data
@04219145 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] will process data
@04222009 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] did process 7390 packages' data in  (  2.86 secs)
@04222009 {I} <t:22> [ServerPkgDataUpdateProcess<HaikuPorts>] did process data
@04222009 {I} <t:22> [Node<ServerPkgDataUpdateProcess<HaikuPorts>>] finished process in thread 22.810651 seconds
@04222010 {I} <t:14> process coordinator [BulkLoad] did complete
@04344838 {I} <t:0> will stop all queued process coordinators

Does the redirect take that very long? Can it be sped up?

[humdinger]

Forgot to mention: these delays occur whenever you launch HaikuDepot. One might expect that through caching, subsequent launches should be faster, no?

[apl-haiku]

I have also observed this recently; I will try to find some time to look at it soon.

[madmax]

Server and not app issue?

I've tried a few requests with curl for https://depot.haiku-os.org/__pkg/all-haikuports_x86_64-en.json.gz. The 302 response from the server to https://depot.haiku-os.org/__secured/jobdata/<guid>/download takes 15-20 seconds, except when two consecutive requests return the same guid, in which case the second one takes virtually nothing.

[humdinger]

@kallisti5: Did you see madmax' comment above? Anything on the server side as far as you can see?

I fear this slowness won't reflect positively on our nice package system in those beta4 reviews...

[apl-haiku]

I am working on getting some masked data to do some performance testing with. In the meantime, to temporarily reduce the impact of the problem I have just done a build 1.0.137 which has a longer cache time for this specific data and so it's less likely to impact users. I will draw up some notes about a longer-term resolution soon.

[apl-haiku]

Hopefully the change in 1.0.137 improved the situation for people for now. You can find my thoughts on how to ideally improve the performance of this here; ​http://www.lindesay.co.nz/blog/2023/2023-hds-bulk-data-design-options/