[HaikuDepot] cache featured packages

[diver]

HaikuDepot populates featured packages from depot.haiku-os.org on every launch which takes 2-3 seconds making it feel slow.

[stippi]

Actually, this information is already cached. But it could be cached better. At the moment, HaikuDepot is reading a cache file on disk for every known package. Inbetween featured packages, it is reading the cached information for a lot of non-featured packages which accounts for the time between two featured packages to appear in the list. Better would be to cache featured packages separately and use that information even before reading the individual package cache files.

[diver]

Could it be fixed in hrev50905?

[apl-haiku]

No, it is not fixed in hrev50905, but I do have a plan for that. Unfortunately it is going to take me a while to get to it though.

[diver]

With hrev51514 it now works much better. It downloads 25 MB of data in ~/config/cache/HaikuDepot (it stays in Deskbar after closing HD until it fninished downloading everything) and on the next HD run icon loading is quite fast. Is it possible to speed up featured packages icon loading on the first run?

[pulkomandy]

Would it be possible to pre-load this when compiling Haiku, so that there is some data already available on first start?

[apl-haiku]

@diver; The next improvement is to download all of the packages' meta-data in one HTTP request and to stream-load them into HD. This should be faster. The server-side for this change is complete and now I will slowly work on the client-side soon. I want the schema-to-class generation to happen in the Jamfile as suggested by Ingo, but it may take me a while to figure that out.

@PulkoMandy; I am sure there is a mechanism to do that, but the data would then be stale and so I think it would be better if it continued to be downloaded.

[diver]

I just tried hrev51541 and it got slower now. Before there was a 3 sec delay before icons were shown, now it's 8 seconds.

[apl-haiku]

@diver; Loading from a "cold start" (no cached data) should however be much faster now. I have more changes to come to help with starting with cached data.

[apl-haiku]

I was hoping that this recent set of changes in commit 3094fef308a6aaaef9827863a0a99640bdaaa5af would help quite a bit and it might have done, but perhaps a bit less than I had hoped.

Currently the desktop application has an entirely in-memory model. This means that it loads all of the packages' from the local operating system into memory and then gets the data from the server and merges that server data in. It does this for all packages in all repositories. Now that the community has a large number of packages, this is unavoidably going to take some time on each start.

A better approach would be for the desktop application to synchronize the local operating system's packages and the server data into a persistent local sqlite3 database. This would replace the in-memory model and the desktop application would query the database when it needs data instead of holding everything in-memory.

In this way, the desktop application would be able to startup very quickly because it would not need to go through standing-up the in-memory data-set each time it starts.

[diver]

In hrev51696 (with no /boot/home/config/cache/HaikuDepot) packages are shown after 5 seconds. Icons are loaded after 12 seconds counting from the app launch.

[diver]

I see that icons are being downloaded in one go. Maybe icons for featured packages could be downloaded first (if Only featured packages is enabled) and only then the rest?

[stippi]

My intentions were that only the data needed for the list view is loaded for all packages at start up. Maybe I am missing something, but I don't see how this data could not be in memory. It needs to be loaded from somewhere at startup and I don't quite see a difference whether it is from a cache file or a database. The list view most likely needs all data that is displayed in the columns, simply because any column could provide the sorting criteria. Of course packages don't need to be displayed all at once, so the list view could fill up asynchronously after start (I think it already does). But there shouldn't really be a way to prioritize what package meta data to load first, given the random sorting setting. But even a few thousand packages should load quickly when only the data in the list view columns is loaded. Maybe it depends on how the cache file is organized (and maybe the mentioned data base is the best way). I know that my version of fetching package meta data was quite slow, but does this separation of package list view data from package "deep info" (needed only for the selected package) still exist?

[apl-haiku]

Hi Stephan; What you built is still happening except that the caching and bulk-loading is somewhat different. It is all working well. The question here is just the pause between starting and seeing all of the data merged into the in-memory model.

but does this separation of package list view data from package "deep info" (needed only for the selected package) still exist?

Yes, just the same.

so the list view could fill up asynchronously after start (I think it already does)

Yes it still does this.

Maybe I am missing something, but I don't see how this data could not be in memory.

My thinking was that if the data were housed in a local database and the application read what it needed on-demand then the cost of the initial data-merging could be avoided on subsequent starts because the persisted state of the database would be the result of the last merge between the local system and the remote data. However this is a _big_ job and I don't think it is worthwhile at the moment.

Maybe icons for featured packages could be downloaded first (if Only featured packages is enabled) and only then the rest?

The logic will just get too complex doing that sort of thing given the amount of time available. I will take a look again when I get a moment.

[diver]

It looks like recent changes broke Featured packages (on x86_64 only) See #13940.

[apl-haiku]

Diver; Why do you think that these changes are specifically related?

[diver]

I was just guessing. Since you worked on featured packages I though there might be a regression.

[cb88]

This is kind of related to this....

People reviewing the Beta as first time Haiku users open up HaikuDepot don't see anything for awhile (this bug) and when they do its only like 10-20 Featured packages.

So they assumingly think there is very little software available.

Rather than just speeding up Featured packages, how about disabling featured packages by default and show populating package statistics as HaikuDepot refreshes the info. in the empty panel below the package list eg, available package count, perhaps counting up as the data is loaded, latest updated packages list perhaps, then add filters for Featured/Curated packages and/or other relevant filters (I'd like Native/Java/QT/Console/Libraries etc...) so all packages are shown by default and users can filter them on their own terms.

The reason I say this is you're much less likely to loose people's attention if you show you are doing something and give them something to look at for a bit. The Find... dialog is a great example imagine if it just hung and displayed nothing until it had completed it's search yes it would still be fast but it would be boring to use.

[apl-haiku]

Yes it would be really good to show a spinner whilst it pulled down material from the server etc... even (like you say) just to show it is doing something. It is something I have meant to look into.

[cb88]

Replying to apl-haiku:

Yes it would be really good to show a spinner whilst it pulled down material from the server etc... even (like you say) just to show it is doing something. It is something I have meant to look into.

A spinner is a very MacOS way of doing things, I would suggest not doing that but actually show statistics and information about what HaikuDepot is doing.

[waddlesplash]

We already have the "barber pole progress bar"; we should just continue using that.

[cb88]

Replying to waddlesplash:

We already have the "barber pole progress bar"; we should just continue using that.

While aesthetically nice, it doesn't do anything to hold the users attention... same boat as a spinner. It's nice that it's there but it barely conveys any information.

You have to thing about things like this from a new users perspective. And even someone that is very familiar with HaikuDepot would much rather see actual information instead of just a blank screen a spinning barber pole.

[diver]

What do you think should be displayed instead?

[Janus]

I agreed with waddlespash for the barber pole.

A progress bar is useful if you have an approximation of the duration of the operation currently performed. When you don't have this information usually a barber pole is used.

A barber pole conveys two information: one the application is doing something, two the application is not frozen.

In HaikuDepot there is a message near the progress bar/barber pole, that message can be used to specify in detail what the application is doing.

[apl-haiku]

| We already have the "barber pole progress bar"; we should just continue using that.

Absolutely agree; that was my thinking as well. The barber pole is great.

[waddlesplash]

The first-start performance is now much improved, but after every package installation it does this refresh, which takes 3-5 seconds and makes HaikuDepot impossible to use while it's happening, which is really annoying.

[apl-haiku]

I have investigated the reload that is happening on each package install / uninstall. This is happening because of a BMessage sent from the Volume class in the Package Server. It seems to be triggering when there is a change in the packages and the change is sent from the message Volume::_SetLatestState. The code of the BMessage is B_PACKAGE_UPDATE.

Because when you install or uninstall a package, the B_PACKAGE_UPDATE is sent and consequently the reload of the package data is triggered.

This message ends up being received in the HaikuDepot desktop application and it triggers the large-refresh of it's data by invoking the MainWindow::_StartBulkLoad. I guess what happened is that this was fine in 2013 with a small number of packages but is slow now with a large number of packages.

I assume this is implemented like this so that when a change external to HaikuDepot happens (for example a command line package install) then the HaikuDepot application is able to react and reflect this change.

Here are some options;

• The code in the Volume class could try to somehow more accurately describe the change and somehow convey this in the BMessage so that HaikuDepot could manipulate it's state more precisely rather than re-load. My guess would be that this may require a good understanding of the code in LocalPkgDataLoadProcess which probably won't be too easy.
• New code in HaikuDepot (a new Process) could be implemented which diff'ed the state of the packages in the system with its internal model and make changes to the internal model as necessary. At a guess, this would again require an in-depth understanding of what is happening in LocalPkgDataLoadProcess which doesn't seem easy.
• HaikuDepot could stop watching the specific B_PACKAGE_UPDATE message and thereby not reflect the system's package changes. This is a compromise solution, but would be a quick and easy fix.

[diver]

Recent changes made the featured packages appear after 12-16 seconds.

[apl-haiku]

Recent changes made the featured packages appear after 12-16 seconds.

Can you please be more specific -- it is very time consuming to guess. Is this what you mean?

• You view the featured packages
• You select a package
• You choose the "Install" button to install the package
• The installation progresses to completion
• You need to wait 12-16 seconds for the system to refresh

Do you mean that 12-16 seconds is now longer than it was before? What effect did the "recent changes" have compared to how it was before?

[apl-haiku]

This option is taken in ​this PR...

HaikuDepot could stop watching the specific B_PACKAGE_UPDATE message and thereby not reflect the system's package changes. This is a compromise solution, but would be a quick and easy fix.

...and another ticket #15879 is added to re-factor this area so that it works properly in a more performant manner later.

[diver]

I start HaikuDepot and the list of featured packages appears after 12-16 seconds. I don't perform any package installations.

[diver]

Now it takes 15 seconds to show icons in the featured tab. It badly needs in profiling.

[apl-haiku]

From cold with no cached data, having to download everything from the server, start time is approximately 15-20 seconds. With cached data and including doing all the checks with the server for fresh data it takes approximately 3-4 seconds. With no networking (--nonetworking) it takes around 1-2 seconds.

If I run Haiku in a virtual machine on the same host as a locally running server (ie; software network) then it starts from cold with no cached data in approximately 3-4 seconds. In the same setting with cached data is approximately 1-2 seconds.

Any delays should have very little or nothing to do with icon loading now because for some time now, the application loads the icons on demand rather than at start.

The following are the compressed transmission sizes of the downloads;

• package data for HaikuPorts (764KB)
• data for icons (3.35MB)
• reference data (4.5KB)

Those are not big numbers.

There may be some sporadic delays in the server packaging this stuff up on demand and maybe I'll look into that at some point and maybe it would be worth looking at a CDN at some point, but probably this is overkill for now and will just make work for the admin team.

Possibly it would be worth looking at the server deployment setup, but otherwise I don't think there is a HaikuDepot application-level problem here.

BTW: I am actually slowly working on the B_PACKAGE_UPDATE issue discussed above, but this has little or nothing to do with the start time.

[pulkomandy]

How do we end up with several megabytes of icons? Are these in hvif format, or are they png files?

[apl-haiku]

curl -L -O "https://depot.haiku-os.org/__pkgicon/all.tar.gz"

[diver]

~> HaikuDepot 
{I} last version [0.0.4] and current version [0.0.4] match -> cache retained
{I} did clear the featured packages view
{I} adding and starting a process coordinator [BulkLoad]
{I} [Node<LocalRepositoryUpdateProcess>] initiating threaded
{I} [Node<LocalRepositoryUpdateProcess>] starting process in thread
{I} [LocalRepositoryUpdateProcess] will update local repositories' caches
{I} [Node<ServerReferenceDataUpdateProcess>] initiating threaded
{I} [Node<ServerReferenceDataUpdateProcess>] starting process in thread
{I} [ServerReferenceDataUpdateProcess] will fetch data
{I} [ServerReferenceDataUpdateProcess] will stream 'https://depot.haiku-os.org/__reference/all-ru.json.gz' to [/boot/system/cache/tmp/fileEU0B7m]
{I} [LocalRepositoryUpdateProcess] did update 4 local repositories' caches
{I} [Node<LocalPkgDataLoadProcess>] initiating threaded
{I} [Node<LocalPkgDataLoadProcess>] starting process in thread
{I} [Node<LocalRepositoryUpdateProcess>] finished process in thread 0.778828 seconds
{I} [ServerReferenceDataUpdateProcess] remote data has not changed since [Tue, 16 Mar 2021 10:39:56 GMT]
{I} [ServerReferenceDataUpdateProcess] did fetch data
{I} [ServerReferenceDataUpdateProcess] will process data
{I} [ServerReferenceDataUpdateProcess] will populate 181 natural languages
{I} [ServerReferenceDataUpdateProcess] did add 181 supported languages
{I} [ServerReferenceDataUpdateProcess] will populate 11 pkg categories
{I} [ServerReferenceDataUpdateProcess] will populate 5 rating stabilities
{I} [ServerReferenceDataUpdateProcess] did process data
{I} [Node<ServerReferenceDataUpdateProcess>] finished process in thread 0.702304 seconds
{I} [Node<ServerIconExportUpdateProcess>] initiating threaded
{I} [Node<ServerIconExportUpdateProcess>] starting process in thread
{I} [ServerIconExportUpdateProcess] will fetch data
{I} [ServerIconExportUpdateProcess] will stream 'https://depot.haiku-os.org/__pkgicon/all.tar.gz' to [/boot/system/cache/tmp/fileKbAvBs]
{I} [Node<ServerRepositoryDataUpdateProcess>] initiating threaded
{I} [Node<ServerRepositoryDataUpdateProcess>] starting process in thread
{I} [ServerRepositoryDataUpdateProcess] will fetch data
{I} [ServerRepositoryDataUpdateProcess] will stream 'https://depot.haiku-os.org/__repository/all-ru.json.gz' to [/boot/system/cache/tmp/fileB2BysG]
{I} [ServerRepositoryDataUpdateProcess] remote data has not changed since [Tue, 22 Jun 2021 20:18:00 GMT]
{I} [ServerRepositoryDataUpdateProcess] did fetch data
{I} [ServerRepositoryDataUpdateProcess] will process data
{I} [DepotMatchingRepositoryListener] associated depot [HaikuPorts] with server repository source [haikuports_x86_64]
{I} [DepotMatchingRepositoryListener] associated depot [HaikuPorts] with server repository source [haikuports_x86_64]
{I} [ServerRepositoryDataUpdateProcess] did process data
{I} [Node<ServerPkgDataUpdateProcess<HAKILO>>] initiating threaded
{I} [Node<ServerPkgDataUpdateProcess<HAKILO>>] starting process in thread
{I} [ServerPkgDataUpdateProcess<HAKILO>] am not updating data for depot [HAKILO] as there is no web app repository source code available
{I} [InfoJsonExtractEntryListener] did extract [hicn/info.json]
{I} did list 3211 tar items
{I} [Node<ServerPkgDataUpdateProcess<Haiku>>] initiating threaded
{I} [Node<ServerPkgDataUpdateProcess<Haiku>>] starting process in thread
{I} [ServerPkgDataUpdateProcess<Haiku>] am not updating data for depot [Haiku] as there is no web app repository source code available
{I} [ServerIconExportUpdateProcess] remote data has not changed since [Mon, 28 Jun 2021 20:17:22 GMT]
{I} [ServerIconExportUpdateProcess] did fetch data
{I} [ServerIconExportUpdateProcess] will process data
{I} will init icon model from tar [/boot/home/config/cache/HaikuDepot/pkgicon-all.tar]
{I} will read [/boot/home/config/cache/HaikuDepot/pkgicon-all.tar] and collect the tar pointers
{I} did collect 1935 tar pointers (0.0246 secs)
{I} [ServerIconExportUpdateProcess] did process data
{I} [Node<ServerPkgDataUpdateProcess<HaikuPorts>>] initiating threaded
{I} [Node<ServerPkgDataUpdateProcess<HaikuPorts>>] starting process in thread
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] will fetch data
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] will stream 'https://depot.haiku-os.org/__pkg/all-haikuports_x86_64-ru.json.gz' to [/boot/system/cache/tmp/filee4Mv0L]
{I} [Node<ServerPkgDataUpdateProcess<LOTE>>] initiating threaded
{I} [Node<ServerPkgDataUpdateProcess<LOTE>>] starting process in thread
{I} [ServerPkgDataUpdateProcess<LOTE>] am not updating data for depot [LOTE] as there is no web app repository source code available
{I} [Node<LocalPkgDataLoadProcess>] finished process in thread 3.995180 seconds
{I} [Node<ServerPkgDataUpdateProcess<HAKILO>>] finished process in thread 1.001135 seconds
{I} [Node<ServerRepositoryDataUpdateProcess>] finished process in thread 1.252999 seconds
{I} [Node<ServerPkgDataUpdateProcess<Haiku>>] finished process in thread 0.752531 seconds
{I} [Node<ServerIconExportUpdateProcess>] finished process in thread 1.503544 seconds
{I} [Node<ServerPkgDataUpdateProcess<LOTE>>] finished process in thread 0.252130 seconds
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] will redirect to; https://depot.haiku-os.org/__secured/jobdata/5695031e-38be-4ce3-a007-bc37ef894d8d/download
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] will stream 'https://depot.haiku-os.org/__secured/jobdata/5695031e-38be-4ce3-a007-bc37ef894d8d/download' to [/boot/system/cache/tmp/filee4Mv0L]
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] did complete streaming data [783970 bytes]
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] did fetch data
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] will process data
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] did process 5576 packages' data in  (   4.7 secs)
{I} [ServerPkgDataUpdateProcess<HaikuPorts>] did process data
{I} process coordinator [BulkLoad] did complete
{I} [Node<ServerPkgDataUpdateProcess<HaikuPorts>>] finished process in thread 12.188511 seconds
{I} will stop all process coordinators
{I} will wait until the process coordinator has stopped
{I} did stop all process coordinators

Anything useful here? It would be nice to print at the end overall amount of time startup + fetch + populating icons it took.

[apl-haiku]

diver; can you please get some timings on the time to download those URLs in the logs down to your computer from cold without proxies / caches?