Opened 5 years ago

Last modified 8 months ago

#11674 assigned enhancement

[HaikuDepot] cache featured packages

Reported by: diver Owned by: apl-haiku
Priority: normal Milestone: R1
Component: Applications/HaikuDepot Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

HaikuDepot populates featured packages from depot.haiku-os.org on every launch which takes 2-3 seconds making it feel slow.

Change History (25)

comment:1 by stippi, 5 years ago

Actually, this information is already cached. But it could be cached better. At the moment, HaikuDepot is reading a cache file on disk for every known package. Inbetween featured packages, it is reading the cached information for a lot of non-featured packages which accounts for the time between two featured packages to appear in the list. Better would be to cache featured packages separately and use that information even before reading the individual package cache files.

comment:2 by diver, 3 years ago

Owner: changed from stippi to apl-haiku
Status: newassigned

Could it be fixed in hrev50905?

comment:3 by apl-haiku, 3 years ago

No, it is not fixed in hrev50905, but I do have a plan for that. Unfortunately it is going to take me a while to get to it though.

comment:4 by diver, 22 months ago

With hrev51514 it now works much better. It downloads 25 MB of data in ~/config/cache/HaikuDepot (it stays in Deskbar after closing HD until it fninished downloading everything) and on the next HD run icon loading is quite fast. Is it possible to speed up featured packages icon loading on the first run?

comment:5 by pulkomandy, 22 months ago

Would it be possible to pre-load this when compiling Haiku, so that there is some data already available on first start?

comment:6 by apl-haiku, 22 months ago

@diver; The next improvement is to download all of the packages' meta-data in one HTTP request and to stream-load them into HD. This should be faster. The server-side for this change is complete and now I will slowly work on the client-side soon. I want the schema-to-class generation to happen in the Jamfile as suggested by Ingo, but it may take me a while to figure that out.

@PulkoMandy; I am sure there is a mechanism to do that, but the data would then be stale and so I think it would be better if it continued to be downloaded.

comment:7 by diver, 22 months ago

I just tried hrev51541 and it got slower now. Before there was a 3 sec delay before icons were shown, now it's 8 seconds.

comment:8 by apl-haiku, 22 months ago

@diver; Loading from a "cold start" (no cached data) should however be much faster now. I have more changes to come to help with starting with cached data.

comment:9 by apl-haiku, 20 months ago

I was hoping that this recent set of changes in commit 3094fef308a6aaaef9827863a0a99640bdaaa5af would help quite a bit and it might have done, but perhaps a bit less than I had hoped.

Currently the desktop application has an entirely in-memory model. This means that it loads all of the packages' from the local operating system into memory and then gets the data from the server and merges that server data in. It does this for all packages in all repositories. Now that the community has a large number of packages, this is unavoidably going to take some time on each start.

A better approach would be for the desktop application to synchronize the local operating system's packages and the server data into a persistent local sqlite3 database. This would replace the in-memory model and the desktop application would query the database when it needs data instead of holding everything in-memory.

In this way, the desktop application would be able to startup very quickly because it would not need to go through standing-up the in-memory data-set each time it starts.

comment:10 by diver, 20 months ago

In hrev51696 (with no /boot/home/config/cache/HaikuDepot) packages are shown after 5 seconds. Icons are loaded after 12 seconds counting from the app launch.

comment:11 by diver, 20 months ago

I see that icons are being downloaded in one go. Maybe icons for featured packages could be downloaded first (if Only featured packages is enabled) and only then the rest?

comment:12 by stippi, 20 months ago

My intentions were that only the data needed for the list view is loaded for all packages at start up. Maybe I am missing something, but I don't see how this data could not be in memory. It needs to be loaded from somewhere at startup and I don't quite see a difference whether it is from a cache file or a database. The list view most likely needs all data that is displayed in the columns, simply because any column could provide the sorting criteria. Of course packages don't need to be displayed all at once, so the list view could fill up asynchronously after start (I think it already does). But there shouldn't really be a way to prioritize what package meta data to load first, given the random sorting setting. But even a few thousand packages should load quickly when only the data in the list view columns is loaded. Maybe it depends on how the cache file is organized (and maybe the mentioned data base is the best way). I know that my version of fetching package meta data was quite slow, but does this separation of package list view data from package "deep info" (needed only for the selected package) still exist?

comment:13 by apl-haiku, 20 months ago

Hi Stephan; What you built is still happening except that the caching and bulk-loading is somewhat different. It is all working well. The question here is just the pause between starting and seeing all of the data merged into the in-memory model.

but does this separation of package list view data from package "deep info" (needed only for the selected package) still exist?

Yes, just the same.

so the list view could fill up asynchronously after start (I think it already does)

Yes it still does this.

Maybe I am missing something, but I don't see how this data could not be in memory.

My thinking was that if the data were housed in a local database and the application read what it needed on-demand then the cost of the initial data-merging could be avoided on subsequent starts because the persisted state of the database would be the result of the last merge between the local system and the remote data. However this is a _big_ job and I don't think it is worthwhile at the moment.

Maybe icons for featured packages could be downloaded first (if Only featured packages is enabled) and only then the rest?

The logic will just get too complex doing that sort of thing given the amount of time available. I will take a look again when I get a moment.

comment:14 by diver, 18 months ago

It looks like recent changes broke Featured packages (on x86_64 only) See #13940.

comment:15 by apl-haiku, 18 months ago

Diver; Why do you think that these changes are specifically related?

comment:16 by diver, 18 months ago

I was just guessing. Since you worked on featured packages I though there might be a regression.

comment:17 by cb88, 11 months ago

This is kind of related to this....

People reviewing the Beta as first time Haiku users open up HaikuDepot don't see anything for awhile (this bug) and when they do its only like 10-20 Featured packages.

So they assumingly think there is very little software available. As there is nothing blatantly indicating otherwise.

Rather than just speeding up Featured packages, how about unchecking featured packages by default and show populating package statistics as HaikuDepot refreshes the info. in the empty panel below the package list eg, available package count, perhaps counting up as the data is loaded, latest updated packages list perhaps, then add filters for Featured/Curated packages and/or other relevant filters (I'd like Native/Java/QT/Console/Libraries etc...) so all packages are shown by default and users can filter them on their own terms.

The reason I say this is you're much less likely to loose people's attention if you show you are doing something and give them something to look at for a bit. The Find... dialog is a great example imagine if it just hung and displayed nothing until it had completed it's search yes it would still be fast but it would be boring to use.

This would solve needing to speed up refreshing the package list by slight of hand.

Last edited 11 months ago by cb88 (previous) (diff)

comment:18 by apl-haiku, 11 months ago

Yes it would be really good to show a spinner whilst it pulled down material from the server etc... even (like you say) just to show it is doing something. It is something I have meant to look into.

in reply to:  18 comment:19 by cb88, 11 months ago

Replying to apl-haiku:

Yes it would be really good to show a spinner whilst it pulled down material from the server etc... even (like you say) just to show it is doing something. It is something I have meant to look into.

A spinner is a very MacOS way of doing things, I would suggest not doing that but actually show statistics and information about what HaikuDepot is doing.

comment:20 by waddlesplash, 11 months ago

We already have the "barber pole progress bar"; we should just continue using that.

in reply to:  20 comment:21 by cb88, 11 months ago

Replying to waddlesplash:

We already have the "barber pole progress bar"; we should just continue using that.

While aesthetically nice, it doesn't do anything to hold the users attention... same boat as a spinner. It's nice that it's there but it barely conveys any information.

You have to thing about things like this from a new users perspective. And even someone that is very familiar with HaikuDepot would much rather see actual information instead of just a blank screen a spinning barber pole.

Last edited 11 months ago by cb88 (previous) (diff)

comment:22 by diver, 11 months ago

What do you think should be displayed instead?

comment:23 by Janus, 11 months ago

I agreed with waddlespash for the barber pole.

A progress bar is useful if you have an approximation of the duration of the operation currently performed. When you don't have this information usually a barber pole is used.

A barber pole conveys two information: one the application is doing something, two the application is not frozen.

In HaikuDepot there is a message near the progress bar/barber pole, that message can be used to specify in detail what the application is doing.

comment:24 by apl-haiku, 11 months ago

| We already have the "barber pole progress bar"; we should just continue using that.

Absolutely agree; that was my thinking as well. The barber pole is great.

comment:25 by waddlesplash, 8 months ago

The first-start performance is now much improved, but after every package installation it does this refresh, which takes 3-5 seconds and makes HaikuDepot impossible to use while it's happening, which is really annoying.

Note: See TracTickets for help on using tickets.