HaikuDepot: Refresh repositories blanks featured packages list

[vidrep]

hrev51740 x86_64

Launching HaikuDepot for the first time after a fresh install, I see two issues:

1. "All repositories" is not default. Instead "none" is checkmarked

2. Selecting "All repositories", then refreshing causes the "Featured Packages" list to go blank.

See attached screenshots

[diver]

Featured Packages is blank here as well but I don't have empty repository in the list. What do you have in pkgman list?

[vidrep]

Welcome to the Haiku shell.

~> pkgman list

Haiku

base-url: ​https://packages.haiku-os.org/haiku/master/x86_64/current priority: 1

HaikuPorts

base-url: ​https://eu.hpkg.haiku-os.org/haikuports/master/repository/x86_64/current priority: 1

~>

[vidrep]

Fresh install of hrev 51758 x86_64. HaikuDepot "Featured Packages" is blank on first launch. Refresh repositories does not change situation. Looked in syslog and found this:

USER: instantiate_object failed: No signature specified in archive, looking for class "ColorControlPrivate::ColorMenuItem". USER: instantiate_object failed: No signature specified in archive, looking for class "FontMenuItem".

USER: instantiate_object failed: No signature specified in archive, looking for class "ColorControlPrivate::ColorMenuItem".

USER: instantiate_object failed: No signature specified in archive, looking for class "FontMenuItem".

KERN: slab memory manager: created area 0xffffffffa5801000 (7957)

KERN: slab memory manager: created area 0xffffffffa6001000 (7958)

KERN: slab memory manager: created area 0xffffffffa6801000 (7959)

[diver]

"Featured Packages" view work in gcc2 just fine. Looks like x86_64 only issue.

[diver]

Same problem on a new install.

[apl-haiku]

(as discussed on the mailing list)

---

I am currently (slowly) trying to track down a couple of odd things with "HaikuDepot" on x86_64 build. One of them has me a bit confused. HD asks for repositories from BPackageRoster and gets back;

• Haiku
• HaikuPorts

However when it then uses the solver to get a full list of packages, it finds packages that belong to the depots with names including;

• HaikuPorts-BuildPackages

Does anybody know where does the "-BuildPackages" suffix would be coming from and why?

---

The test system is a fresh install of x86_64 off revision 51803.

        Haiku
                base-url:  https://packages.haiku-os.org/haiku/master/x86_64/current
                priority:  1
        HaikuPorts
                base-url:  https://eu.hpkg.haiku-os.org/haikuports/master/repository/x86_64/current
                priority:  1

There is no repository with name "HaikuPorts-BuildPackages" listed here, but when HaikuDepot works through the list of packages, it finds the following packages (as examples) are in a repository with the name "HaikuPorts-BuildPackages";

• keymapswitcher
• mesa_devel
• freetype

These should be associated with a repository called "HaikuPorts". The discrepancy in the name causes problems within HaikuDepot setting up its working data.

---

Hello; Thank you for those pointers. I can see the build-time repository name "HaikuPorts-BuildPackages" in the associated "repo.info" ​file, but I can't see that repository name on-host anywhere post-install. Short-term, I'll at least patch HaikuDepot to cope with this anomaly and log.

---

It is internally invoking (~MainWindow.cpp::963)...

result = manager.Solver()->FindPackages(...)

...this is then returning instances of BSolverPackage and the result of (~MainWindow.cpp:1005)...

package->Repository()->Name()

Yields "HaikuDepot-BuildPackages".

[vidrep]

There are two separate issues here. I did a series of installs trying to narrow down a range where these issues first became apparent. The selection of the missing "local" repository (see HaikuDepot1.png) predates hrev51675, and will require further investigation. The disappearance of the "featured packages" upon refresh (HaikuDepot2.png) was introduced some time between hrev51691 and 51700. I installed hrev51691 and hrev51700 3 times each to make sure the problem was reproducible.

[vidrep]

Did a fresh install of hrev51638 x86_64. Upon opening Preferences->Repositories, I saw that one of the enabled entries is out of place in the list (see Repositories.png). Could it be that the entry was not removed from the code during cleanup, and is the cause of the current issue?

[apl-haiku]

The repository in in "Repositories.png" looks wrong; I suspect that the entry for ".../build-packages" should not be there.

[diver]

I can't reproduce it anymore.

[vidrep]

Working on 64 bit for me as well. Probably OK to close this one.

[vidrep]

hrev51942 x86_gcc2h, fresh install

Launching HaikuDepot leaves the featured packages list empty, even after waiting several minutes. The list is not populated until the repositories are refreshed.

[vidrep]

I am not seeing this issue specifically on hrev52119 x86_64. However, what is happening still, is that the packages list is not being fully populated with all of the available packages. I took a screenshot of HaikuDepot first launch, after a fresh install. The second screenshot is after the repositories have been refreshed.

[apl-haiku]

Yes, I have seen this as well; the packages are not updated until a pkgman refresh is issued.

[waddlesplash]

If this doesn't happen very often, can we move it out of beta1?

[waddlesplash]

After some discussion with Vidrep on IRC and cursory investigation, it seems what's really going on here is that it's just showing the packages the repository cache knows about; which is for initial installs just the build-packages set. I'm not sure what we should do to solve that: try to grab the full current repository state at build time? Add a repository refresh as part of the "first install" script? Run a repository refresh the first time HaikuDepot is opened?

[humdinger]

Is refreshing repos a costly action? It seems pretty fast and being done in the background without impacting the GUI for me. HaikuDepot could refresh on every launch... hmm... it seems it does here already...
Are you talking about a different "repository refresh" than the one available from the "Tools" menu?

[apl-haiku]

I think an install-time script refresh would be the most optimal solution because it is, in a way, separate from HaikuDepot and more of a system level update process.

[waddlesplash]

The problem there is that users are not often connected to the internet when that script runs.

If it does so already, then why is this a problem? Is it somehow not updating the featured packages list after that sync?

[apl-haiku]

I think the refresh of the repository information is happening at;

• MainWindow::MainWindow
• MainWindow::_StartRefreshWorker
• MainWindow::_RefreshModelThreadWorker (threaded)

This seems to then execute these synchronously;

• MainWindow::_RefreshRepositories (refresh repositories)
• MainWindow::_RefreshPackageList (update from HDS)

I don't have time to look into it deeply at the moment sorry, but that seems like it ought to be working.

[vidrep]

I did a fresh install of both x86_gcc2h and x86_64 the other night after work. x86_gcc2h fully populated the featured packages list on first launch of HaikuDepot after booting. x86_64 showed only 9 packages in the list. After refreshing the repositories it fully populated the list with all featured packages. So, as it stands now, it appears to be only a x86_64 issue.

[vidrep]

The last two fresh installs of 64 bit do not show this issue any more. Fixed???

[apl-haiku]

I didn't do anything that would specifically fix this.

[diver]

The bug is still there in beta1.

On the first start (right after friesh install) only a few feature packaged are shown. Note 'did process 347 packages' in the first output. After refreshing repos it suddenly did process 4820 packages.

~> HaikuDepot 
bulk load - start
bulk load - initiating [ServerIconExportUpdateProcess]
bulk load - starting process [ServerIconExportUpdateProcess]
bulk load - initiating [RepositoryDataUpdateProcess]
will start fetching icons
[ServerIconExportUpdateProcess] will stream 'https://depot.haiku-os.org/__pkgicon/all.tar.gz' to [/boot/system/cache/tmp/filegyvS1r]
bulk load - starting process [RepositoryDataUpdateProcess]
[RepositoryDataUpdateProcess] will fetch data
[RepositoryDataUpdateProcess] will stream 'https://depot.haiku-os.org/__repository/all-en.json.gz' to [/boot/system/cache/tmp/filerxxbSA]
received 302 --> will not store download to file
[RepositoryDataUpdateProcess] will redirect to; http://depot.haiku-os.org/__secured/jobdata/03ee53ba-3194-4342-b77d-fff689d157c7/download
[RepositoryDataUpdateProcess] will stream 'http://depot.haiku-os.org/__secured/jobdata/03ee53ba-3194-4342-b77d-fff689d157c7/download' to [/boot/system/cache/tmp/filerxxbSA]
received 302 --> will not store download to file
[ServerIconExportUpdateProcess] will redirect to; http://depot.haiku-os.org/__secured/jobdata/9bd4de7c-d17e-4a33-a14c-2b40a5b9a2df/download
[ServerIconExportUpdateProcess] will stream 'http://depot.haiku-os.org/__secured/jobdata/9bd4de7c-d17e-4a33-a14c-2b40a5b9a2df/download' to [/boot/system/cache/tmp/filegyvS1r]
received 302 --> will not store download to file
[RepositoryDataUpdateProcess] will redirect to; https://depot.haiku-os.org:443/__secured/jobdata/03ee53ba-3194-4342-b77d-fff689d157c7/download
[RepositoryDataUpdateProcess] will stream 'https://depot.haiku-os.org:443/__secured/jobdata/03ee53ba-3194-4342-b77d-fff689d157c7/download' to [/boot/system/cache/tmp/filerxxbSA]
received 302 --> will not store download to file
[ServerIconExportUpdateProcess] will redirect to; https://depot.haiku-os.org:443/__secured/jobdata/9bd4de7c-d17e-4a33-a14c-2b40a5b9a2df/download
[ServerIconExportUpdateProcess] will stream 'https://depot.haiku-os.org:443/__secured/jobdata/9bd4de7c-d17e-4a33-a14c-2b40a5b9a2df/download' to [/boot/system/cache/tmp/filegyvS1r]
[RepositoryDataUpdateProcess] did complete streaming data [635 bytes]
[RepositoryDataUpdateProcess] did fetch data
[RepositoryDataUpdateProcess] will process data
associated depot [HaikuPorts] with server repository source [haikuports_x86_gcc2]
[RepositoryDataUpdateProcess] did process data
depot [Haiku] (https://eu.hpkg.haiku-os.org/haiku/r1beta1/x86_gcc2/current) correlates with no repository in the haikudepot server system
the depot [Haiku] has no repository source code
bulk load - initiating [PkgDataUpdateProcess<HaikuPorts>]
did initiate populate package data for 1 depots
bulk load - starting process [PkgDataUpdateProcess<HaikuPorts>]
[PkgDataUpdateProcess<HaikuPorts>] will fetch data
[PkgDataUpdateProcess<HaikuPorts>] will stream 'https://depot.haiku-os.org/__pkg/all-haikuports_x86_gcc2-ru.json.gz' to [/boot/system/cache/tmp/filevSkkcM]
received 302 --> will not store download to file
[PkgDataUpdateProcess<HaikuPorts>] will redirect to; http://depot.haiku-os.org/__secured/jobdata/2782735c-597d-4199-896f-fee899fe53aa/download
[PkgDataUpdateProcess<HaikuPorts>] will stream 'http://depot.haiku-os.org/__secured/jobdata/2782735c-597d-4199-896f-fee899fe53aa/download' to [/boot/system/cache/tmp/filevSkkcM]
received 302 --> will not store download to file
[PkgDataUpdateProcess<HaikuPorts>] will redirect to; https://depot.haiku-os.org:443/__secured/jobdata/2782735c-597d-4199-896f-fee899fe53aa/download
[PkgDataUpdateProcess<HaikuPorts>] will stream 'https://depot.haiku-os.org:443/__secured/jobdata/2782735c-597d-4199-896f-fee899fe53aa/download' to [/boot/system/cache/tmp/filevSkkcM]
[ServerIconExportUpdateProcess] did complete streaming data [3242588 bytes]
delete any existing stored data
will unpack to [/boot/home/config/cache/HaikuDepot/__allicons]
[PkgDataUpdateProcess<HaikuPorts>] did complete streaming data [484647 bytes]
[PkgDataUpdateProcess<HaikuPorts>] did fetch data
[PkgDataUpdateProcess<HaikuPorts>] will process data
[PkgDataUpdateProcess<HaikuPorts>] did process 347 packages' data in  (  3.06 secs)
[PkgDataUpdateProcess<HaikuPorts>] did process data
detected end of tar-ball
did unpack icon tgz in (  5.34 secs)
did complete fetching icons
did populate 29 packages' icons (0.00367 secs)
found deskbar entry: data/deskbar/menu/Applications/BePDF -> ../../../../apps/BePDF/BePDF

Tools->Refresh repositories

bulk load - start
bulk load - initiating [ServerIconExportUpdateProcess]
bulk load - starting process [ServerIconExportUpdateProcess]
bulk load - initiating [RepositoryDataUpdateProcess]
will start fetching icons
bulk load - starting process [RepositoryDataUpdateProcess]
[RepositoryDataUpdateProcess] will fetch data
[RepositoryDataUpdateProcess] will stream 'https://depot.haiku-os.org/__repository/all-en.json.gz' to [/boot/system/cache/tmp/fileRKOXWI]
[ServerIconExportUpdateProcess] will stream 'https://depot.haiku-os.org/__pkgicon/all.tar.gz' to [/boot/system/cache/tmp/filev0zjnp]
Fatal exception occurred while resolving system dependencies: No such file or directory, details: 
received 304 --> will not store download to file
[RepositoryDataUpdateProcess] remote data has not changed since [Mon, 10 Sep 2018 14:00:48 GMT]
[RepositoryDataUpdateProcess] did fetch data
[RepositoryDataUpdateProcess] will process data
associated depot [HaikuPorts] with server repository source [haikuports_x86_gcc2]
[RepositoryDataUpdateProcess] did process data
depot [Haiku] (https://eu.hpkg.haiku-os.org/haiku/r1beta1/x86_gcc2/current) correlates with no repository in the haikudepot server system
the depot [Haiku] has no repository source code
bulk load - initiating [PkgDataUpdateProcess<HaikuPorts>]
bulk load - starting process [PkgDataUpdateProcess<HaikuPorts>]
[PkgDataUpdateProcess<HaikuPorts>] will fetch data
did initiate populate package data for 1 depots
[PkgDataUpdateProcess<HaikuPorts>] will stream 'https://depot.haiku-os.org/__pkg/all-haikuports_x86_gcc2-ru.json.gz' to [/boot/system/cache/tmp/fileNJfHxn]
received 304 --> will not store download to file
[ServerIconExportUpdateProcess] remote data has not changed since [Wed, 03 Oct 2018 10:41:55 GMT]
did populate 325 packages' icons (0.0594 secs)
received 304 --> will not store download to file
[PkgDataUpdateProcess<HaikuPorts>] remote data has not changed since [Fri, 05 Oct 2018 09:41:40 GMT]
[PkgDataUpdateProcess<HaikuPorts>] did fetch data
[PkgDataUpdateProcess<HaikuPorts>] will process data
[PkgDataUpdateProcess<HaikuPorts>] did process 4820 packages' data in  (  4.39 secs)
[PkgDataUpdateProcess<HaikuPorts>] did process data

[waddlesplash]

Changes to progress indication in hrev52660 seem to have improved this greatly. However, there is still a performance issue in that every time a package is installed or uninstalled, the repository cache is rebuilt apparently from scratch, which takes 4-5 seconds and locks up HaikuDepot completely (100% CPU usage) while it happens. So that seems to need a change...

[waddlesplash]

Oh, there's already a ticket for that: #11674. So this is fixed, then.

[nielx]

Assign tickets with status=closed and resolution=fixed within the R1/beta2 development window to the R1/beta2 Milestone