Opened 11 months ago

Last modified 2 months ago

#14927 reopened bug

HaikuDepot doesn't show any packages

Reported by: KapiX Owned by: apl-haiku
Priority: normal Milestone: Unscheduled
Component: Applications/HaikuDepot Version: R1/Development
Keywords: Cc:
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

Since some time HaikuDepot doesn't show any packages on my machine. What's interesting is that while loading they are visible but then disappear, like some kind of filter (search perhaps) was applied. Searching for something else doesn't help. There were three Qt packages visible (installed from local hpkg IIRC), but since I uninstalled them the list is empty. I tried removing local settings but it didn't help. I have selected to show packages from all repositories.

I can provide logs but I need to know which ones.

Attachments (1)

log.txt (14.6 KB ) - added by KapiX 11 months ago.

Download all attachments as: .zip

Change History (24)

comment:1 by waddlesplash, 11 months ago

Owner: changed from stippi to apl-haiku
Status: newassigned

comment:2 by apl-haiku, 11 months ago

That is strange. I have just played around with it locally and it seems to be working fine. Can you please first of all provide a console log and attach it here. I can perhaps see if there are any clues from that.

Secondly, try to moved the ~/config/cache/HaikuDepot somewhere else (eg; add an underscore to the directory) and re-load HaikuDepot. Does this have any effect?

Third, with the Qt packages, do I understand correctly...

  1. Take the three Qt packages and drop them into /system/packages
  2. Load HD and observe the packages loading correctly in the UI
  3. Quit HD
  4. Remove the three Qt packages from /system/packages
  5. Load HD and observe the problem

Are you able to easily try this again to see if this is the problem at your end or give me the three "hpkg" files?

comment:3 by waddlesplash, 11 months ago

I just realized that HDS was down at the time this ticket was created, but service was restored about 7 hours ago now. So perhaps that's the difference.

comment:4 by apl-haiku, 11 months ago

Hello @waddlesplash; Thanks for the pointer. I did not see any outage information in the mailing list; is there an alert for HDS being offline and do you know what was the cause for the outage?

I guess if that were happening, the desktop application should have been able to run off cached data just fine. I've tested that.

@KapiX are you able to confirm that you are still seeing the problem now that HDS is operational.

comment:5 by KapiX, 11 months ago

HDS being operational has nothing to do with it because I'm seeing the issue for a few months now. Anyway it still doesn't work (the package list view).

Moving the cache doesn't fix the issue. I can only see packages for a bit longer. As soon as loading done, packages in the list are gone.

Load HD and observe the packages loading correctly in the UI

No, only those three packages, not all of them. It looks like at some point in time some kind of permanent filter was applied. I don't have the three packages so I can't provide them.

by KapiX, 11 months ago

Attachment: log.txt added

comment:6 by apl-haiku, 11 months ago

I wonder if there may be a mismatch on your system with your repositories' identifiers. Can you please do a quick check on this. If you run pkgman list you should see a value for;

  • base-url
  • url

Can you please check that the "url" matches the "url" value from the repository's "repo.info" file. For example from...

https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo.info

...or see HDS is also able to show you those "url" values as well -- see the field with the label "URL (repo.info)".

comment:7 by KapiX, 11 months ago

Resolution: no change required
Status: assignedclosed

Yes, url for both repos was empty. Readding them fixed it. Thanks.

comment:8 by diver, 11 months ago

Maybe haikuDepot should at least warn that there are no repositories currently configured in this case?

comment:9 by stippi, 2 months ago

Resolution: no change required
Status: closedreopened

Why was this bug closed? I've tracked this down to this commit:

https://github.com/haiku/haiku/commit/3b17d8dd7f1f25ad91f64ea12770200043d56af1#diff-f48033892fb37ef28ba8e79bc997c1d1

In particular the changes in src/kits/package/RepositoryConfig.cpp to BRepositoryConfig::SetTo(BEntry). From what I understand, the changes are meant to deal with old repository driver settings that still have the old keys. For an "old" repository driver settings file such as this:

url=https://eu.hpkg.haiku-os.org/haikuports/master/x86_gcc2/current
priority=1

the result is that fBaseURL = "<the url from above>" and fURL = "".

In HaikuDepot, where packages are populated and associated with a Depot, no installed package (except for local packages) is associated with any Depot, since no Depot has a URL. This happens in LocalPkgDataLoadProcess.cpp in RunInternal() where it uses repoConfig.URL() to associate the Depot's URL.

It seems one of the two places is incorrect. Either it is unintentional that BRepositoryConfig::URL() can be empty for old driver settings, or HaikuDepot needs to use BaseURL() instead of URL().

The code is not correct as is, if the intention is to support a seamless upgrade, as should be the case.

Last edited 2 months ago by stippi (previous) (diff)

comment:10 by apl-haiku, 2 months ago

Hello Stephan;

This was handled a while ago so hopefully my memory serves me correctly...

Your assessment is correct that "url" is an identifier for the repository and "base url" is the URL to the resource either on network or disk. The "base url" could point to _any_ mirror, but the "url" (identifier) is expected to remain constant across mirrors for the same logical repository. I agree with you that it would be more logical and less confusing to rename "url" to "identifier" and use a UUID. If I remember correctly this was discussed but rejected.

'Correct behaviour' for an upgrade here is a bit complicated.

The "base url" (in version 1 this was stored in the "url") was modified and/or people started to use mirrors. So just taking the old "base url" and using it as the "url" (identifier) doesn't always work unfortunately. For example, the correct current settings for HaikuPorts x86_64 are currently;

url (repo.info) = https://hpkg.haiku-os.org/haikuports/master/x86_64/current
Base URL = https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current

You will note that these no longer correlate as they once did.

Your solution in the pull-request will work if the "base url" they did have matches the actual "url" stored in the repo.info file and in HDS; otherwise there will be a mismatch between HD and HDS identifiers and the loading of data will also not work. I opted to intentionally fail in this case because otherwise this will lead to further confusion.

A better approach would be to alert the user via a BAlert as to the cause of the problem and/or to correct the repository configuration file.

comment:11 by stippi, 2 months ago

OK, I understand. I have a few questions to help me pick the best solution:

1 What website could the alert point users to in order for them to extract the information they need and fix things themselves?

2 Is it likely that the URL currently used as identifier at least for HaikuPorts and Haiku will ever change again? If not, wouldn't it be OK to apply some fuzzy logic to automatically fix this situation without the user becoming aware of it?

3 In my old install, the base-URL is incorrect for use as the identifier URL. Yet the change fixes display of packages in the "all packages list" for me. The featured packages are still empty. Maybe I have to investigate this deeper, but from what you write above, shouldn't it refuse to work still?

4 If the old URL, now base-URL still points to a valid repository, can't the indentifier-URL be extracted from there over the network?

Last edited 2 months ago by stippi (previous) (diff)

comment:12 by pulkomandy, 2 months ago

I agree with you that it would be more logical and less confusing to rename "url" to "identifier" and use a UUID. If I remember correctly this was discussed but rejected.

What was rejected is breaking the repository format, and preventing seamless upgrades. If we had replaced the URL with an UUID, we would have prevented users from upgrading from the old to the new format, because the ID of the repos would have changed. So we are stuck with the url as an identifier.

This has, however, no impact on how we name things, at least in the sourcecode. So, yes, think of 'url' as an unique identifier for the repo, which happens to be in URL form, but that could change for new repos (for existing ones, we can't change it).

4 If the old URL, now base-URL still points to a valid repository, can't the indentifier-URL be extracted from there over the network?

Do this carefully. You can get it from the package repo file on the server, but you have to make sure you are reaching the actual repo and not some man-in-the-middle trying to add malware to your install. The more secure way would be to hardcode a list of well-known repos to migrate? But, maybe it's ok if we just trust the server for the case where no URL is currently set. We don't have signing of the repos or packages currently, so maybe it isn't worth too much effort securing this migration (and once we have migrated, this problem doesn't remain)

Still, base-url and url are not, in general, interchangeable. base-url is where to find the data on the web, and url is a badly named identifier for the data.

comment:13 by apl-haiku, 2 months ago

1 What website could the alert point users

I am not aware of there being such a web page at this point in time. In retrospect this would have been a good idea -- or a blog post or something. I think I did post in a mailing list, but I don't recall.

2 Is it likely that the URL currently used as identifier at least for HaikuPorts and Haiku will ever change again?

I hope not, but I am not too sure; Alex would be best placed to answer this question. I hope not!

... fuzzy logic to automatically fix this situation without the user becoming aware of it?

I guess as Adrien mentioned it would be possible to hard-code former repository "url" values in there and have some sort of mapping old-to-new. Probably safer would be an instructional web page.

... The featured packages are still empty ...

I did not appreciate that this issue is causing the "all packages list" to fail as well -- HD should definitely handle this case so that the "all packages list" will work even if this problem happens. The "all packages list" should be displaying even if there is no "url" (identifier) because the "url" should only be required to marry-up the data on-host with the data from HDS. Probably some handling needs to be corrected that can cope with a missing "url" for "all packages list".

Because the icon data is common across all repositories, the icons will be loading fine even if the "url" is corrupted.

It is not surprising that the "featured packages" is empty because HD will be unable to correlate data from HDS and thus will be unable to find out which packages are currently "featured". In this sort of situation where there are no "featured" packages, maybe it is best to disable this mode?

... What was rejected is breaking the repository format, and preventing seamless upgrades. ...

If an "identifier" with a UUID were added into the "repo.info" file in the repository then the HPKR update operation could be amended to update the "identifier" when it refreshes if it is missing locally. After 6-12 months it would probably be possible to make this change without impacting too many people. It would be some work to achieve this.

... This has, however, no impact on how we name things ...

Previous to this change, it was very confusing having multiple uses for the term "url" and also different terms for this conceptual "identifier". At the moment the term "url" is consistent in the HDS database, the HDS source, the HDS API signatures, the HD source, the package library and through the repo.info file. Somebody can trace it from start to finish by name and that is of significant value for maintenance. It was quite a bit of work to achieve that so I would be averse to renaming anything without ensuring follow-through to ensure the work were done consistently across both data and software.

... and not some man-in-the-middle trying to add malware to your install

That is a good point.

Still, base-url and url are not, in general, interchangeable. base-url is where ...

Yes that is correct.

in reply to:  13 comment:14 by pulkomandy, 2 months ago

2 Is it likely that the URL currently used as identifier at least for HaikuPorts and Haiku will ever change again?

I hope not, but I am not too sure; Alex would be best placed to answer this question. I hope not!

That's the point of an identifier, and that is what 'url' is, despite the confusing name.

... What was rejected is breaking the repository format, and preventing seamless upgrades. ...

If an "identifier" with a UUID were added into the "repo.info" file in the repository then the HPKR update operation could be amended to update the "identifier" when it refreshes if it is missing locally. After 6-12 months it would probably be possible to make this change without impacting too many people. It would be some work to achieve this.

What would an UUID rather than an URI bring us here? I find the URI more generic. In fact, if someone were to create a new repo they could very well use "uuid:7673868d-231e-490d-9c4f-19288e7e668d" as the URL (it's technically an URI, not URL, but we already agree the current naming is confusing). Using an https URL here is a bit more descriptive (we also use it as an user-visible identifier for the repo, so an HTTP URL is more readable than an UUID).

The idea is to identify a resource here (namely, the repository), and that's what URI are for. Unlike URLs, they only identify, but don't always allow to locate the resource. A typical example is, you can use isbn: URIs to identify books by their barcode. This does not tell you where the book is available, but it allows you to uniquely identify the book.

... This has, however, no impact on how we name things ...

Previous to this change, it was very confusing having multiple uses for the term "url" and also different terms for this conceptual "identifier". At the moment the term "url" is consistent in the HDS database, the HDS source, the HDS API signatures, the HD source, the package library and through the repo.info file. Somebody can trace it from start to finish by name and that is of significant value for maintenance. It was quite a bit of work to achieve that so I would be averse to renaming anything without ensuring follow-through to ensure the work were done consistently across both data and software.

Yes, of course, but the name is still a bit confusing. We could rename "url" to "identifier" or "uri" or something like that in all the code and documentation, and add a note that in the HPKR file format, it is stored as "url" for legacy reasons. Or we can introduce a new version of HPKR files that uses the new name instead.

However, what I would prefer to avoid is having both "url" and "identifier" or "uri" live side by side in the HPKR file, as this would further confuse things.

So:

  • +1 to renaming "url" in the code, I don't know if it's worth introducing a new version of HPKR files or if we can live with keeping "url" just there
  • +1 to using an "uuid:" or other non-http scheme for the identifier to reduce risk of confusion, but only for new repos. For the existing ones, keeping the existing identifiers is probably less problematic (unless we do some special-casing to handle the migration? is it worth the effort?)
  • -1 to hardcoding the identifier to be an UUID, as I don't like that they are not helpful to the end-user, and they are a less flexible solution than using an uri (which can then effectively be used with an uuid: URI, but gvies us more flexibility if we want to use something else)

comment:15 by apl-haiku, 2 months ago

What would an UUID rather than an URI bring us here? I find the URI more generic.

Yes instead of UUID, some URI approach like hpkr:haikuports_x86_64 would be a good solution. In fact there is already a system of such codes in HDS [1] and if those were used it would also help comprehension across the wider data and software.

+1 to renaming "url" in the code,

It would be OK for "url" to remain as-is in the HPKR format as you suggest, but I would only want to see this change happen if there is capacity for it to be coordinated through everything including HDS. That will take time because it means coordinating API change.

---

IMHO, as nice as it would be, given the effort involved, it is probably best sorted out later.

[1] https://depot.haiku-os.org/#!/repository/haikuports

comment:16 by waddlesplash, 2 months ago

Why not use something like the following scheme:

  • a url field which is the true "origin" of the repository (likely eu.hpkg.haiku-os.org/... in our case
  • a mirror-url field which is the URL that packages will be downloaded from

Then we download the repo.sha256 from url, and everything else from the mirror-url. In the case that the url field changes, then users should probably have to do something manually anyway, indeed. But I think all other concerns this solves?

comment:17 by pulkomandy, 2 months ago

Haiku IP addresses are blocked in Russia. If we need to download anything from the original repo, some people will not be able to use any mirror.

The "url" is not an url, it's just an identifier. It should never be used as an url. Its naming is legacy from when we didn't plan to have mirrors, and assumed the url could be used both to locate and identify the repo, which is not the case.

For trusting the repo, sha256 is useless as long as there is no signing. And when there is signing, you can trust the content without trusting the transport, so using any random mirror is fine as long as the packages are signed with a known/trusted key.

comment:18 by waddlesplash, 2 months ago

For trusting the repo, sha256 is useless as long as there is no signing.

Uh... why? The sha256 authenticates the "repo" file as authentic, and the "repo" file itself contains sha256sums for the HPKG files. So as long as the checksum for the "repo" is retrieved from a trusted source securely, all subsequent downloads will also be verifiable. It's not as good as GPG signing, sure; but it's very close by many metrics.

comment:19 by waddlesplash, 2 months ago

We can also add a flag for "trust mirror for sha256 download also." Then this solves the problem of IP blocks.

in reply to:  18 comment:20 by pulkomandy, 2 months ago

Replying to waddlesplash:

For trusting the repo, sha256 is useless as long as there is no signing.

Uh... why? The sha256 authenticates the "repo" file as authentic, and the "repo" file itself contains sha256sums for the HPKG files. So as long as the checksum for the "repo" is retrieved from a trusted source securely, all subsequent downloads will also be verifiable. It's not as good as GPG signing, sure; but it's very close by many metrics.

It's close only if you trust the whole transport between you and Haiku. It's easier to do an MITM especially when you are providing the mirror (you would just need to alter the "url" field, basically). Let's not hack around security, let's sign our package repos (GPG or whatever) and then we can stop worrying about how we distribute the packages. This allows separation of concerns: on one side we worry about signing, and on the other we worry, independently, about distribution. This way we don't multiply problems with each other.

comment:21 by kallisti5, 2 months ago

For context here is the change that was reverted trying to fix the URL / BaseURL mess:

https://git.haiku-os.org/haiku/commit/src/kits/package?id=fda0bf77fbdf5876ce81a39da324145143d6ad18

There was some compatibility stuff in the code to prevent it from flat-out breaking everyone, but there were definitely some issues.

As for repo signing... Pulkomandy is correct..ish. The sha256 is valuable since the repo file "contains a sha256 for every package". But since anyone can generate a new repo file with "fake sha256 checksums for malicious packages the chain of trust is incomplete.

This is why the repo files built by concourse (the new CI i'm trying to beat into submission) are signed... http://test.hpkg.haiku-os.org/master/x86_64/current/repo.minisig

We could also sign all packages... but that's a lot of work.

in reply to:  21 comment:22 by pulkomandy, 2 months ago

Replying to kallisti5:

We could also sign all packages... but that's a lot of work.

And unneeded, that's the point of having the sha256 of the packages in the repo file and then signing the repo file. As long as sha256 is safe to use, we're fine.

Signing each package would add an extra feature: the ability to safely share packages offline (just copy an hpkg from some random person machine to yours, and you can check if it's really a package from Haiku even without access to the matching repo file). It could also be used to control the integrity of an existing haiku install, even if the packages in it are out of date and not in the current repofile anymore.

But all that is extra features for later, and only useful if the signature is stored inside the hpkg file itself (or as an xattr, maybe?)

comment:23 by axeld, 2 months ago

To get back to the original point, I would favour Stippi's suggestion to put in some simple heuristics to set the correct URL/Base URL -- that will resolve the problem, at least.

Signing and all that would be great, as would renaming to avoid confusion. It won't help us solve this bug, though, and that discussion should better be moved to the mailing lists, I suppose :-)

Note: See TracTickets for help on using tickets.