HaikuDepot doesn't show any packages

[KapiX]

Since some time HaikuDepot doesn't show any packages on my machine. What's interesting is that while loading they are visible but then disappear, like some kind of filter (search perhaps) was applied. Searching for something else doesn't help. There were three Qt packages visible (installed from local hpkg IIRC), but since I uninstalled them the list is empty. I tried removing local settings but it didn't help. I have selected to show packages from all repositories.

I can provide logs but I need to know which ones.

[apl-haiku]

That is strange. I have just played around with it locally and it seems to be working fine. Can you please first of all provide a console log and attach it here. I can perhaps see if there are any clues from that.

Secondly, try to moved the ~/config/cache/HaikuDepot somewhere else (eg; add an underscore to the directory) and re-load HaikuDepot. Does this have any effect?

Third, with the Qt packages, do I understand correctly...

1. Take the three Qt packages and drop them into /system/packages
2. Load HD and observe the packages loading correctly in the UI
3. Quit HD
4. Remove the three Qt packages from /system/packages
5. Load HD and observe the problem

Are you able to easily try this again to see if this is the problem at your end or give me the three "hpkg" files?

[waddlesplash]

I just realized that HDS was down at the time this ticket was created, but service was restored about 7 hours ago now. So perhaps that's the difference.

[apl-haiku]

Hello @waddlesplash; Thanks for the pointer. I did not see any outage information in the mailing list; is there an alert for HDS being offline and do you know what was the cause for the outage?

I guess if that were happening, the desktop application should have been able to run off cached data just fine. I've tested that.

@KapiX are you able to confirm that you are still seeing the problem now that HDS is operational.

[KapiX]

HDS being operational has nothing to do with it because I'm seeing the issue for a few months now. Anyway it still doesn't work (the package list view).

Moving the cache doesn't fix the issue. I can only see packages for a bit longer. As soon as loading done, packages in the list are gone.

Load HD and observe the packages loading correctly in the UI

No, only those three packages, not all of them. It looks like at some point in time some kind of permanent filter was applied. I don't have the three packages so I can't provide them.

[apl-haiku]

I wonder if there may be a mismatch on your system with your repositories' identifiers. Can you please do a quick check on this. If you run pkgman list you should see a value for;

• base-url
• url

Can you please check that the "url" matches the "url" value from the repository's "repo.info" file. For example from...

​https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current/repo.info

...or see HDS is also able to show you those "url" values as well -- see the field with the label "URL (repo.info)".

[KapiX]

Yes, url for both repos was empty. Readding them fixed it. Thanks.

[diver]

Maybe haikuDepot should at least warn that there are no repositories currently configured in this case?

[stippi]

Why was this bug closed? I've tracked this down to this commit:

​https://github.com/haiku/haiku/commit/3b17d8dd7f1f25ad91f64ea12770200043d56af1#diff-f48033892fb37ef28ba8e79bc997c1d1

In particular the changes in src/kits/package/RepositoryConfig.cpp to BRepositoryConfig::SetTo(BEntry). From what I understand, the changes are meant to deal with old repository driver settings that still have the old keys. For an "old" repository driver settings file such as this:

url=https://eu.hpkg.haiku-os.org/haikuports/master/x86_gcc2/current
priority=1

the result is that fBaseURL = "<the url from above>" and fURL = "".

In HaikuDepot, where packages are populated and associated with a Depot, no installed package (except for local packages) is associated with any Depot, since no Depot has a URL. This happens in LocalPkgDataLoadProcess.cpp in RunInternal() where it uses repoConfig.URL() to associate the Depot's URL.

It seems one of the two places is incorrect. Either it is unintentional that BRepositoryConfig::URL() can be empty for old driver settings, or HaikuDepot needs to use BaseURL() instead of URL().

The code is not correct as is, if the intention is to support a seamless upgrade, as should be the case.

[apl-haiku]

Hello Stephan;

This was handled a while ago so hopefully my memory serves me correctly...

Your assessment is correct that "url" is an identifier for the repository and "base url" is the URL to the resource either on network or disk. The "base url" could point to _any_ mirror, but the "url" (identifier) is expected to remain constant across mirrors for the same logical repository. I agree with you that it would be more logical and less confusing to rename "url" to "identifier" and use a UUID. If I remember correctly this was discussed but rejected.

'Correct behaviour' for an upgrade here is a bit complicated.

The "base url" (in version 1 this was stored in the "url") was modified and/or people started to use mirrors. So just taking the old "base url" and using it as the "url" (identifier) doesn't always work unfortunately. For example, the correct current settings for HaikuPorts x86_64 are currently;

url (repo.info) = https://hpkg.haiku-os.org/haikuports/master/x86_64/current
Base URL = https://eu.hpkg.haiku-os.org:443/haikuports/master/x86_64/current

You will note that these no longer correlate as they once did.

Your solution in the pull-request will work if the "base url" they did have matches the actual "url" stored in the repo.info file and in HDS; otherwise there will be a mismatch between HD and HDS identifiers and the loading of data will also not work. I opted to intentionally fail in this case because otherwise this will lead to further confusion.

A better approach would be to alert the user via a BAlert as to the cause of the problem and/or to correct the repository configuration file.

[stippi]

OK, I understand. I have a few questions to help me pick the best solution:

1 What website could the alert point users to in order for them to extract the information they need and fix things themselves?

2 Is it likely that the URL currently used as identifier at least for HaikuPorts and Haiku will ever change again? If not, wouldn't it be OK to apply some fuzzy logic to automatically fix this situation without the user becoming aware of it?

3 In my old install, the base-URL is incorrect for use as the identifier URL. Yet the change fixes display of packages in the "all packages list" for me. The featured packages are still empty. Maybe I have to investigate this deeper, but from what you write above, shouldn't it refuse to work still?

4 If the old URL, now base-URL still points to a valid repository, can't the indentifier-URL be extracted from there over the network?

[pulkomandy]

I agree with you that it would be more logical and less confusing to rename "url" to "identifier" and use a UUID. If I remember correctly this was discussed but rejected.

What was rejected is breaking the repository format, and preventing seamless upgrades. If we had replaced the URL with an UUID, we would have prevented users from upgrading from the old to the new format, because the ID of the repos would have changed. So we are stuck with the url as an identifier.

This has, however, no impact on how we name things, at least in the sourcecode. So, yes, think of 'url' as an unique identifier for the repo, which happens to be in URL form, but that could change for new repos (for existing ones, we can't change it).

4 If the old URL, now base-URL still points to a valid repository, can't the indentifier-URL be extracted from there over the network?

Do this carefully. You can get it from the package repo file on the server, but you have to make sure you are reaching the actual repo and not some man-in-the-middle trying to add malware to your install. The more secure way would be to hardcode a list of well-known repos to migrate? But, maybe it's ok if we just trust the server for the case where no URL is currently set. We don't have signing of the repos or packages currently, so maybe it isn't worth too much effort securing this migration (and once we have migrated, this problem doesn't remain)

Still, base-url and url are not, in general, interchangeable. base-url is where to find the data on the web, and url is a badly named identifier for the data.

[apl-haiku]

1 What website could the alert point users

I am not aware of there being such a web page at this point in time. In retrospect this would have been a good idea -- or a blog post or something. I think I did post in a mailing list, but I don't recall.

2 Is it likely that the URL currently used as identifier at least for HaikuPorts and Haiku will ever change again?

I hope not, but I am not too sure; Alex would be best placed to answer this question. I hope not!

... fuzzy logic to automatically fix this situation without the user becoming aware of it?

I guess as Adrien mentioned it would be possible to hard-code former repository "url" values in there and have some sort of mapping old-to-new. Probably safer would be an instructional web page.

... The featured packages are still empty ...

I did not appreciate that this issue is causing the "all packages list" to fail as well -- HD should definitely handle this case so that the "all packages list" will work even if this problem happens. The "all packages list" should be displaying even if there is no "url" (identifier) because the "url" should only be required to marry-up the data on-host with the data from HDS. Probably some handling needs to be corrected that can cope with a missing "url" for "all packages list".

Because the icon data is common across all repositories, the icons will be loading fine even if the "url" is corrupted.

It is not surprising that the "featured packages" is empty because HD will be unable to correlate data from HDS and thus will be unable to find out which packages are currently "featured". In this sort of situation where there are no "featured" packages, maybe it is best to disable this mode?

... What was rejected is breaking the repository format, and preventing seamless upgrades. ...

If an "identifier" with a UUID were added into the "repo.info" file in the repository then the HPKR update operation could be amended to update the "identifier" when it refreshes if it is missing locally. After 6-12 months it would probably be possible to make this change without impacting too many people. It would be some work to achieve this.

... This has, however, no impact on how we name things ...

Previous to this change, it was very confusing having multiple uses for the term "url" and also different terms for this conceptual "identifier". At the moment the term "url" is consistent in the HDS database, the HDS source, the HDS API signatures, the HD source, the package library and through the repo.info file. Somebody can trace it from start to finish by name and that is of significant value for maintenance. It was quite a bit of work to achieve that so I would be averse to renaming anything without ensuring follow-through to ensure the work were done consistently across both data and software.

... and not some man-in-the-middle trying to add malware to your install

That is a good point.

Still, base-url and url are not, in general, interchangeable. base-url is where ...

Yes that is correct.

[pulkomandy]

2 Is it likely that the URL currently used as identifier at least for HaikuPorts and Haiku will ever change again?

I hope not, but I am not too sure; Alex would be best placed to answer this question. I hope not!

That's the point of an identifier, and that is what 'url' is, despite the confusing name.

... What was rejected is breaking the repository format, and preventing seamless upgrades. ...

If an "identifier" with a UUID were added into the "repo.info" file in the repository then the HPKR update operation could be amended to update the "identifier" when it refreshes if it is missing locally. After 6-12 months it would probably be possible to make this change without impacting too many people. It would be some work to achieve this.

What would an UUID rather than an URI bring us here? I find the URI more generic. In fact, if someone were to create a new repo they could very well use "uuid:7673868d-231e-490d-9c4f-19288e7e668d" as the URL (it's technically an URI, not URL, but we already agree the current naming is confusing). Using an https URL here is a bit more descriptive (we also use it as an user-visible identifier for the repo, so an HTTP URL is more readable than an UUID).

The idea is to identify a resource here (namely, the repository), and that's what URI are for. Unlike URLs, they only identify, but don't always allow to locate the resource. A typical example is, you can use isbn: URIs to identify books by their barcode. This does not tell you where the book is available, but it allows you to uniquely identify the book.

... This has, however, no impact on how we name things ...

Previous to this change, it was very confusing having multiple uses for the term "url" and also different terms for this conceptual "identifier". At the moment the term "url" is consistent in the HDS database, the HDS source, the HDS API signatures, the HD source, the package library and through the repo.info file. Somebody can trace it from start to finish by name and that is of significant value for maintenance. It was quite a bit of work to achieve that so I would be averse to renaming anything without ensuring follow-through to ensure the work were done consistently across both data and software.

Yes, of course, but the name is still a bit confusing. We could rename "url" to "identifier" or "uri" or something like that in all the code and documentation, and add a note that in the HPKR file format, it is stored as "url" for legacy reasons. Or we can introduce a new version of HPKR files that uses the new name instead.

However, what I would prefer to avoid is having both "url" and "identifier" or "uri" live side by side in the HPKR file, as this would further confuse things.

So:

• +1 to renaming "url" in the code, I don't know if it's worth introducing a new version of HPKR files or if we can live with keeping "url" just there
• +1 to using an "uuid:" or other non-http scheme for the identifier to reduce risk of confusion, but only for new repos. For the existing ones, keeping the existing identifiers is probably less problematic (unless we do some special-casing to handle the migration? is it worth the effort?)
• -1 to hardcoding the identifier to be an UUID, as I don't like that they are not helpful to the end-user, and they are a less flexible solution than using an uri (which can then effectively be used with an uuid: URI, but gvies us more flexibility if we want to use something else)

[apl-haiku]

What would an UUID rather than an URI bring us here? I find the URI more generic.

Yes instead of UUID, some URI approach like hpkr:haikuports_x86_64 would be a good solution. In fact there is already a system of such codes in HDS [1] and if those were used it would also help comprehension across the wider data and software.

+1 to renaming "url" in the code,

It would be OK for "url" to remain as-is in the HPKR format as you suggest, but I would only want to see this change happen if there is capacity for it to be coordinated through everything including HDS. That will take time because it means coordinating API change.

---

IMHO, as nice as it would be, given the effort involved, it is probably best sorted out later.

[1] ​https://depot.haiku-os.org/#!/repository/haikuports

[waddlesplash]

Why not use something like the following scheme:

• a url field which is the true "origin" of the repository (likely eu.hpkg.haiku-os.org/... in our case
• a mirror-url field which is the URL that packages will be downloaded from

Then we download the repo.sha256 from url, and everything else from the mirror-url. In the case that the url field changes, then users should probably have to do something manually anyway, indeed. But I think all other concerns this solves?

[pulkomandy]

Haiku IP addresses are blocked in Russia. If we need to download anything from the original repo, some people will not be able to use any mirror.

The "url" is not an url, it's just an identifier. It should never be used as an url. Its naming is legacy from when we didn't plan to have mirrors, and assumed the url could be used both to locate and identify the repo, which is not the case.

For trusting the repo, sha256 is useless as long as there is no signing. And when there is signing, you can trust the content without trusting the transport, so using any random mirror is fine as long as the packages are signed with a known/trusted key.

[waddlesplash]

For trusting the repo, sha256 is useless as long as there is no signing.

Uh... why? The sha256 authenticates the "repo" file as authentic, and the "repo" file itself contains sha256sums for the HPKG files. So as long as the checksum for the "repo" is retrieved from a trusted source securely, all subsequent downloads will also be verifiable. It's not as good as GPG signing, sure; but it's very close by many metrics.

[waddlesplash]

We can also add a flag for "trust mirror for sha256 download also." Then this solves the problem of IP blocks.

[pulkomandy]

Replying to waddlesplash:

For trusting the repo, sha256 is useless as long as there is no signing.

Uh... why? The sha256 authenticates the "repo" file as authentic, and the "repo" file itself contains sha256sums for the HPKG files. So as long as the checksum for the "repo" is retrieved from a trusted source securely, all subsequent downloads will also be verifiable. It's not as good as GPG signing, sure; but it's very close by many metrics.

It's close only if you trust the whole transport between you and Haiku. It's easier to do an MITM especially when you are providing the mirror (you would just need to alter the "url" field, basically). Let's not hack around security, let's sign our package repos (GPG or whatever) and then we can stop worrying about how we distribute the packages. This allows separation of concerns: on one side we worry about signing, and on the other we worry, independently, about distribution. This way we don't multiply problems with each other.

[kallisti5]

For context here is the change that was reverted trying to fix the URL / BaseURL mess:

​https://git.haiku-os.org/haiku/commit/src/kits/package?id=fda0bf77fbdf5876ce81a39da324145143d6ad18

There was some compatibility stuff in the code to prevent it from flat-out breaking everyone, but there were definitely some issues.

As for repo signing... Pulkomandy is correct..ish. The sha256 is valuable since the repo file "contains a sha256 for every package". But since anyone can generate a new repo file with "fake sha256 checksums for malicious packages the chain of trust is incomplete.

This is why the repo files built by concourse (the new CI i'm trying to beat into submission) are signed... ​http://test.hpkg.haiku-os.org/master/x86_64/current/repo.minisig

We could also sign all packages... but that's a lot of work.

[pulkomandy]

Replying to kallisti5:

We could also sign all packages... but that's a lot of work.

And unneeded, that's the point of having the sha256 of the packages in the repo file and then signing the repo file. As long as sha256 is safe to use, we're fine.

Signing each package would add an extra feature: the ability to safely share packages offline (just copy an hpkg from some random person machine to yours, and you can check if it's really a package from Haiku even without access to the matching repo file). It could also be used to control the integrity of an existing haiku install, even if the packages in it are out of date and not in the current repofile anymore.

But all that is extra features for later, and only useful if the signature is stored inside the hpkg file itself (or as an xattr, maybe?)

[axeld]

To get back to the original point, I would favour Stippi's suggestion to put in some simple heuristics to set the correct URL/Base URL -- that will resolve the problem, at least.

Signing and all that would be great, as would renaming to avoid confusion. It won't help us solve this bug, though, and that discussion should better be moved to the mailing lists, I suppose :-)

[apl-haiku]

Link; see ​PR.

[X512]

What are current valid repository URLs? hpkg.haiku-os.org says that server not found.

[apl-haiku]

See my comment ten above. The "baseURL" is actually the URL to the repository, but the "url" is a (badly named for historical reasons) identifier for the repository that is consistent across mirrors. The "url" will most likely not resolve to a resource on the internet whereas the "baseURL" should.

[kallisti5]

the "url" should have been a uuid or something (see my past complaining + attempted fixes lol)

baseURL is the actual repository location. In the case of haiku's repos. I added "hpkg.haiku-os.org" as the identifier, and eu.hpkg.haiku-os.org as the repo.

The sysadmin team could likely add some "informative" static page at hpkg.haiku-os.org about it being a identifier and not an actual repo. (kind of like the page I added at eu.hpkg.haiku-os.org)

[pulkomandy]

So, I've been digging in RFCs a bit. I would suggest using a tag URI as specified in ​https://tools.ietf.org/html/rfc4151. It would look something like this (the scheme requires us to use a past date at which we owned the haiku-os.org domain or can proove that it was not registered to anyone else, we can pick any, I just went with the current year):

• tag:haiku-os.org,2020:repositories/haiku/r1beta2/x86_gcc2 for Haiku
• tag:haiku-os.org,2020:repositories/haikuports/r1beta2/x86_gcc2 for Haikuports

This makes it clear that the "url" is not an http url, while being more readable than an UUID. Anyone can easily assign their own tags using their own domain name or email address when they create a repo.

The current identifier specifies both the branch name ("master") and a tag ("current", I don't know if we stil have per-hrev repos available), should this be kept? Tag URIs support fragments, so we could simply do something like this: tag:haiku-os.org,2020:repositories/haiku/r1beta2/x86_gcc2#hrev12345

(edited to add the missing CPU architecture)

[kallisti5]

will "tag:haiku-os.org,2020:repositories/haiku/r1beta2/x86_gcc2" parse properly under BUrl?

Keep in mind that changing this identifier will "break everyone's existing haiku install" since it will begin tracking under a different unique identifier. Folks will need to manually re-add repos.

[pulkomandy]

I have implemented this in a way that should not create backwards compatibility issues:

​https://review.haiku-os.org/c/haiku/+/2542 ​https://review.haiku-os.org/c/haiku/+/2543

I am moving this to the beta2 milestone as it allows us to get a first step done in renaming the confusing "url" field and using a better identifying scheme for our repositories.

I did not test the repository creation from the modified files (I don't know how to do that). I think the second change should be applied in the beta2 branch only. It is unfortunately more breaking to change the identifier for the "master" nightly builds repository.

[pulkomandy]

Replying to kallisti5:

will "tag:haiku-os.org,2020:repositories/haiku/r1beta2/x86_gcc2" parse properly under BUrl?

It does, updating the change with an unit test. Although it seems BUrl is not used anywhere with the "url" field which is just a BString.

Keep in mind that changing this identifier will "break everyone's existing haiku install" since it will begin tracking under a different unique identifier. Folks will need to manually re-add repos.

Yes, which is why I suggest doign this for beta2 (users will have to switch to the new repos to use that on existing installs anyway) and leaving the master repo unchanged, at least for now.

[apl-haiku]

I am making changes in the HDS system so that the url is just treated as a string and not as a URL. This change should be ready to go soon and then any repository-unique (the same across all mirrors) string for the url field should work as long as it verbatim string-matches between the HDS records and the desktop configuration.

[apl-haiku]

Finally this is resolved in; ​https://review.haiku-os.org/c/haiku/+/1963