Pkgman: partial packages are not picked up on connection drops.

[khallebal]

pkgman DOES pick up the partials if the user interrupts the download manually, but it DOES not and starts over from scratch if the connection drops without user intervention with the error message "connection refused" what ever that means? nothing is written to syslog unfortunately.

[khallebal]

Some more info:

1) why does the partial pkg get deleted the moment the connection drops by itself? this happens only when downloading from the haiku repo tho?, but when the download is interrupted by the user the partial pkg is kept and continues from that file when you retry.

2) When downloading from the haikuports repo the connection is much less flaky than the haiku repo, are the 2 repos not on the same infrastructure? if they are why the difference in behavior mentioned above?

3) Are there 2 distinct problems? one with packagefs deleting partials and one with the infrastructure with unstable connection?

[khallebal]

I have added two pkgman update with strace logs, one with -s the other with -t flags, in case they might have some clues.

[kallisti5]

For others on this ticket, all downloads for the haiku repository are done via HTTP 302 redirect to s3 buckets. haikuports is directly hosted on our infrastructure.

example:

$ curl https://eu.hpkg.haiku-os.org/haiku/master/x86_64/current/repo -v 2>&1 | grep -E "location|30"
{ [130 bytes data]
< HTTP/2 303 
< location: https://s3.wasabisys.com/haiku-repositories-us/master/x86_64/r1~beta4_hrev56627/repo

Actually, i'm not 100% sure we're doing an HTTP/2 303. Maybe related to a recent upgrade of rocket / hpkgbouncer.

​https://github.com/haiku/hpkgbouncer/blob/master/src/main.rs#L105 is the code doing the redirect.

Anyway, in theory the redirect shouldn't prevent the package kit from doing partials, and wasabi s3 supports partials as far as I know.

[kallisti5]

That 303 seems wrong. To be honest, i'm not even sure a partial request will go "through" an HTTP 3xx redirect... however at minimum the 303 vs 302 might be breaking it as 303 is supposed to be reserved for POST requests. I've asked in the rocket forum about it... ​https://github.com/SergioBenitez/Rocket/discussions/2417

curl -r 0-99 https://eu.hpkg.haiku-os.org/haiku/master/x86_64/current/repo -v

redirect via HTTP 303... no partial.

vs

curl -r 0-99 https://s3.wasabisys.com/haiku-repositories-us/master/x86_64/r1~beta4_hrev56627/repo -v

partial success

vs

curl -r 0-99 https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/repo -v

partial success

Keep in mind this was all in place long before we added resuming package downloads in the package kit.

[korli]

What about

curl -L -r 0-99 https://eu.hpkg.haiku-os.org/haiku/master/x86_64/current/repo -v

?

[kallisti5]

Yeah, you're right. The -L to curl fixes it as it instructs the client to resend the full request to the redirected URL.

Now the big question is how to do that within our Package kit. It feels like a good change since paired with the range it will ensure complete requests are forwarded to redirected URLs. This would apply to http -> https redirections as well.

[kallisti5]

Here's the code location:

​https://cgit.haiku-os.org/haiku/tree/src/kits/network/libnetservices/HttpRequest.cpp#n452

[pulkomandy]

The code in libnetservices already preserves the full request. As you can see in that code, only fUrl is modified, and the request is restarted with the same data (headers and everything). Unless the request is indeed a POST request (which it shouldn't be in this case).

Does the package kit use this code for auto-redirect, or does it disable it and manage redirect on its own?