pkgman download stall

[kallisti5]

A pkgman stall results in a long loop of retries:

hrev54880, x86_64

The following changes will be made:
  in system:
    install package llvm9-9.0.1-2 from repository HaikuPorts
    install package llvm9_clang-9.0.1-2 from repository HaikuPorts
    uninstall package llvm8-8.0.0-1
    uninstall package llvm8_clang-8.0.0-1
Continue? [yes/no] (yes) : yes
  0%downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9-9.0.1-2-x86_64.hpkg
100% llvm9-9.0.1-2-x86_64.hpkg [60.45 MiB]
Validating checksum for https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9-9.0.1-2-x86_64.hpkg...done.
Re-using download '/boot/system/packages/administrative/state_2021-01-11_09:50:20/llvm9_clang-9.0.1-2-x86_64.hpkg' from previous transaction (partial)
  0%downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185792964
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185793177
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185793390
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185793603
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185793816
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185794029
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185794242
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185794455
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185794668
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185794881
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185795094
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185795307
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185795520
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185795733
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185795946
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185796159
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185796372
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185796585
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185796798
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185797011
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185797224
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185797437
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185797650
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185797863
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185798076
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185798289
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185798502
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185798715
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185798928
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185799141
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185799354
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185799567
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185799780
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185799993
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185800206
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185800419
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185800632
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185800845
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185801058
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185801271
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185801484
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185801697
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185801910
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185802123
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185802336
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg
requesting range start 185802549
downloading https://eu.hpkg.haiku-os.org/haikuports/master/x86_64/current/packages/llvm9_clang-9.0.1-2-x86_64.hpkg

[kallisti5]

You can force pkgman to redownload everything via:

rm -rf /boot/system/packages/administrative/transaction-*

We really need a few new commands to pkgman...

• pkgman clean (transactions + all states minus last 5)
• pkgman clean transactions (delete all transactions)
• pkgman clean states (all states minus last 5)

[kallisti5]

btw.. the results reported above might be "working as designed"? It's just ugly as sin :-)

[pulkomandy]

First this should be merged: ​https://review.haiku-os.org/c/haiku/+/3619 to automatically delete corrupt files

Probably also this but it's still work in progress: ​https://review.haiku-os.org/c/haiku/+/3621

[stippi]

As you can see from the "requesting range start X" log messages, it requests a higher start offset on each retry. Isn't that exactly what people asked for? I would consider it a problem, if this were an endless loop, i.e. always requesting the same start offset. But the offset increases, it is getting some bytes each time around. That means it is making progress. Why do you call it a "stall", and also, what do you mean by "pkgman stall results in..." Perhaps you mean that the retries give the appearance that pkgman has stalled? But you see on the command line that it is making progress.

[waddlesplash]

Requesting small chunks at a time seems inefficient, can't we request the entire rest of the file at once?

[pulkomandy]

We request until the end of the file. If people have unstable connection and it keeps failing multiple time, the code will retry with increasing start offset until it manages to download the whole file.

We don't request small chunks.

Also, that debug trace has been removed.

[stippi]

Replying to pulkomandy:

We request until the end of the file. If people have unstable connection and it keeps failing multiple time, the code will retry with increasing start offset until it manages to download the whole file.

Yes, and that was the whole point and why I didn't include any maximum number of retries. You should be able to let it run over night, come back next morning and it finally updated.

Close as invalid?

[kallisti5]

The issue is i'm on a 200Mbit connection, it shouldn't take "overnight" to download a few hundred MiB :-)

I haven't tried recently however after Pulkomandy's fixes. Let me give it another try and report back here.

[stippi]

Your own connection doesn't necessarily have anything to do with it. I'm on a 16MBit connection (congratulations to you), but a while ago, I experienced issues with performing a Haiku system update. The bytes were trickling in and connections broke up. This is when I first had some motivation to look into this.

In any case, you can see from your own log output that the retries work. Unless Adrien's and my recent changes caused the connection resets you experienced in the first place, then there is no regression. Instead, the output shows that it is working as intended, as you wonder yourself in comment:4.

Can you reproduce this? What other explanations do we have for the resetting connections? (Assuming that is what it is...)

[kallisti5]

I just ran another test after updating to the current hrev, Pulkomandy fixed the retry issue it looks like... however pkgman isn't "resuming" the download when I interrupt it.

​https://gateway.pinata.cloud/ipfs/QmTzV3u7EPqSPKzKisQyTjm61qrceAqVMHk1rP7pAQw55H

It says resuming, but it still redownloads all 4MiB. Is that the desired result?

[stippi]

Sorry, I don't understand what exactly happened from the screenshot. I think between the time of this ticket and now, Pulkomandy merged two possibly related commits. One makes sure that corrupt downloads are deleted (those that fail the checksum check after having finished downloading). The other one shouldn't really have an effect. It's more of an additional check that is performed. But from the original log output, resuming clearly already worked before this commit was merged. The commit should only help in those cases where resuming didn't work yet, i.e. the SSL layer emits no error, but the download is still incomplete according to the Content-Length header.

That leaves the issue:

It says resuming, but it still redownloads all 4MiB. Is that the desired result?

What do you mean by that? When resuming, it should skip ahead to the already downloaded amount in the progress bar, but it would still display the full size of the package. I /thought/ this was less confusing. Maybe not. Maybe only with a 200 MBit connection when you can't really see progress anyway. For a slow connection, the user has a good chance of noticing that the progress bar jumped to 3 MB and now the rest is actually being transferred. What do you think?

[kallisti5]

What do you mean by that? When resuming, it should skip ahead to the already downloaded amount in the progress bar, but it would still display the full size of the package. I

It seemingly restarts every time. I ctl+c at ~3MiB downloaded, and re-run the command again. It downloads the entirety of the 4MiB again with a message it's resuming. (counting upward from 0)

You're correct.. it could be my fast connection makes it seem like it's not working. Could someone else confirm following the same steps?

The messaging Pulkomandy pushed is definitely an improvement.

[pulkomandy]

So, I checked what happens. I think there is still a problem.

Here is what I did:

pkgman install libreoffice_x86

waited for it to download some packages. Pressed control+C when flite_x86 was at 4MB downlaoded (out of 14MB).

pkgman install libreoffice_x86

download of flite resumes at 4MB and continues. I pressed control+C again at about 8MB.

pkgman install libreoffice_x86

The download resumes from the 4MB mark instead of the expected 8MB mark.

Repeated this a few times.

It appears the download is always resumed from the oldest partially downloaded file (the one in transaction-1) and the next partially downloaded ones in transaction-{2,3,4} are ignored.

So, the HTTP range request code is working fine. But, we are not resuming the download from the optimal place.

The problem is in code I added in BPackageManager::_PreparePackageChanges(). It just gets the first file it can find in any transaction directory. It should look harder and find the largest one instead.

[pulkomandy]

​https://review.haiku-os.org/c/haiku/+/3672

[stippi]

Picking best download to resume merged in hrev54920.