package_daemon inspects copied hpkg files too early

[jua]

When an HPKG file is copied to the package dir manually, it sometimes fails to activate. This only happens when copying, and only if the HPKG is sufficiently large and not in the block cache yet (so that the copy isn't too fast).

It seems that package_daemon inspects the hpkg file already while it is still being copied, and thus rejects the incomplete file as invalid (thanks to Rene for the tip).

For testing, I have a really large (800MB) HPKG. If I copy it into the package dir after a fresh reboot, the problem always occurs. But if I first copy the HPKG to somewhere else on the boot drive and then move it into the package dir, it works fine.

In the failure case (file was copied), syslog shows the following:

KERN: package_daemon [37661302:   657] Volume::_PackagesEntryCreated("TeXLive.hpkg")
KERN: package_daemon [37661763:   657] failed to init package for file "TeXLive.hpkg"

This syslog output already appears before Tracker has finished copying the file, hence the assumption of what the problem is.

[diver]

I believe it's a missing feature, see wiki:PackageManagement/TODO#PackageDaemon

[jua]

Oh, right, I wasn't aware of this TODO.

Then maybe as a workaround we could at least add special handling for the packages directory to Tracker, so that it doesn't allow directly copying into it for the time being? (in case such a thing can be hacked into Tracker without being too ugly...)

[axeld]

That change might be more work than fixing the package daemon, though :-)

[pulkomandy]

I wonder if it is possible to detect that the package is incomplete, and wait for more node monitor notifications (for stat changes?) until the copy is complete. It should be possible to know the package size from the header (at least the package tool knows and complains about truncated files).

However, it is possible that Tracker (or other copy tools: rsync, cp, scp, wget, ...) uses fallocate or similar to pre-set the size of the file, and in that case, this wouldn't work. More validation would then be needed, and another way to know when to retry.

[axeld]

We already send out notifications that allow to differentiate between "someone IS writing" and "someone HAS written". One just have to listen for stat changes for that file, and don't react on a "file created" event directly.