Opened 4 years ago

Last modified 3 years ago

#13219 new bug

package_daemon inspects copied hpkg files too early

Reported by: jua Owned by: bonefish
Priority: normal Milestone: Unscheduled
Component: Servers/package_daemon Version: R1/Development
Keywords: Cc: luroh
Blocked By: Blocking:
Platform: All


When an HPKG file is copied to the package dir manually, it sometimes fails to activate. This only happens when copying, and only if the HPKG is sufficiently large and not in the block cache yet (so that the copy isn't too fast).

It seems that package_daemon inspects the hpkg file already while it is still being copied, and thus rejects the incomplete file as invalid (thanks to Rene for the tip).

For testing, I have a really large (800MB) HPKG. If I copy it into the package dir after a fresh reboot, the problem always occurs. But if I first copy the HPKG to somewhere else on the boot drive and then move it into the package dir, it works fine.

In the failure case (file was copied), syslog shows the following:

KERN: package_daemon [37661302:   657] Volume::_PackagesEntryCreated("TeXLive.hpkg")
KERN: package_daemon [37661763:   657] failed to init package for file "TeXLive.hpkg"

This syslog output already appears before Tracker has finished copying the file, hence the assumption of what the problem is.

Change History (6)

comment:1 by diver, 4 years ago

I believe it's a missing feature, see wiki:PackageManagement/TODO#PackageDaemon

comment:2 by jua, 4 years ago

Oh, right, I wasn't aware of this TODO.

Then maybe as a workaround we could at least add special handling for the packages directory to Tracker, so that it doesn't allow directly copying into it for the time being? (in case such a thing can be hacked into Tracker without being too ugly...)

comment:3 by axeld, 4 years ago

That change might be more work than fixing the package daemon, though :-)

comment:4 by pulkomandy, 4 years ago

I wonder if it is possible to detect that the package is incomplete, and wait for more node monitor notifications (for stat changes?) until the copy is complete. It should be possible to know the package size from the header (at least the package tool knows and complains about truncated files).

However, it is possible that Tracker (or other copy tools: rsync, cp, scp, wget, ...) uses fallocate or similar to pre-set the size of the file, and in that case, this wouldn't work. More validation would then be needed, and another way to know when to retry.

comment:5 by axeld, 3 years ago

We already send out notifications that allow to differentiate between "someone IS writing" and "someone HAS written". One just have to listen for stat changes for that file, and don't react on a "file created" event directly.

comment:6 by luroh, 3 years ago

Cc: luroh added
Note: See TracTickets for help on using tickets.