haikuporter cannot unmount packages after building some packages

[3dEyes]

At the end of building large packages like mesa or libreoffice I get an error:

unmount: unmounting failed: Device/File/Resource busy
Unable to unmount /FastData/HaikuPorts/haikuports/app-office/libreoffice/work-24.2.1.1/boot/system.
FDs in use by applications:
Haikuporter could not unmount 'system' volume in chroot. Would you like to force an unmount? [y/N]

Apparently the problem occurred after changes href57501 - href57508.

PS: Haiku x86_64, hrev57581

[waddlesplash]

Any chance this happens with any smaller packages than those?

That range contains not only my refactors but also korli's change to open() behavior. Any chance it could be narrowed down further?

[3dEyes]

At the moment I can only reproduce every time on two recipes - mesa and libreoffice. However, today I built a few dozen small packages - everything is fine.

[begasus]

Happens also on R1B4 (both arch's).

[bipolar]

One "easy" way for me to trigger this (not sure if 100% reliably thou) is:

Start a haikuporter build, say... for Python. While the build is in process... Use Tracker's drill-down menus to look up something under the work-* directory.

That usually ends up with:

Unable to unmount /boot/home/SourceCode/haikuports/haikuports/dev-libs/tvision/work-2023.10.03~git/boot/system.
FDs in use by applications:
  317 105  R    18:257 /boot/system/Tracker
  317 107  R    18:255 /boot/system/Tracker
  317 108  R    18:264 /boot/system/Tracker
  317 111  R    18:226 /boot/system/Tracker
  317 113  R    18:504 /boot/system/Tracker
Haikuporter could not unmount 'system' volume in chroot. Would you like to force an unmount? [y/N]

And replying yes results in the nasty looking output:

Forcing unmount
Command '['bash', '-c', '\n\ncheckedUnmount()\n{\n\tlocal mountPoint="$1"\n\n\tif ! [[ $mountPoint = /* ]]; then\n\t\tmountPoint=$PWD/$mountPoint\n\tfi\n\n\t# retry up to 5 times to unmount the given mountpoint\n\tlocal x=0\n\twhile true; do\n\t\tif unmount "$mountPoint"; then\n\t\t\tbreak\n\t\tfi\n\n\t\tlet x+=1\n\t\tif [ $x -ge 5 ]; then\n\t\t\techo -e "Unable to unmount $mountPoint.\\nFDs in use by applications:"\n\t\t\tfdinfo -d "$mountPoint"\n\n\t\t\tread -r -d \'\' message <<-"EOF"\n\t\t\t\tHaikuporter could not unmount "\'$(basename $mountPoint)\'" volume\n\t\t\t\tin chroot. Would you like to force an unmount? [y/N]\n\t\t\t\tEOF\n\t\t\tmessage=$(eval echo -e $message)\n\n\t\t\tnoForceUnmount=1\n\t\t\tif [ -t 0 ]; then\n\t\t\t\tread -p "$message" -n 1 -r\n\t\t\t\t[[ $REPLY =~ ^[Yy]$ ]]\n\t\t\t\tnoForceUnmount=$?\n\t\t\t\techo "$noForceUnmount"\n\t\t\telse\n\t\t\t\t# not running interactively, force an unmount anyway\n\t\t\t\tnoForceUnmount=0\n\t\t\tfi\n\n\t\t\tif [ $noForceUnmount -eq 0 ]; then\n\t\t\t\techo "Forcing unmount"\n\t\t\t\tunmount -f "$mountPoint"\n\t\t\tfi\n\n\t\t\t# fail no matter what was decided\n\t\t\texit 1\n\t\tfi\n\n\t\techo "unmounting $mountPoint failed - wait and retry ..."\n\t\tsleep $x\n\tdone\n}\n\n# ignore sigint\ntrap \'\' SIGINT\n\n# try to make sure we really are in a work directory\nif ! echo $(basename $PWD) | grep -qE \'^work-\'; then\n\techo "cleanupChroot invoked in $PWD, which doesn\'t seem to be a work dir!"\n\texit 1\nfi\n\n# if it is defined, unmount the cross-build sysroot\nif [[ -n $crossSysrootDir && -e $crossSysrootDir/boot/system/develop ]]; then\n\tcheckedUnmount $crossSysrootDir/boot/system\nfi\n\ncheckedUnmount dev\ncheckedUnmount boot/system\n\n# wipe files and directories if it is ok to do so\nif [[ $buildOk ]]; then\n\techo "cleaning chroot folder"\n\trm -rf \\\n\t\tboot \\\n\t\tbuild-packages \\\n\t\tdev \\\n\t\tpackage-infos \\\n\t\tpackages \\\n\t\tpackaging \\\n\t\tprereq-repository \\\n\t\trepository\n\trm -f \\\n\t\t.PackageInfo \\\n\t\tbin \\\n\t\tetc \\\n\t\tport.recipe \\\n\t\tsystem \\\n\t\ttmp \\\n\t\tvar\nelse\n\techo "cleaning \'chroot/boot\' folder"\n\trm -rf boot\n\techo "keeping chroot folder $PWD intact for inspection"\nfi\n']' returned non-zero exit status 1.

And then me having to manually unmount system from the chroot anyway.

[X512]

Use Tracker's drill-down menus to look up something under the work-* directory.

I suspect it is a different problem when some files in work directory are opened by Tracker or some other application.

[bipolar]

FWIW, the only other way I've ever seen logs exactly like in 3dEyes's description (where FDs in use by applications: shows nothing before Haikuporter could not unmount 'system' volume in chroot.) is when:

Running parallel builds or --test (for Python or uncrustify, for example), and I need to kill some of the sub-processes, when they run amok.

But yeah, both of my cases need "user-interaction" to trigger, so I guess that makes them a different issue.

[begasus]

Just had one for building kdenlive, adding syslog from the first run, then when running with "haikuporter -F *".

[begasus]

Unmount failure syslog

[begasus]

Running haikuporter with -F

[waddlesplash]

Seems the problem can be reproduced with Mesa even on an incremental build/install. So that makes testing pretty simple.

[kallisti5]

Yeah. I've found a workaround for getting stuff built on riscv64:

• Build recipe
• Once complete, you see:

  mimesetting files for package libtool_libltdl-2.4.6-3-riscv64.hpkg ...
  creating package libtool_libltdl-2.4.6-3-riscv64.hpkg ...
  ----- Package Info ----------------
  header size:                     80
  heap size:                    52133
  TOC size:                       230
  package attributes size:        578
  total size:                   52213
  -----------------------------------
  waiting for build package libtool-2.4.6-3 to be deactivated
  waiting for build package libtool_libltdl-2.4.6-3 to be deactivated
  unmount: unmounting failed: Device/File/Resource busy
  unmounting /boot/home/haikuports/dev-build/libtool/work-2.4.6/boot/system failed - wait and retry ...
  unmount: unmounting failed: Device/File/Resource busy
  unmounting /boot/home/haikuports/dev-build/libtool/work-2.4.6/boot/system failed - wait and retry ...
  unmount: unmounting failed: Device/File/Resource busy
  unmounting /boot/home/haikuports/dev-build/libtool/work-2.4.6/boot/system failed - wait and retry ...
  unmount: unmounting failed: Device/File/Resource busy
  unmounting /boot/home/haikuports/dev-build/libtool/work-2.4.6/boot/system failed - wait and retry ...
  unmount: unmounting failed: Device/File/Resource busy
  Unable to unmount /boot/home/haikuports/dev-build/libtool/work-2.4.6/boot/system.
  FDs in use by applications:
  Haikuporter could not unmount 'system' volume in chroot. Would you like to force an unmount? [y/N]n1
  

say n. Reboot. rebuild, and then the packaging is successful since you don't get as much disk activity to potentially "hold" open the packagefs mount.

[waddlesplash]

The culprit appears to be hrev57507; reverting that fixes the problem here.

[waddlesplash]

(Specifically, just 77cf55c4ad5e09116e956223cfdfd7695bb903ae.)

I've been looking at that commit for a while and added some asserts around its edge cases, but so far I haven't figured out what could be leaking vnode references in there.

[waddlesplash]

I found a few edge cases and pushed a change for review of these: ​https://review.haiku-os.org/c/haiku/+/7529

However this does not fix the problem.

[waddlesplash]

(Specifically, just 77cf55c4ad5e09116e956223cfdfd7695bb903ae.)

Seems I mistested somehow, because that's not actually the problem, the real culprit is 79c0b6288f5b21adbe2902a10cbcf468cc9d0fd2.

[waddlesplash]

Fixed in hrev57654.

[X512]

@waddlesplash, can you use hrev* for references, not raw Git hashes? Or at least use links to git.haiku-os.org.

[waddlesplash]

Some of those commits were in the middle of hrevs. I suppose I could've used the hrevxxx~1 etc. syntax.