Opened 10 years ago

Last modified 5 years ago

#5498 reopened bug

bfs_shell : stray processes stay active

Reported by: mmadia Owned by: bonefish
Priority: normal Milestone: R1
Component: Build System Version: R1/Development
Keywords: Cc: mati@…
Blocked By: Blocking:
Has a Patch: no Platform: All

Description

For yet undetermined reasons, bfs_shell will remain as an idle process, long after the build cycle completes. This has been observed on FreeBSD 8.0 and on Haiku itself. Attached is some ps output from FreeBSD.

Attachments (2)

ps-bfs_shell.txt (4.2 KB ) - added by mmadia 10 years ago.
lsof.txt (150.9 KB ) - added by mmadia 8 years ago.
from 2011-06-11 on BOM

Download all attachments as: .zip

Change History (11)

by mmadia, 10 years ago

Attachment: ps-bfs_shell.txt added

comment:1 by bonefish, 9 years ago

Resolution: fixed
Status: newclosed

Hopefully fixed in hrev40097/hrev40100.

comment:2 by Wizard, 9 years ago

Cc: mati@… added
Platform: AllPowerPC
Resolution: fixed
Status: closedreopened

Same problem occures when building ppc (hrev41933) on ppc (osx leopard and debian squeeze hosts tested).

comment:3 by bonefish, 9 years ago

Platform: PowerPCAll
Resolution: fixed
Status: reopenedclosed

Please don't reopen and re-purpose closed tickets. Since Matt hasn't added anymore feedback, I assume the original problem is gone. The communication between bfs_shell and client has been rewritten completely, using FIFOs instead of Unix sockets. At least theoretically it shouldn't be possible anymore that bfs_shell continues to wait for commands after build_haiku_image terminated unexpectedly, which I believe was the original issue.

Please open a new ticket and attach the relevant parts of the ps and lsof output. It would also be nice, if you could attach a gdb (gdb <pid>) to the running bfs_shell and get a back trace (bt).

comment:4 by mmadia, 9 years ago

Resolution: fixed
Status: closedreopened

Sorry about that. I must've wanted to wait some time after hrev40097 + hrev40100 and simply forgot to check. Here's the output from the Build-O-Matic box. The bfs_shell processes do seem to be lingering around.

[mmadia@1156 ~]$ uptime
 6:23PM  up 11 days,  3:54, 9 users, load averages: 0.03, 0.30, 0.44

[mmadia@1156 ~]$ ps -aux | grep /
USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
mmadia 21775  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku.image
mmadia 25457  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.image
mmadia 27007  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.vmdk
mmadia 27950  0.0  0.1  6228  2336  ??  D     6:10PM   0:00.03 /Build-O-Matic/output/generated.x86gcc4/objects/freebsd/x86/release/tools/copyattr -d -r /Build-O-Matic/output/generated.x86gcc4/tmp/extract/./. /Build-O-Matic/output/generated.x86gcc4/tmp/c
mmadia 60066  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku.image
mmadia 63667  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.image
mmadia 65200  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.vmdk
mmadia 69804  0.0  0.0 12288     0  ??  IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86
mmadia 87355  0.0  0.0 11264     0  ??  IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86
mmadia 88822  0.0  0.0 11264     0  ??  IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /storage/haiku-releases/r1alpha3/BOM-alpha3
mmadia 90320  0.0  0.0  8192     0   0- IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86
mmadia 27952  0.0  0.0  3500  1444   2  S+    6:10PM   0:00.00 grep /

edit:

Also, there's two instances of Build-O-Matic running concurrently. However, each is running on a different partition, which are also on different drives. Aside from that, each individual BOM is executing its builds in serial.

Last edited 9 years ago by mmadia (previous) (diff)

comment:5 by bonefish, 9 years ago

If lsof or something equivalent exists on FreeBSD, please attach the output, and another ps output, if something has changed. It would help if you could point out a process that should be gone -- at least from the ps output I can't guess that. Please also attach gdb to the process and get a back trace.

by mmadia, 8 years ago

Attachment: lsof.txt added

from 2011-06-11 on BOM

comment:6 by mmadia, 8 years ago

Here's a new block of stats from today. It seems that any process with TT = "??" or STAT = "IW" is an abandoned process (21775, 25457, 27007, 60066, 63667, 65200, 69804, 87355, 88822, 90320)

[mmadia@1156 ~]$ uptime
11:18AM  up 12 days, 20:49, 9 users, load averages: 0.88, 0.81, 0.57

[mmadia@1156 ~]$ ps -aux | grep /
mmadia 21775  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener
mmadia 25457  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener
mmadia 27007  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build
mmadia 60066  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener
mmadia 63667  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener
mmadia 65200  0.0  0.0 12288     0  ??  IW   -         0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build
mmadia 69804  0.0  0.0 12288     0  ??  IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -
mmadia 87355  0.0  0.0 11264     0  ??  IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -
mmadia 88822  0.0  0.0 11264     0  ??  IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -
mmadia 90320  0.0  0.0  8192     0   0- IW   -         0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -
mmadia 66875  0.0  0.0  3500  1444   2  S+   11:15AM   0:00.00 grep /
[mmadia@1156 ~]$ 

Oddly, there seems to be an issue with gdb on my FreeBSD box ...

[mmadia@1156 ~]$ gdb program 21775
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...program: No such file or directory.

Attaching to process 21775
/usr/src/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1443: internal-error: legacy_fetch_link_map_offsets called without legacy link_map support enabled.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) 
[mmadia@1156 ~]$ gdb --pid 25457
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Attaching to process 25457
/usr/src/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1443: internal-error: legacy_fetch_link_map_offsets called without legacy link_map support enabled.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) 

comment:7 by bonefish, 8 years ago

The bfs_shell's open files look OK. The open FIFO ends are as expected:

929	bfs_shell 21775 mmadia    3r                                        0xcb5852d8 file struct, ty=0x4, op=0xc0d89ce0
930	bfs_shell 21775 mmadia    4w                                        0xc6eb57e0 file struct, ty=0x4, op=0xc0d89ce0
931	bfs_shell 21775 mmadia    5r                                        0xcb5852d8 file struct, ty=0x4, op=0xc0d89ce0
932	bfs_shell 21775 mmadia    6w                                        0xc6eb57e0 file struct, ty=0x4, op=0xc0d89ce0

The reason why bfs_shell doesn't quit is that there's still a shell around which holds the other ends of the FIFOs open (FDs 3 and 4):

911	sh        21773 mmadia    3r                                        0xc6eb4d20 file struct, ty=0x4, op=0xc0d89ce0
912	sh        21773 mmadia    4w                                        0xc6f33b98 file struct, ty=0x4, op=0xc0d89ce0
913	sh        21773 mmadia    5r                                        0xcb5852d8 file struct, ty=0x4, op=0xc0d89ce0
914	sh        21773 mmadia    6w                                        0xc6eb57e0 file struct, ty=0x4, op=0xc0d89ce0

Interestingly, as we can see, FDs 5 and 6 are still referring to bfs_shell's FIFO ends. build_haiku_image, however, closes those FDs right after starting bfs_shell. This leads to the conclusion that either closing the FDs doesn't work as it should (bug in FreeBSD's shell?) or this isn't the shell executing build_haiku_image, but one that had been forked off earlier. Please try to find out more about that shell. ps it at least. Don't know what other tools FreeBSD has. Attaching gdb to get a back trace might help too. Hopefully that works better for the shell.

Regarding your gdb session. Have you tried to continue it ("n")? You could also try to start gdb with an LD_LIBRARY_PATH including <generated>/objects/freebsd/lib/ -- maybe the bug is triggered by gdb not finding libroot_build.so.

comment:8 by diver, 7 years ago

Same problem under ubuntu 12.04

ps ax
18127 pts/3    S      0:00 /bin/sh /home/diver/haiku/haiku/build/scripts/build_haiku_image /home/diver/haiku/haiku/generated.x86gcc2
18129 pts/3    S      0:00 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell -n /home/di

ps xauww
diver    18127  0.0  0.0   2236   308 pts/3    S    15:30   0:00 /bin/sh /home/diver/haiku/haiku/build/scripts/build_haiku_image /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-init-vars /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-make-dirs /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-copy-files /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-extract-files
diver    18129  0.0  0.1   8716  5480 pts/3    S    15:30   0:00 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell -n /home/diver/haiku/haiku/generated.x86gcc2/haiku.image
lsof -p 18127
COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
build_hai 18127 diver  cwd    DIR    8,1     4096 32918733 /home/diver/haiku/haiku/generated.x86gcc2
build_hai 18127 diver  rtd    DIR    8,1     4096        2 /
build_hai 18127 diver  txt    REG    8,1   100284 38010908 /bin/dash
build_hai 18127 diver  mem    REG    8,1  1730024 25952269 /lib/i386-linux-gnu/libc-2.15.so
build_hai 18127 diver  mem    REG    8,1   134344 25952387 /lib/i386-linux-gnu/ld-2.15.so
build_hai 18127 diver    0r   CHR    1,3      0t0     5957 /dev/null
build_hai 18127 diver    1w   CHR    1,3      0t0     5957 /dev/null
build_hai 18127 diver    2u   CHR  136,3      0t0        6 /dev/pts/3
build_hai 18127 diver    3r  FIFO    8,1      0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted)
build_hai 18127 diver    4w  FIFO    8,1      0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted)
build_hai 18127 diver    5r  FIFO    8,1      0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted)
build_hai 18127 diver    6w  FIFO    8,1      0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted)
build_hai 18127 diver   10u   CHR  136,3      0t0        6 /dev/pts/3
build_hai 18127 diver   11r  FIFO    8,1      0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted)
build_hai 18127 diver   12w  FIFO    8,1      0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted)
lsof -p 18129
COMMAND     PID  USER   FD   TYPE DEVICE  SIZE/OFF     NODE NAME
bfs_shell 18129 diver  cwd    DIR    8,1      4096 32918733 /home/diver/haiku/haiku/generated.x86gcc2
bfs_shell 18129 diver  rtd    DIR    8,1      4096        2 /
bfs_shell 18129 diver  txt    REG    8,1    438740 33427161 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell
bfs_shell 18129 diver  mem    REG    8,1   1730024 25952269 /lib/i386-linux-gnu/libc-2.15.so
bfs_shell 18129 diver  mem    REG    8,1    116232 25953204 /lib/i386-linux-gnu/libgcc_s.so.1
bfs_shell 18129 diver  mem    REG    8,1    905712 14422997 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16
bfs_shell 18129 diver  mem    REG    8,1    204183 33033908 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/lib/libroot_build.so
bfs_shell 18129 diver  mem    REG    8,1    173576 25952386 /lib/i386-linux-gnu/libm-2.15.so
bfs_shell 18129 diver  mem    REG    8,1    134344 25952387 /lib/i386-linux-gnu/ld-2.15.so
bfs_shell 18129 diver    0r   CHR    1,3       0t0     5957 /dev/null
bfs_shell 18129 diver    1w   CHR    1,3       0t0     5957 /dev/null
bfs_shell 18129 diver    2u   CHR  136,3       0t0        6 /dev/pts/3
bfs_shell 18129 diver    3r  FIFO    8,1       0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted)
bfs_shell 18129 diver    4w  FIFO    8,1       0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted)
bfs_shell 18129 diver    5r  FIFO    8,1       0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted)
bfs_shell 18129 diver    6w  FIFO    8,1       0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted)
bfs_shell 18129 diver    7u   REG    8,1 734003200 32918970 /home/diver/haiku/haiku/generated.x86gcc2/haiku.image (deleted)
sudo gdb -p 18127
Attaching to process 18127
Reading symbols from /bin/dash...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/i386-linux-gnu/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
0xb778d424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb778d424 in __kernel_vsyscall ()
#1  0xb76859ee in wait4 () from /lib/i386-linux-gnu/libc.so.6
#2  0xb76859c7 in wait3 () from /lib/i386-linux-gnu/libc.so.6
#3  0x080506c5 in ?? ()
#4  0x080516ec in ?? ()
#5  0x0804c19e in ?? ()
#6  0x0804b209 in ?? ()
#7  0x0804baa3 in ?? ()
#8  0x0804c349 in ?? ()
#9  0x0804b209 in ?? ()
#10 0x0804b7fc in ?? ()
#11 0x0804b209 in ?? ()
#12 0x0804b209 in ?? ()
#13 0x0804b29e in ?? ()
#14 0x0804b29e in ?? ()
#15 0x0804b29e in ?? ()
#16 0x0804b209 in ?? ()
#17 0x08051ba4 in ?? ()
#18 0x08049801 in ?? ()
#19 0xb75e74d3 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6
#20 0x08049919 in ?? ()
(gdb) 
sudo gdb -p 18129
Attaching to process 18129
Reading symbols from /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell...(no debugging symbols found)...done.
Reading symbols from /lib/i386-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/i386-linux-gnu/libm.so.6
Reading symbols from /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/lib/libroot_build.so...(no debugging symbols found)...done.
Loaded symbols for /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/lib/libroot_build.so
Reading symbols from /usr/lib/i386-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/i386-linux-gnu/libstdc++.so.6
Reading symbols from /lib/i386-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/i386-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/i386-linux-gnu/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
0xb771c424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb771c424 in __kernel_vsyscall ()
#1  0xb74643c3 in read () from /lib/i386-linux-gnu/libc.so.6
#2  0xb73f822b in _IO_file_underflow () from /lib/i386-linux-gnu/libc.so.6
#3  0xb73f9449 in _IO_default_uflow () from /lib/i386-linux-gnu/libc.so.6
#4  0xb73f9260 in __uflow () from /lib/i386-linux-gnu/libc.so.6
#5  0xb73ecc9a in _IO_getline_info () from /lib/i386-linux-gnu/libc.so.6
#6  0xb73ecbe3 in _IO_getline () from /lib/i386-linux-gnu/libc.so.6
#7  0xb73ebb80 in fgets () from /lib/i386-linux-gnu/libc.so.6
#8  0x08086ffd in FSShell::get_external_command(char*, int) ()
#9  0x0807170a in main ()
(gdb) 

comment:9 by pulkomandy, 5 years ago

does this still happen?

Note: See TracTickets for help on using tickets.