Opened 14 years ago
Last modified 4 years ago
#5498 reopened bug
bfs_shell : stray processes stay active
| Reported by: | mmadia | Owned by: | bonefish |
|---|---|---|---|
| Priority: | normal | Milestone: | R1.1 |
| Component: | Build System | Version: | R1/Development |
| Keywords: | Cc: | mati@… | |
| Blocked By: | Blocking: | ||
| Platform: | All |
Description
For yet undetermined reasons, bfs_shell will remain as an idle process, long after the build cycle completes. This has been observed on FreeBSD 8.0 and on Haiku itself. Attached is some ps output from FreeBSD.
Attachments (2)
Change History (12)
by , 14 years ago
| Attachment: | ps-bfs_shell.txt added |
|---|
comment:1 by , 13 years ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
comment:2 by , 13 years ago
| Cc: | added |
|---|---|
| Platform: | All → PowerPC |
| Resolution: | fixed |
| Status: | closed → reopened |
Same problem occures when building ppc (hrev41933) on ppc (osx leopard and debian squeeze hosts tested).
comment:3 by , 13 years ago
| Platform: | PowerPC → All |
|---|---|
| Resolution: | → fixed |
| Status: | reopened → closed |
Please don't reopen and re-purpose closed tickets. Since Matt hasn't added anymore feedback, I assume the original problem is gone. The communication between bfs_shell and client has been rewritten completely, using FIFOs instead of Unix sockets. At least theoretically it shouldn't be possible anymore that bfs_shell continues to wait for commands after build_haiku_image terminated unexpectedly, which I believe was the original issue.
Please open a new ticket and attach the relevant parts of the ps and lsof output. It would also be nice, if you could attach a gdb (gdb <pid>) to the running bfs_shell and get a back trace (bt).
comment:4 by , 13 years ago
| Resolution: | fixed |
|---|---|
| Status: | closed → reopened |
Sorry about that. I must've wanted to wait some time after hrev40097 + hrev40100 and simply forgot to check. Here's the output from the Build-O-Matic box. The bfs_shell processes do seem to be lingering around.
[mmadia@1156 ~]$ uptime 6:23PM up 11 days, 3:54, 9 users, load averages: 0.03, 0.30, 0.44 [mmadia@1156 ~]$ ps -aux | grep / USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND mmadia 21775 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku.image mmadia 25457 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.image mmadia 27007 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.vmdk mmadia 27950 0.0 0.1 6228 2336 ?? D 6:10PM 0:00.03 /Build-O-Matic/output/generated.x86gcc4/objects/freebsd/x86/release/tools/copyattr -d -r /Build-O-Matic/output/generated.x86gcc4/tmp/extract/./. /Build-O-Matic/output/generated.x86gcc4/tmp/c mmadia 60066 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku.image mmadia 63667 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.image mmadia 65200 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build-O-Matic/output/generated.x86gcc2/haiku-alpha.vmdk mmadia 69804 0.0 0.0 12288 0 ?? IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86 mmadia 87355 0.0 0.0 11264 0 ?? IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86 mmadia 88822 0.0 0.0 11264 0 ?? IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /storage/haiku-releases/r1alpha3/BOM-alpha3 mmadia 90320 0.0 0.0 8192 0 0- IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86 mmadia 27952 0.0 0.0 3500 1444 2 S+ 6:10PM 0:00.00 grep /
edit:
Also, there's two instances of Build-O-Matic running concurrently. However, each is running on a different partition, which are also on different drives. Aside from that, each individual BOM is executing its builds in serial.
comment:5 by , 13 years ago
If lsof or something equivalent exists on FreeBSD, please attach the output, and another ps output, if something has changed. It would help if you could point out a process that should be gone -- at least from the ps output I can't guess that. Please also attach gdb to the process and get a back trace.
comment:6 by , 13 years ago
Here's a new block of stats from today. It seems that any process with TT = "??" or STAT = "IW" is an abandoned process (21775, 25457, 27007, 60066, 63667, 65200, 69804, 87355, 88822, 90320)
[mmadia@1156 ~]$ uptime 11:18AM up 12 days, 20:49, 9 users, load averages: 0.88, 0.81, 0.57 [mmadia@1156 ~]$ ps -aux | grep / mmadia 21775 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener mmadia 25457 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener mmadia 27007 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build mmadia 60066 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener mmadia 63667 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n /Build-O-Matic/output/gener mmadia 65200 0.0 0.0 12288 0 ?? IW - 0:00.00 /Build-O-Matic/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell -n --start-offset 65536 /Build mmadia 69804 0.0 0.0 12288 0 ?? IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell - mmadia 87355 0.0 0.0 11264 0 ?? IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell - mmadia 88822 0.0 0.0 11264 0 ?? IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell - mmadia 90320 0.0 0.0 8192 0 0- IW - 0:00.00 /storage/haiku-releases/r1alpha3/BOM-alpha3/output/generated.x86gcc2/objects/freebsd/x86/release/tools/bfs_shell/bfs_shell - mmadia 66875 0.0 0.0 3500 1444 2 S+ 11:15AM 0:00.00 grep / [mmadia@1156 ~]$
Oddly, there seems to be an issue with gdb on my FreeBSD box ...
[mmadia@1156 ~]$ gdb program 21775 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"...program: No such file or directory. Attaching to process 21775 /usr/src/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1443: internal-error: legacy_fetch_link_map_offsets called without legacy link_map support enabled. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n)
[mmadia@1156 ~]$ gdb --pid 25457 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Attaching to process 25457 /usr/src/gnu/usr.bin/gdb/libgdb/../../../../contrib/gdb/gdb/solib-svr4.c:1443: internal-error: legacy_fetch_link_map_offsets called without legacy link_map support enabled. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n)
comment:7 by , 13 years ago
The bfs_shell's open files look OK. The open FIFO ends are as expected:
929 bfs_shell 21775 mmadia 3r 0xcb5852d8 file struct, ty=0x4, op=0xc0d89ce0 930 bfs_shell 21775 mmadia 4w 0xc6eb57e0 file struct, ty=0x4, op=0xc0d89ce0 931 bfs_shell 21775 mmadia 5r 0xcb5852d8 file struct, ty=0x4, op=0xc0d89ce0 932 bfs_shell 21775 mmadia 6w 0xc6eb57e0 file struct, ty=0x4, op=0xc0d89ce0
The reason why bfs_shell doesn't quit is that there's still a shell around which holds the other ends of the FIFOs open (FDs 3 and 4):
911 sh 21773 mmadia 3r 0xc6eb4d20 file struct, ty=0x4, op=0xc0d89ce0 912 sh 21773 mmadia 4w 0xc6f33b98 file struct, ty=0x4, op=0xc0d89ce0 913 sh 21773 mmadia 5r 0xcb5852d8 file struct, ty=0x4, op=0xc0d89ce0 914 sh 21773 mmadia 6w 0xc6eb57e0 file struct, ty=0x4, op=0xc0d89ce0
Interestingly, as we can see, FDs 5 and 6 are still referring to bfs_shell's FIFO ends. build_haiku_image, however, closes those FDs right after starting bfs_shell. This leads to the conclusion that either closing the FDs doesn't work as it should (bug in FreeBSD's shell?) or this isn't the shell executing build_haiku_image, but one that had been forked off earlier. Please try to find out more about that shell. ps it at least. Don't know what other tools FreeBSD has. Attaching gdb to get a back trace might help too. Hopefully that works better for the shell.
Regarding your gdb session. Have you tried to continue it ("n")? You could also try to start gdb with an LD_LIBRARY_PATH including <generated>/objects/freebsd/lib/ -- maybe the bug is triggered by gdb not finding libroot_build.so.
comment:8 by , 11 years ago
Same problem under ubuntu 12.04
ps ax 18127 pts/3 S 0:00 /bin/sh /home/diver/haiku/haiku/build/scripts/build_haiku_image /home/diver/haiku/haiku/generated.x86gcc2 18129 pts/3 S 0:00 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell -n /home/di ps xauww diver 18127 0.0 0.0 2236 308 pts/3 S 15:30 0:00 /bin/sh /home/diver/haiku/haiku/build/scripts/build_haiku_image /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-init-vars /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-make-dirs /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-copy-files /home/diver/haiku/haiku/generated.x86gcc2/haiku.image-extract-files diver 18129 0.0 0.1 8716 5480 pts/3 S 15:30 0:00 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell -n /home/diver/haiku/haiku/generated.x86gcc2/haiku.image
lsof -p 18127 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME build_hai 18127 diver cwd DIR 8,1 4096 32918733 /home/diver/haiku/haiku/generated.x86gcc2 build_hai 18127 diver rtd DIR 8,1 4096 2 / build_hai 18127 diver txt REG 8,1 100284 38010908 /bin/dash build_hai 18127 diver mem REG 8,1 1730024 25952269 /lib/i386-linux-gnu/libc-2.15.so build_hai 18127 diver mem REG 8,1 134344 25952387 /lib/i386-linux-gnu/ld-2.15.so build_hai 18127 diver 0r CHR 1,3 0t0 5957 /dev/null build_hai 18127 diver 1w CHR 1,3 0t0 5957 /dev/null build_hai 18127 diver 2u CHR 136,3 0t0 6 /dev/pts/3 build_hai 18127 diver 3r FIFO 8,1 0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted) build_hai 18127 diver 4w FIFO 8,1 0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted) build_hai 18127 diver 5r FIFO 8,1 0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted) build_hai 18127 diver 6w FIFO 8,1 0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted) build_hai 18127 diver 10u CHR 136,3 0t0 6 /dev/pts/3 build_hai 18127 diver 11r FIFO 8,1 0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted) build_hai 18127 diver 12w FIFO 8,1 0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted)
lsof -p 18129 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bfs_shell 18129 diver cwd DIR 8,1 4096 32918733 /home/diver/haiku/haiku/generated.x86gcc2 bfs_shell 18129 diver rtd DIR 8,1 4096 2 / bfs_shell 18129 diver txt REG 8,1 438740 33427161 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell bfs_shell 18129 diver mem REG 8,1 1730024 25952269 /lib/i386-linux-gnu/libc-2.15.so bfs_shell 18129 diver mem REG 8,1 116232 25953204 /lib/i386-linux-gnu/libgcc_s.so.1 bfs_shell 18129 diver mem REG 8,1 905712 14422997 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 bfs_shell 18129 diver mem REG 8,1 204183 33033908 /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/lib/libroot_build.so bfs_shell 18129 diver mem REG 8,1 173576 25952386 /lib/i386-linux-gnu/libm-2.15.so bfs_shell 18129 diver mem REG 8,1 134344 25952387 /lib/i386-linux-gnu/ld-2.15.so bfs_shell 18129 diver 0r CHR 1,3 0t0 5957 /dev/null bfs_shell 18129 diver 1w CHR 1,3 0t0 5957 /dev/null bfs_shell 18129 diver 2u CHR 136,3 0t0 6 /dev/pts/3 bfs_shell 18129 diver 3r FIFO 8,1 0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted) bfs_shell 18129 diver 4w FIFO 8,1 0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted) bfs_shell 18129 diver 5r FIFO 8,1 0t0 29884469 /tmp/build_haiku_image-18113-fifo-to-shell (deleted) bfs_shell 18129 diver 6w FIFO 8,1 0t0 29884470 /tmp/build_haiku_image-18113-fifo-from-shell (deleted) bfs_shell 18129 diver 7u REG 8,1 734003200 32918970 /home/diver/haiku/haiku/generated.x86gcc2/haiku.image (deleted)
sudo gdb -p 18127 Attaching to process 18127 Reading symbols from /bin/dash...(no debugging symbols found)...done. Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i386-linux-gnu/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 0xb778d424 in __kernel_vsyscall () (gdb) bt #0 0xb778d424 in __kernel_vsyscall () #1 0xb76859ee in wait4 () from /lib/i386-linux-gnu/libc.so.6 #2 0xb76859c7 in wait3 () from /lib/i386-linux-gnu/libc.so.6 #3 0x080506c5 in ?? () #4 0x080516ec in ?? () #5 0x0804c19e in ?? () #6 0x0804b209 in ?? () #7 0x0804baa3 in ?? () #8 0x0804c349 in ?? () #9 0x0804b209 in ?? () #10 0x0804b7fc in ?? () #11 0x0804b209 in ?? () #12 0x0804b209 in ?? () #13 0x0804b29e in ?? () #14 0x0804b29e in ?? () #15 0x0804b29e in ?? () #16 0x0804b209 in ?? () #17 0x08051ba4 in ?? () #18 0x08049801 in ?? () #19 0xb75e74d3 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6 #20 0x08049919 in ?? () (gdb)
sudo gdb -p 18129 Attaching to process 18129 Reading symbols from /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/x86/release/tools/bfs_shell/bfs_shell...(no debugging symbols found)...done. Reading symbols from /lib/i386-linux-gnu/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i386-linux-gnu/libm.so.6 Reading symbols from /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/lib/libroot_build.so...(no debugging symbols found)...done. Loaded symbols for /home/diver/haiku/haiku/generated.x86gcc2/objects/linux/lib/libroot_build.so Reading symbols from /usr/lib/i386-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/i386-linux-gnu/libstdc++.so.6 Reading symbols from /lib/i386-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/i386-linux-gnu/libgcc_s.so.1 Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i386-linux-gnu/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 0xb771c424 in __kernel_vsyscall () (gdb) bt #0 0xb771c424 in __kernel_vsyscall () #1 0xb74643c3 in read () from /lib/i386-linux-gnu/libc.so.6 #2 0xb73f822b in _IO_file_underflow () from /lib/i386-linux-gnu/libc.so.6 #3 0xb73f9449 in _IO_default_uflow () from /lib/i386-linux-gnu/libc.so.6 #4 0xb73f9260 in __uflow () from /lib/i386-linux-gnu/libc.so.6 #5 0xb73ecc9a in _IO_getline_info () from /lib/i386-linux-gnu/libc.so.6 #6 0xb73ecbe3 in _IO_getline () from /lib/i386-linux-gnu/libc.so.6 #7 0xb73ebb80 in fgets () from /lib/i386-linux-gnu/libc.so.6 #8 0x08086ffd in FSShell::get_external_command(char*, int) () #9 0x0807170a in main () (gdb)
comment:10 by , 4 years ago
| Milestone: | R1 → R1.1 |
|---|
Note:
See TracTickets
for help on using tickets.
Hopefully fixed in hrev40097/hrev40100.