Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#12237 closed bug (fixed)

[launch_daemon] boot hangs at rocket icon

Reported by: diver Owned by: axeld
Priority: blocker Milestone: R1/beta1
Component: Servers/launch_daemon Version: R1/Development
Keywords: Cc:
Blocked By: Blocking: #12148
Platform: All

Description (last modified by diver)

Ever since I've updated to launch_daemon in hrev49432 boot process sometimes hangs at rocket icon. It happens ~ 1/5 boots.

Attachments (3)

boot_hang.png (72.8 KB ) - added by diver 9 years ago.
syslog_hrev49453.txt (45.0 KB ) - added by luroh 9 years ago.
kdl_session.png (46.0 KB ) - added by diver 9 years ago.

Download all attachments as: .zip

Change History (25)

by diver, 9 years ago

Attachment: boot_hang.png added

comment:1 by diver, 9 years ago

Description: modified (diff)

comment:2 by diver, 9 years ago

Note that you can ignore Launching x-vnd.haiku-midi_server failed: No such file or directory message as midi_server was blacklisted here. Enabling it didn't change anything.

comment:3 by luroh, 9 years ago

I've noticed the same in VirtualBox 5.0.0, attaching syslog from hung boot attempt, gcc2h hrev49453.

by luroh, 9 years ago

Attachment: syslog_hrev49453.txt added

comment:4 by waddlesplash, 9 years ago

Priority: highblocker

Happened to another user who was trying Haiku out (gcc2hybrid), and happens very often on my fresh x86_64 install. Elevating to Blocker.

comment:5 by diver, 9 years ago

Also reproducible in VMware Fusion 7.1.2.

comment:6 by axeld, 9 years ago

Please do a "teams", and a "threads <id-of-launch-daemon>" and attach the output here when it happens next time, thanks!

comment:7 by axeld, 9 years ago

In KDL, I mean.

comment:8 by diver, 9 years ago

Entered KDL on OS X with VBoxManage controlvm Haiku keyboardputscancode 38 54 20

by diver, 9 years ago

Attachment: kdl_session.png added

comment:9 by michaelvoliveira, 9 years ago

The same here!

comment:10 by fishpond, 9 years ago

On my core i7 quad, boot failed 100% of the time, now works again - maybe due to fix for #12289?

Version 0, edited 9 years ago by fishpond (next)

comment:11 by fishpond, 9 years ago

Another observation: This bug hits me 99% of the time when booting from a USB stick and then switching to a hard disk boot volume in the boot options. I'm getting down to 50% failure rate when disconnecting the USB stick before continuing the boot process. Don't know if that helps pinpointing the root cause...

comment:12 by waddlesplash, 9 years ago

Blocking: 12148 added

(In #12148) There's a newer issue that has more info than this one, closing as duplicate.

comment:13 by mmlr, 9 years ago

Please recheck with hrev49561 as it might affect this. If it does make a difference there could be a timing dependent deadlock that is either related to or just made more obvious by the regression fixed in that change.

comment:14 by diver, 9 years ago

hrev49564. Did 20 reboots (which was more then needed to reproduce it) and was about to close this one when in hung :/

comment:15 by vidrep, 9 years ago

I was having the same issue pre-launch daemon (#12148). Is there a possibility that the launch daemon is not responsible at all for the symptom?

comment:16 by diver, 9 years ago

Never happened for me pre-launch daemon.

comment:17 by fishpond, 9 years ago

Blacklisting cpuidle as indicated in #12148 removes problem for me on i7 quad.

Last edited 9 years ago by fishpond (previous) (diff)

comment:18 by mmlr, 9 years ago

Please recheck with hrev49570 as it fixes a use-after-free due to a race condition in registrar that was made more likely by the parallel application launch process introduced with the launch_daemon.

Generally this ticket might track two unrelated problems with the same generic symptoms.

comment:19 by diver, 9 years ago

hrev49570. Hang after 10 reboots. Anything I can try?

comment:20 by diver, 9 years ago

kdebug> teams
team           id  parent      name
0x828bf600      1  0x00000000  kernel_team
0x828b6c00    171  0x828bf600  launch_daemon
0x828b6600    173  0x828b6c00  app_server
0x828b6000    174  0x828b6c00  cddb_daemon
0x828b5a00    175  0x828b6c00  debug_server
0x828b4e00    177  0x828b6c00  midi_server
0x828b5400    178  0x828b6c00  mount_server
0x828b4800    179  0x828b6c00  net_server
0x828b4200    180  0x828b6c00  notification_server
0x828b3c00    181  0x828b6c00  package_daemon
0x828b3600    182  0x828b6c00  power_daemon
0x828b3000    183  0x828b6c00  print_server
0x828b2a00    184  0x828b6c00  registrar
0x828b2400    185  0x828b6c00  syslog_daemon

kdebug> threads
thread         id  state     wait for  object      cpu pri  stack      team  name
0x801d1f00      1  running   -                       0   0  0x81001000    1  idle thread 1
0x82a0fa20      2  waiting   cvar      0x801e3328    -  15  0x81105000    1  undertaker
0x82a0f5d0      3  zzz                               -   5  0x81a19000    1  kernel daemon
0x82a0f180      4  zzz                               -   5  0x81a1d000    1  resource resizer
0x82a0ed30      5  waiting   sem       12            -  21  0x81a22000    1  acpi_task
0x82a0e8e0      6  zzz                               -   1  0x81a2a000    1  page scrubber
0x82a0e490      7  waiting   cvar      0x801ec934    -  11  0x81a2e000    1  page writer
0x82a0e040      8  waiting   cvar      0x801ec974    -  10  0x81a32000    1  page daemon
0x82a0dbf0      9  waiting   cvar      0x801ec7d0    - 110  0x81a36000    1  object cache resizer
0x82a0d7a0     10  waiting   sem       29            -   5  0x81a3a000    1  low resource manager
0x82a0d350     11  waiting   cvar      0x801cb2b8    -  10  0x81a42000    1  dpc: normal priority
0x82a0cf00     12  waiting   cvar      0x801cb2f8    -  20  0x81a46000    1  dpc: high priority
0x82a0cab0     13  waiting   cvar      0x801cb338    - 100  0x81a4a000    1  dpc: real-time priority
0x82a0c660     14  waiting   sem       42            -   5  0x81a4e000    1  block notifier/writer
0x82a0bdc0    141  waiting   sem       551           -  20  0x81b56000    1  ohci finish thread
0x82a0b970    142  zzz                               -   5  0x81b5a000    1  usb explore
0x82a0b520    143  zzz                               -  10  0x81864000    1  media checker
0x82a0b0d0    157  waiting   sem       618           -  10  0x81868000    1  locked_pool_enlarger
0x82a0ac80    158  waiting   sem       634           -  20  0x81b5e000    1  scsi_bus_service
0x82a0a830    159  waiting   sem       701           -  20  0x81b66000    1  scsi_bus_service
0x82a0a3e0    160  waiting   sem       731           -  20  0x81b6e000    1  scsi_bus_service
0x82a09f90    161  waiting   cvar      0x8296dc54    -  12  0x81bf6000    1  scsi scheduler 1
0x82a09b40    162  waiting   cvar      0x8296dc7c    -  12  0x81bfa000    1  scsi notifier 1
0x82a096f0    163  waiting   cvar      0x8296db54    -  12  0x81c91000    1  scsi scheduler 2
0x82a092a0    164  waiting   cvar      0x8296db7c    -  12  0x81c95000    1  scsi notifier 2
0x82a08e50    169  waiting   sem       806           -  10  0x81914000    1  run_on_exit_loop
0x82a08a00    170  waiting   sem       810           -  10  0x81a0b000    1  invalidate_loop
0x82a085b0    171  waiting   cvar      0x828670f8    -  10  0x81703000  171  launch_daemon
0x82a0c210    172  waiting   sem       829           -  10  0x81a52000  171  worker
0x82a08160    173  waiting   cvar      0x82825cd0    -  10  0x81707000  173  app_server
0x82a07d10    174  waiting   cvar      0x82825910    -  10  0x8170b000  174  cddb_daemon
0x82a078c0    175  waiting   cvar      0xd394adb0    -  10  0x8170f000  175  debug_server
0x82a07020    177  waiting   cvar      0xd394abd0    -  10  0x81968000  177  midi_server
0x82a07470    178  waiting   cvar      0xd394a630    -  10  0x81964000  178  mount_server
0x82a06bd0    179  waiting   cvar      0xd394a3b0    -  10  0x8196c000  179  net_server
0x82a06780    180  waiting   cvar      0xd394a090    -  10  0x81970000  180  notification_server
0x82a06330    181  waiting   cvar      0xd394fd70    -  10  0x81985000  181  package_daemon
0x82a05ee0    182  waiting   cvar      0xd394faf0    -  10  0x81989000  182  power_daemon
0x82a05a90    183  waiting   cvar      0xd394f730    -  10  0x8198d000  183  print_server
0x82a05640    184  waiting   cvar      0xd394f4b0    -  10  0x81991000  184  registrar
0x82a051f0    185  waiting   cvar      0xd394f230    -  10  0x819db000  185  syslog_daemon

kdebug> threads 171
thread         id  state     wait for  object      cpu pri  stack      team  name
0x82a085b0    171  waiting   cvar      0x828670f8    -  10  0x81703000  171  launch_daemon
0x82a0c210    172  waiting   sem       829           -  10  0x81a52000  171  worker

kdebug> bt 171
stack trace for thread 171 "launch_daemon"
    kernel stack: 0x81703000 to 0x81707000
      user stack: 0x7088e000 to 0x7188e000
frame               caller     <image>:function + offset
 0 81706cc4 (+ 224) 80097597   <kernel_x86> reschedule(int32: 6) + 0x1007
 1 81706da4 (+  48) 80097641   <kernel_x86> scheduler_reschedule + 0x61
 2 81706dd4 (+  96) 80089136   <kernel_x86> thread_block_with_timeout + 0x1ae
 3 81706e34 (+  64) 800568d7   <kernel_x86> ConditionVariableEntry<0x81706ea4>::Wait(uint32: 0x11 (17), int64: 9223372036854775807) + 0x11f
 4 81706e74 (+  80) 8006c5c2   <kernel_x86> _get_port_message_info_etc + 0x14a
 5 81706ec4 (+  80) 8006c469   <kernel_x86> port_buffer_size_etc + 0x25
 6 81706f14 (+  48) 8006d8e9   <kernel_x86> _user_port_buffer_size_etc + 0x91
 7 81706f44 (+ 100) 80139c1f   <kernel_x86> handle_syscall + 0xdc
user iframe at 0x81706fa8 (end = 0x81707000)
 eax 0xd8          ebx 0x1babad8      ecx 0x7188dc7c  edx 0x614eb114
 esi 0xffffffff    edi 0x7fffffff     ebp 0x7188dca8  esp 0x81706fdc
 eip 0x614eb114 eflags 0x3202    user esp 0x7188dc7c
 vector: 0x63, error code: 0x0
 8 81706fa8 (+   0) 614eb114   <commpage> commpage_syscall + 0x04
 9 7188dca8 (+  48) 00ba73be   <libbe.so> BLooper<0x18205b48>::ReadRawFromPort(0x7188dd04, int64: 9223372036854775807) + 0x2e
10 7188dcd8 (+  48) 00ba744a   <libbe.so> BLooper<0x18205b48>::ReadMessageFromPort(int64: 9223372036854775807) + 0x2a
11 7188dd08 (+  48) 00ba6bff   <libbe.so> BLooper<0x18205b48>::MessageFromPort(int64: 9223372036854775807) + 0x27
12 7188dd38 (+  64) 00ba75d7   <libbe.so> BLooper<0x18205b48>::task_looper(0x7188dd95) + 0x6f
13 7188dd78 (+  64) 00b9a89d   <libbe.so> BApplication<0x18205b48>::Run(0x0) + 0x75
14 7188ddb8 (+  80) 01448f68   <_APP_> main + 0xf4
15 7188de08 (+  48) 01443893   <_APP_> _start + 0x5b
16 7188de38 (+  48) 01209b72   </boot/system/runtime_loader@0x011fa000> <unknown> + 0xfb72
17 7188de68 (+   0) 614eb250   <commpage> commpage_thread_exit + 0x00

kdebug> bt 172
stack trace for thread 172 "worker"
    kernel stack: 0x81a52000 to 0x81a56000
      user stack: 0x70668000 to 0x706a8000
frame               caller     <image>:function + offset
 0 81a55d34 (+ 224) 80097597   <kernel_x86> reschedule(int32: 6) + 0x1007
 1 81a55e14 (+  48) 80097641   <kernel_x86> scheduler_reschedule + 0x61
 2 81a55e44 (+  96) 80088f4e   <kernel_x86> thread_block + 0x10a
 3 81a55ea4 (+  96) 8006fd68   <kernel_x86> switch_sem_etc + 0x400
 4 81a55f04 (+  64) 80070b8e   <kernel_x86> _user_acquire_sem_etc + 0x9a
 5 81a55f44 (+ 100) 80139c1f   <kernel_x86> handle_syscall + 0xdc
user iframe at 0x81a55fa8 (end = 0x81a56000)
 eax 0x11          ebx 0x1babad8      ecx 0x706a7aec  edx 0x614eb114
 esi 0xffffffff    edi 0x7fffffff     ebp 0x706a7b28  esp 0x81a55fdc
 eip 0x614eb114 eflags 0x3202    user esp 0x706a7aec
 vector: 0x63, error code: 0x0
 6 81a55fa8 (+   0) 614eb114   <commpage> commpage_syscall + 0x04
 7 706a7b28 (+  96) 00d14c1f   <libbe.so> BSupportKit::BPrivate::JobQueue<0x18205ca0>::Pop(int64: 9223372036854775807, false, BSupportKit::BJob*: 0x706a7bc4) + 0x4ff
 8 706a7b88 (+  64) 014542b1   <_APP_> Worker<0x181eeb10>::Process(0x0) + 0x45
 9 706a7bc8 (+  48) 01454392   <_APP_> Worker<0x181eeb10>::_Process(NULL) + 0x2a
10 706a7bf8 (+  48) 01b01d3f   <libroot.so> _get_next_team_info (nearest) + 0x5f
11 706a7c28 (+   0) 614eb250   <commpage> commpage_thread_exit + 0x00

kdebug> threads 184
thread         id  state     wait for  object      cpu pri  stack      team  name
0x82a05640    184  waiting   cvar      0xd394f4b0    -  10  0x81991000  184  registrar

kdebug> bt 184
stack trace for thread 184 "registrar"
    kernel stack: 0x81991000 to 0x81995000
      user stack: 0x71c2a000 to 0x72c2a000
frame               caller     <image>:function + offset
 0 81994cc4 (+ 224) 80097597   <kernel_x86> reschedule(int32: 6) + 0x1007
 1 81994da4 (+  48) 80097641   <kernel_x86> scheduler_reschedule + 0x61
 2 81994dd4 (+  96) 80089136   <kernel_x86> thread_block_with_timeout + 0x1ae
 3 81994e34 (+  64) 800568d7   <kernel_x86> ConditionVariableEntry<0x81994ea4>::Wait(uint32: 0x11 (17), int64: 9223372036854775807) + 0x11f
 4 81994e74 (+  80) 8006c5c2   <kernel_x86> _get_port_message_info_etc + 0x14a
 5 81994ec4 (+  80) 8006c469   <kernel_x86> port_buffer_size_etc + 0x25
 6 81994f14 (+  48) 8006d8e9   <kernel_x86> _user_port_buffer_size_etc + 0x91
 7 81994f44 (+ 100) 80139c1f   <kernel_x86> handle_syscall + 0xdc
user iframe at 0x81994fa8 (end = 0x81995000)
 eax 0xd8          ebx 0x1e4cad8      ecx 0x72c28d0c  edx 0x620db114
 esi 0xffffffff    edi 0x7fffffff     ebp 0x72c28d38  esp 0x81994fdc
 eip 0x620db114 eflags 0x3202    user esp 0x72c28d0c
 vector: 0x63, error code: 0x0
 8 81994fa8 (+   0) 620db114   <commpage> commpage_syscall + 0x04
 9 72c28d38 (+  48) 00ff1a0b   <libbe.so> __cl__Q38BPrivate11BLooperList12FindPortPredRQ38BPrivate11BLooperList10LooperData (nearest) + 0x13b
10 72c28d68 (+ 128) 00ff5c02   <libbe.so> BMessage<0x72c28fb0>::_SendMessage(BMessage: 0xb, int32: 171, int32: -2, int32: 1925353320, BMessage*: 0xffffffff, int64: -2147483649, int64: 8243666964875051007) + 0x17a
11 72c28de8 (+  96) 00ffdc29   <libbe.so> BMessenger<0x180f6a60>::SendMessage(BMessenger: 0x72c28fb0, BMessage*: 0x72c28f68, BMessage*: 0xffffffff, int64: -2147483649, int64: 72212189238263807) + 0x61
12 72c28e48 (+  64) 01008d2e   <libbe.so> BRoster::Private<0x72c28ed4>::SendTo(BMessage*: 0x72c28fb0) + 0xa2
13 72c28e88 (+ 368) 00fe792a   <libbe.so> __10BClipboardPCcb + 0x122
14 72c28ff8 (+1104) 01296f12   <_APP_> main + 0x42
15 72c29448 (+  48) 01277cd3   <_APP_> _start + 0x5b
16 72c29478 (+  48) 0174eb72   </boot/system/runtime_loader@0x0173f000> <unknown> + 0xfb72
17 72c294a8 (+   0) 620db250   <commpage> commpage_thread_exit + 0x00
kdebug>

comment:21 by mmlr, 9 years ago

Resolution: fixed
Status: newclosed

Fixed in hrev49583.

comment:22 by michaelvoliveira, 9 years ago

Thanks mmlr!!!

Finally Haiku boots successful in my machine after 2 years, and with Wi-Fi enabled!!!

Now I can get back to work in Haikuports recipes.

Note: See TracTickets for help on using tickets.