Read beyond allocation in linprog/LayoutOptimizer on tiling windows

[mmlr]

When running the app_server with the guarded heap and tiling two windows the following out of bound access happens as soon as you release the mouse button:

KERN: vm_page_fault: vm_soft_fault returned error 'Permission denied' on fault at 0x4aac000, ip 0x52e00d, write 0, user 1, thread 0x76
KERN: vm_page_fault: thread "event loop" (118) in team "app_server" (72) tried to read address 0x4aac000, ip 0x52e00d ("app_server_seg0ro" +0x12400d)
KERN: 118: DEBUGGER: thread 118 tried accessing address 0x4aac000 which is a guard page (base: 0x4aab9d0, size: 1584, alignment: 1, allocated by thread: 118)
KERN: debug_server: Thread 118 entered the debugger: Debugger call: `thread 118 tried accessing address 0x4aac000 which is a guard page (base: 0x4aab9d0, size: 1584, alignment: 1, allocated by thr'
KERN: stack trace, current PC 0xffff0114  :
KERN:   (0x701841e4)  0x2a7c79  panic__FPCce + 0x45
KERN:   (0x70184614)  0x2a8e25  dump_guarded_heap_page__FPvb + 0x11d
KERN:   (0x70184654)  0x2a7cda  guarded_heap_segfault_handler__FiP11__siginfo_tPv + 0x46
KERN:   (0x70184684)  0xffff0142  
KERN:   (0x701846b4)  0x52e00d  multiply_matrices__FPCPCdT0PPdiii + 0x55
KERN:   (0x7018498c)  0x52ebbf  SetConstraints__15LayoutOptimizerRCt11BObjectList1ZQ217LinearProgramming10Constraintl + 0x2c7
KERN:   (0x70184adc)  0x52e8a7  __15LayoutOptimizerRCt11BObjectList1ZQ217LinearProgramming10Constraintl + 0x5f
KERN:   (0x70184b0c)  0x52d195  Solve__15ActiveSetSolver + 0x2bd
KERN:   (0x70184cbc)  0x52b287  Solve__Q217LinearProgramming10LinearSpec + 0x37
KERN:   (0x70184cec)  0x51bbb4  DoGroupLayout__10WindowArea + 0x1e4
KERN:   (0x70184d4c)  0x523793  DoGroupLayout__9SATWindow + 0x37
KERN:   (0x70184d7c)  0x527345  JoinCandidates__9SATTiling + 0x59
KERN:   (0x70184dbc)  0x523747  JoinCandidates__9SATWindow + 0x2f
KERN:   (0x70184dec)  0x52579f  _StopSAT__12StackAndTile + 0x27
KERN:   (0x70184e1c)  0x5251a2  MouseUp__12StackAndTileP6WindowP8BMessageRC6BPoint + 0x26
KERN:   (0x70184e4c)  0x476cf7  NotifyMouseUp__17DesktopObservableP6WindowP8BMessageRC6BPoint + 0x57
KERN:   (0x70184e7c)  0x46c6c4  Filter__11MouseFilterP8BMessagePP11EventTargetPlT1 + 0x32c
KERN:   (0x70184eec)  0x47e1be  _EventLoop__15EventDispatcher + 0x28e
KERN:   (0x70184f7c)  0x47e837  _event_looper__15EventDispatcherPv + 0x1f
KERN:   (0x70184fac)  0x22c5e7  thread_entry + 0x23

Which indicates that multiply_matrices() called from LayoutOptimizer::SetConstraints() reads beyond one of the allocations. Since I'm not really comfortable with the math that's going on there I figured it'd be better to open a ticket and let someone fix it who actually understands what's happening.

[czeidler]

cool nice catch, think I found the bug but how to enable the guarded heap to test it? thanks

[mmlr]

Replying to czeidler:

cool nice catch, think I found the bug but how to enable the guarded heap to test it? thanks

First you'd need to swap out the heap.cpp vs. the guarded_heap.cpp in the src/system/libroot/posix/malloc_debug/Jamfile and then run the app_server (at least) with libroot_debug.so instead of libroot.so. I did that by prefixing the launch line in the Bootscript with LD_PRELOAD=libroot_debug.so. So instead of:

	if [ -f "/boot/$toLaunch" ]
	then
		"/boot/$toLaunch" $* &
		[ "$toWaitFor" != "" ] && waitfor "$toWaitFor"
		return 1

You get:

	if [ -f "/boot/$toLaunch" ]
	then
		LD_PRELOAD=libroot_debug.so "/boot/$toLaunch" $* &
		[ "$toWaitFor" != "" ] && waitfor "$toWaitFor"
		return 1

[luroh]

Moving S&T related tickets out of R1 milestone (Prop #16).

[luroh]

Moving S&T related tickets back into R1 milestone as S&T is in fact now the default. My apologies.