Strace: mangled arguments

[nephele]

Observed when trying to debug webkit issues.
Info: #16898
hrev 55400

Testcase:

#include <syscalls.h>
#include <stdbool.h>

int main() {
	void* address = NULL;
	_kern_map_file("libWebKitLegacy.so.1 mmap area", &address, 0x6, 0x100800000, 0x3, 0x1, true, 0xffffffff, 0x0);
	_kern_set_memory_protection(address, 0xff800000, 0x0);
}

Which results in:

~ strace ./a.out                                                                                                            
[ 18378] image_relocated(0x35296) (53 us)
[ 18378] set_area_protection(0x13d8d4, 0x5) = 0x0 No error (2 us)
[ 18378] set_area_protection(0x13d8d6, 0x5) = 0x0 No error (3 us)
[ 18378] set_area_protection(0x13d8d9, 0x5) = 0x0 No error (3 us)
[ 18378] get_system_info(0x7fd5ce695710) = 0x0 No error (3 us)
[ 18378] get_system_info(0x7fd5ce695520) = 0x0 No error (0 us)
[ 18378] reserve_address_range([0x116e4751b000], 0x7, 0x1000000000) = 0x0 No error (3 us)
[ 18378] create_area("heap", 0x20295f2fca0, 0x1, 0x40000, 0x0, 0x103) = 0x13d8dc (8 us)
[ 18378] resize_area(0x13d8dc, 0x50000) = 0x0 No error (3 us)
[ 18378] resize_area(0x13d8dc, 0x70000) = 0x0 No error (3 us)
[ 18378] open(0xffffffff, "/dev/random", 0x0, 0x0) = 0x3 (7 us)
[ 18378] read(0x3, 0xffffffffffffffff, 0x20295f0b808, 0x8) = 0x8 (2 us)
[ 18378] close(0x3) = 0x0 No error (4 us)
[ 18378] resize_area(0x13d8dc, 0x90000) = 0x0 No error (4 us)
[ 18378] map_file("libWebKitLegacy.so.1 mmap area", 0x7fd5ce695a08, 0x6, 0x100800000, 0x3, 0x1, true, 0x0, 0x0) = 0x13d8dd (7 us)
[ 18378] set_memory_protection(0xf39400e000, 0xff800000, 0x0) = 0x80000000 Out of memory (2 us)
[ 18378] exit_team(0x0) (3 us)

The interesting part here is: 0xffffffff, while the trace sais 0x0. It's possible the last two args here for map_file may be swapped aswell, in the original trace it showed as 0xfffffffff for the last arg, but it seems to has to be the second to last arg in order to actually work.

_kern_map_file("libWebKitLegacy.so.1 mmap area", &address, 0x6, 0x100800000, 0x3, 0x1, true, 0xffffffff, 0x0);
[ 18378] map_file("libWebKitLegacy.so.1 mmap area", 0x7fd5ce695a08, 0x6, 0x100800000, 0x3, 0x1, true, 0x0, 0x0) = 0x13d8dd (7 us)

[korli]

Should be x86_64 specific, there map_file passes some arguments on the stack.

[waddlesplash]

Indeed 0xffffffff is the 8th argument, so it should still be passed in a register, while the 9th will be on the stack.