Opened 5 months ago

Last modified 7 days ago

#14082 new bug

PAE paging leads to random reboots/freezing using AMD-FX 6300 6-core processor

Reported by: nzimmermann Owned by: nobody
Priority: normal Milestone: Unscheduled
Component: System/Kernel Version: R1/Development
Keywords: vm Cc:
Blocked By: Blocking: #10279
Has a Patch: no Platform: x86

Description

My system consist of a AMD-FX 6300 6-core machine with 16GB DDR3 RAM and Haiku is installed os primary OS. The machine randomly reboots and/or freezes (this is more seldom) after a few minutes, when the system is under heavy load (compiling a large software project, using make -j 6).

I replaced the motherboard, CPU, RAM, power-supply etc, but things kept unstable. Finally, after playing with the safe mode settings I found a work-around: Enabling 4gb_memory_limit. (This was a suggestion from korli on ticket #10279).

In src/system/kernel/arch/x86/arch_vm_translation_map.cpp a decision is made based on this setting. PAE paging is disabled, and 32 bit paging is enabled. The paging method is switched from X86PagingMethodPAE to X86PagingMethod32Bit.

Using the latter paging method, my system is stable: no random reboots, no random binary crashes, nothing. I'm confident that the actual memory limit of 4 GB is not curing the problem, as I can reproduce the system reboots also when using only one RAM stick with 4 GB alone --> it must be the paging.

NOTE: Other people reported that the 4gb_memory_limit does not help with the random binary crashes (as stated on ticket #10279, by kallisti5), but in my case it does make the difference. Apparently in earlier days, the 4gb_memory_limit did NOT deactivate PAE paging, but nowadays it does.

Attachments (1)

0000-Test-Apply-Microcode-Update.diff (44.4 KB) - added by nzimmermann 4 months ago.
WIP patch implementing microcode updates

Download all attachments as: .zip

Change History (34)

comment:1 Changed 5 months ago by korli

One big difference between 32bit and PAE is the size of page_directory_entry and pae_page_table_entry (32 vs 64 bits). Now, these entries on x86 are not updated atomically, which might or might not be an issue (I heard that PAE usually requires cmpxchg64). It's only a hunch.

comment:2 in reply to:  1 ; Changed 5 months ago by nzimmermann

Replying to korli:

One big difference between 32bit and PAE is the size of page_directory_entry and pae_page_table_entry (32 vs 64 bits). Now, these entries on x86 are not updated atomically, which might or might not be an issue (I heard that PAE usually requires cmpxchg64). It's only a hunch.

Thanks korli for commenting. Forgive my ignorance, this topic is new to me, but Isn't setting page table entries done atomically?

/*static*/ inline pae_page_table_entry
X86PagingMethodPAE::SetPageTableEntry(pae_page_table_entry* entry,
        pae_page_table_entry newEntry)
{
        return atomic_get_and_set64((int64*)entry, newEntry);
}

comment:3 in reply to:  2 ; Changed 5 months ago by korli

Replying to Nikolas Zimmermann:

Thanks korli for commenting. Forgive my ignorance, this topic is new to me, but Isn't setting page table entries done atomically?

/*static*/ inline pae_page_table_entry
X86PagingMethodPAE::SetPageTableEntry(pae_page_table_entry* entry,
        pae_page_table_entry newEntry)
{
        return atomic_get_and_set64((int64*)entry, newEntry);
}

This one is used only when unmapping. For the mapping, see here

/*static*/ void
X86PagingMethodPAE::PutPageTableInPageDir(pae_page_directory_entry* entry,
	phys_addr_t physicalTable, uint32 attributes)
{
	*entry = (physicalTable & X86_PAE_PDE_ADDRESS_MASK)
		| X86_PAE_PDE_PRESENT
		| X86_PAE_PDE_WRITABLE
		| X86_PAE_PDE_USER;
		// TODO: We ignore the attributes of the page table -- for compatibility
		// with BeOS we allow having user accessible areas in the kernel address
		// space. This is currently being used by some drivers, mainly for the
		// frame buffer. Our current real time data implementation makes use of
		// this fact, too.
		// We might want to get rid of this possibility one day, especially if
		// we intend to port it to a platform that does not support this.
}

It's used here Then compare with 64bit

comment:4 Changed 5 months ago by korli

Cc: ingo_weinhold@… axeld@… kallisti5@… removed

comment:5 Changed 5 months ago by nzimmermann

NOTE: The random binary crashes are still present, even though the likelihood is decreased when disabling PAE paging. The link to bug #10279 should be removed.

Last edited 5 months ago by nzimmermann (previous) (diff)

comment:6 in reply to:  3 Changed 5 months ago by nzimmermann

Replying to korli:

This one is used only when unmapping. For the mapping, see here

Okay, I'll give it a try, thanks for the suggestions.

comment:7 Changed 5 months ago by kallisti5

What's odd is this issue seems completely limited to AMD Bulldozer systems (MBR/BIOS booted)

I upgraded my desktop to Ryzen, and the few times I've successfully booted with EFI I haven't seen the issue. *however*... MBR/BIOS boot doesn't work on my Ryzen (#13370), so maybe related?

comment:8 Changed 5 months ago by korli

My suggestions as a patch https://review.haiku-os.org/#/c/120/

comment:9 in reply to:  8 Changed 5 months ago by nzimmermann

Replying to korli:

My suggestions as a patch https://review.haiku-os.org/#/c/120/

Thanks korli, you were quicker than me :-(

My attempt was to simply call atomic_get_and_set64 in PutPageTableInPageDir(), but your cleanup is indeed nicer. Unfortunately I couldn't test it yet - due to #13980, which currently makes it impossible for me to compile Haiku.

I'm trying waddlesplashs attempt to fix it, together with your patch - hopefully I have a new build available in the next few hours, and can confirm whether the PAE problem is fixed or not.

comment:10 Changed 5 months ago by nzimmermann

New kernel is compiled, and running, let's see if I can still reproduce problems with PAE enabled.

comment:11 in reply to:  10 Changed 5 months ago by nzimmermann

Replying to Nikolas Zimmermann:

New kernel is compiled, and running, let's see if I can still reproduce problems with PAE enabled.

Unfortunately it crashed again, in the same way, during make -j6. I've captured the syslog using a serial cable, but it's not informative I fear:

...
bfs: bfs_rename:1145: No such file or directory
Last message repeated 8 times.
slab memory manager: created area 0x93801000 (124296)
slab memory manager: created area 0x94001000 (124297)
bfs: bfs_rename:1145: No such file or directory
hda: Unsolicited response: 00000080/00000010
hda: sensed pin widget 27, 1
hda: Unsolicited response: 00000000/00000010
hda: sensed pin widget 27, 0
slab memory manager: created area 0x94801000 (167682)
slab memory manager: created area 0x95001000 (1169484)
hda: Unsolicited response: 00000080/00000010
hda: sensed pin widget 27, 1
hda: Unsolicited response: 00000000/00000010
hda: sensed pin widget 27, 0
slab memory manager: created area 0x95801000 (1365358)

comment:12 Changed 5 months ago by korli

Thanks for the test, it would have be surprising given the changes involved. Is it with USB booting?

comment:13 in reply to:  12 Changed 5 months ago by nzimmermann

Replying to korli:

Thanks for the test, it would have be surprising given the changes involved. Is it with USB booting?

No, Haiku is installed on a SATA HDD on my machine.

comment:14 Changed 5 months ago by nzimmermann

I've further analyzed this, and checked all locks / CPU pinning are done correctly, by comparing to the 64bit implementation. There is only one really interresting difference: X86VMTranslationMapPAE::QueryInterrupt(). In the 64bit paging implementation, QueryInterrupt() simply calls Query() wheres in the PAE code a specific implementation exists. The Query() code for both 64bit and PAE, enforce thread cpu pinning, but the QueryInterrupt() code for PAE doesn't do that.

As I said before, I'm not expert in this area, and can't judge whether this is problematic or not, but just wanted to mention this difference. I've observed that I always see "hda: Unsolicited response" in the syslog before the machine crashes, and that message comes from the HDA audio driver IRQ handling.

What do the experts think? Could a missing ThreadCPUPinner in the QueryInterrupt() function cause such problems?

comment:15 Changed 5 months ago by korli

No idea about QueryInterrupt(). About the HDA audio driver, you could blacklist it to check whether it could be a source of problems. This can actually be done with a few drivers. A warning though, audio or network drivers run continuously and tend to exercise the system, that doesn't mean that they are at fault (like for random crashes).

comment:16 Changed 5 months ago by nzimmermann

My QueryInterrupt changes didn't help - I got on the wrong track, as also disabling HDA driver doesn't help. Still random hangs w/o any indication in the syslog what went wrong.

I tried to enable the PAE tracing, but it takes hours to boot into haiku with that. I wish I could only enable it after a certain point in time.

comment:17 Changed 5 months ago by nzimmermann

I've read through all related closed and opened PAE tickets, and came across the memeater tool:

Using several memeater processes at the same time each allocating 1000MB 100 times, I can trigger following page fault: (Log obtained from minicom on a second machine via serial cable)

[2018-04-18 10:32:07] vm_soft_fault: va 0x1b900000 not covered by area in address space
[2018-04-18 10:32:07] vm_page_fault: vm_soft_fault returned error 'Bad address' on fault at 0x1b900100, ip 0x801470b3, write 1, user 0, thref
[2018-04-18 10:32:07] PANIC: vm_page_fault: unhandled page fault in kernel space at 0x1b900100, ip 0x801470b3
[2018-04-18 10:32:07] 
[2018-04-18 10:32:07] Welcome to Kernel Debugging Land...
[2018-04-18 10:32:07] Thread 831 "memeater.c" running on CPU 4
[2018-04-18 10:32:07] stack trace for thread 831 "memeater.c"
[2018-04-18 10:32:07]     kernel stack: 0x822c9000 to 0x822cd000
[2018-04-18 10:32:07]       user stack: 0x70363000 to 0x71363000
[2018-04-18 10:32:07] frame               caller     <image>:function + offset
[2018-04-18 10:32:07]  0 822ccbc4 (+  32) 8014bfde   <kernel_x86> arch_debug_stack_trace + 0x12
[2018-04-18 10:32:07]  1 822ccbe4 (+  16) 800a9183   <kernel_x86> stack_trace_trampoline(NULL) + 0x0b
[2018-04-18 10:32:07]  2 822ccbf4 (+  12) 8013db72   <kernel_x86> arch_debug_call_with_fault_handler + 0x1b
[2018-04-18 10:32:07]  3 822ccc00 (+  48) 800aac0c   <kernel_x86> debug_call_with_fault_handler + 0x60
[2018-04-18 10:32:07]  4 822ccc30 (+  64) 800a9397   <kernel_x86> kernel_debugger_loop(0x801904b7 "PANIC: ", 0x801a71a0 "vm_page_fault: unhax
[2018-04-18 10:32:07] ", 0x822cccdc "", int32: 4) + 0x20f
[2018-04-18 10:32:07]  5 822ccc70 (+  48) 800a973b   <kernel_x86> kernel_debugger_internal(0x801904b7 "PANIC: ", 0x801a71a0 "vm_page_fault: x
[2018-04-18 10:32:07] ", 0x822cccdc "", int32: 4) + 0x77
[2018-04-18 10:32:07]  6 822ccca0 (+  48) 800aaf8e   <kernel_x86> panic + 0x3a
[2018-04-18 10:32:07]  7 822cccd0 (+ 144) 80122ac9   <kernel_x86> vm_page_fault + 0x13d
[2018-04-18 10:32:07]  8 822ccd60 (+  80) 8014d850   <kernel_x86> x86_page_fault_exception + 0x1ec
[2018-04-18 10:32:07]  9 822ccdb0 (+  12) 8014046c   <kernel_x86> int_bottom + 0x3c
[2018-04-18 10:32:07] kernel iframe at 0x822ccdbc (end = 0x822cce0c)
[2018-04-18 10:32:07]  eax 0x4060008     ebx 0x88bd1afc     ecx 0x1b900100  edx 0xf1759018
[2018-04-18 10:32:07]  esi 0x80000001    edi 0x4b1b5000     ebp 0x822cce54  esp 0x822ccdf0
[2018-04-18 10:32:07]  eip 0x801470b3 eflags 0x10206   
[2018-04-18 10:32:07]  vector: 0xe, error code: 0x2
[2018-04-18 10:32:07] 10 822ccdbc (+ 152) 801470b3   <kernel_x86> X86VMTranslationMapPAE<0xdff84818>::UnmapArea(VMArea*: 0xdfc67270, false) 7
[2018-04-18 10:32:07] 11 822cce54 (+  64) 8011f2db   <kernel_x86> delete_area(VMAddressSpace*: 0xdffad360, VMArea*: 0xdfc67270, false) + 0xff
[2018-04-18 10:32:07] 12 822cce94 (+ 144) 8011f5cf   <kernel_x86> vm_delete_area + 0x203
[2018-04-18 10:32:07] 13 822ccf24 (+  32) 80126d8a   <kernel_x86> _user_delete_area + 0x1e
[2018-04-18 10:32:07] 14 822ccf44 (+ 100) 8014066f   <kernel_x86> handle_syscall + 0xdc
[2018-04-18 10:32:07] user iframe at 0x822ccfa8 (end = 0x822cd000)
[2018-04-18 10:32:07]  eax 0xc1          ebx 0x115d138      ecx 0x713622ac  edx 0x620b7114
[2018-04-18 10:32:07]  esi 0x7136353c    edi 0x7136354c     ebp 0x713622d8  esp 0x822ccfdc
[2018-04-18 10:32:07]  eip 0x620b7114 eflags 0x3202    user esp 0x713622ac
[2018-04-18 10:32:07]  vector: 0x63, error code: 0x0
[2018-04-18 10:32:07] 15 822ccfa8 (+   0) 620b7114   <commpage> commpage_syscall + 0x04
[2018-04-18 10:32:07] 16 713622d8 (+  80) 00788be4   <memeater.c> main + 0x214
[2018-04-18 10:32:07] 17 71362328 (+  48) 007888b7   <memeater.c> _start + 0x5b
[2018-04-18 10:32:07] 18 71362358 (+  64) 00bfb000   </boot/system/runtime_loader@0x00be9000> <unknown> + 0x12000
[2018-04-18 10:32:07] 19 71362398 (+   0) 620b7250   <commpage> commpage_thread_exit + 0x00

Not sure if it's at all related to my random system freezes, but at least there's a hint something is still wrong in the PAE code.

comment:18 Changed 5 months ago by nzimmermann

Some news on this item: The system is completely unstable (random reeboots) with PAE paging enabled, and stable with 32bit paging. By now I've replaced the RAM, played with memory settings in BIOS, but it all doesn't help.

I went ahead and installed a x86_64 Haiku version using 64bit paging, and this is even more unstable. The system randomly reboots after a few minutes, even when not doing heavy compilation work. Copying a file via scp, or browsing with Web+ is already sufficient to trigger a reboot -- as always nothing visible in the syslog with any hint. Sometimes I saw a corrupted imagine on the display, eg. a checker board pattern, or just a plain color, which made me initially think it could be related to the graphics card.

Maybe a stupid question, but still: My machine uses an onboard radeon GPU, and the RAM is shared according to the BIOS settings (e.g. 256MB is mapped for the GPU). Does Haiku know about that? Do we have any means to detect that a certain portion of the RAM is dedicated for the GPU?

comment:19 Changed 5 months ago by pulkomandy

You can check the ram size reported by the os (for example in AboutSystem). it should either reduce the total size there or report some "inaccessible" RAM. If it doesn't, indeed we have a problem.

Last edited 5 months ago by pulkomandy (previous) (diff)

comment:20 in reply to:  19 Changed 4 months ago by nzimmermann

Replying to PulkoMandy:

You can check the ram size reported by the os (for example in AboutSystem). it should either reduce the total size there or report some "inaccessible" RAM. If it doesn't, indeed we have a problem.

Thanks for the suggestion. The inaccessible RAM stays at 1 MiB independent of the frame buffer size I chose in the BIOS, but the actual available RAM reflects the BIOS settings --- also the syslog tells me that the radeon_hd driver found the correct frame buffer size - corresponding to the BIOS settings.

comment:21 Changed 4 months ago by nzimmermann

New theory: I start to believe that this is an intrinsic AMD-FX instability, that can be cured with microcode updates, as the system is stable under Linux - which include ucode update facilities. Haiku lacks the ability to update the CPU microcode.

From https://launchpad.net/debian/+source/amd64-microcode/+changelog:

    + This update fixes important processor bugs that cause data corruption
      or unpredictable system behaviour.  It also fixes a performance issue
      and several issues that cause system lockup.

Some of these apply to AMD-FX 6300, the CPU I'm using - according to the AMD 15h documentation.

comment:22 Changed 4 months ago by korli

Does a BIOS update include these microcode updates?

comment:23 in reply to:  22 Changed 4 months ago by nzimmermann

Replying to korli:

Does a BIOS update include these microcode updates?

Nope, the last BIOS update for my ASUS is from 2014.

Changed 4 months ago by nzimmermann

WIP patch implementing microcode updates

comment:24 in reply to:  22 ; Changed 4 months ago by nzimmermann

Replying to korli:

Does a BIOS update include these microcode updates?

Dear korli,

I've started to port the Linux microcode updating facilities to Haiku, with partial success. According to the syslog the microcode is now properly updated, to the latest version that's available for my AMD Bulldozer (0x15) -- but the crashes that I observe on x86_64 remain.

There are a few possibilities:

  • my limited knowledge of this topic introduced a subtle bug, and the microcode is not correctly updated on all CPUs (e.g. eventually I've not properly protected/isolated the individual CPU updates from each other, not sure if that's actually needed, or whether x86_write_msr, can be safely done as-is
  • I update the ucode too late/early in the booting process
  • Outdated microcode is not the issue for my random-reboot-issue

TODO:

  • Microcode updating only works for AMD 0x15 CPUs, simply because I didn't bother to extend to anything else for now.
  • Not sure how to "package" the ucode blobs so that they're available early in the boot process -- for now I've hexdumped the amd-ucode.bin file, and hardcoded it into the kernel - an ugly workaround for now.

comment:25 Changed 4 months ago by nzimmermann

Syslog excerpt:

2018-05-11 18:05:17 KERN: Welcome to kernel debugger output!
2018-05-11 18:05:17 KERN: Haiku revision: hrev51927-3-g645dc27 [master]
2018-05-11 18:05:17 KERN: CPU 0: type 0 family 15 extended_family 6 model 2 extended_model 0 stepping 0, string 'AuthenticAMD'
2018-05-11 18:05:17 KERN: CPU 0: vendor 'AMD' model name 'AMD FX(tm)-6300 Six-Core Processor             '
2018-05-11 18:05:17 KERN: CPU 0: apic id 0, package 0, core 0, smt 0
2018-05-11 18:05:17 KERN: CPU 0: cache sharing: L1 id 0, L2 id 0, L3 id 0
2018-05-11 18:05:17 KERN: CPU 0: features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clfsh mmx fxsr sse sse2 htt sse3 pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c syscall nx mmxext ffxsr long aperfmperf bmi1 
2018-05-11 18:05:17 KERN: microcode: Obtaining AMD microcode update
2018-05-11 18:05:17 KERN: microcode: Using microcode binary 'amd_fam15h.bin' blob starting at 0xffffffff801aad38, size 7877 bytes.
2018-05-11 18:05:17 KERN: microcode: Successfully extracted AMD microcode firmware: patch_id: 100665423, processor_rev_id 24608
2018-05-11 18:05:17 KERN: microcode: Currently loaded revision 100665378
2018-05-11 18:05:17 KERN: microcode: Successfully patched microcode to revision 100665423

comment:26 in reply to:  24 ; Changed 4 months ago by korli

Replying to Nikolas Zimmermann:

I've started to port the Linux microcode updating facilities to Haiku, with partial success. According to the syslog the microcode is now properly updated, to the latest version that's available for my AMD Bulldozer (0x15) -- but the crashes that I observe on x86_64 remain.

Sorry to read that. Is there maybe a pattern when the crashes don't happen? Did you try to blacklist all your unneeded drivers (for instance USB ehci,xhci)? I know the crashes also exist on Intel.

There are a few possibilities:

  • my limited knowledge of this topic introduced a subtle bug, and the microcode is not correctly updated on all CPUs (e.g. eventually I've not properly protected/isolated the individual CPU updates from each other, not sure if that's actually needed, or whether x86_write_msr, can be safely done as-is

One thing missing is re-reading the CPU features after the microcode update.

  • I update the ucode too late/early in the booting process
  • Outdated microcode is not the issue for my random-reboot-issue

TODO:

  • Microcode updating only works for AMD 0x15 CPUs, simply because I didn't bother to extend to anything else for now.
  • Not sure how to "package" the ucode blobs so that they're available early in the boot process -- for now I've hexdumped the amd-ucode.bin file, and hardcoded it into the kernel - an ugly workaround for now.

I imagine the vendor specific code would be better placed in src/add-ons/kernel/cpu/x86/amd.cpp (for AMD). Better confirm that before going forward with ucode update.

comment:27 in reply to:  26 ; Changed 4 months ago by nzimmermann

Replying to korli:

Replying to Nikolas Zimmermann:

I've started to port the Linux microcode updating facilities to Haiku, with partial success. According to the syslog the microcode is now properly updated, to the latest version that's available for my AMD Bulldozer (0x15) -- but the crashes that I observe on x86_64 remain.

Sorry to read that. Is there maybe a pattern when the crashes don't happen? Did you try to blacklist all your unneeded drivers (for instance USB ehci,xhci)? I know the crashes also exist on Intel.

I already disabled all USB busses, the HDA driver, radeon_hd, etc. doesn't help. Also the 4GB RAM limit doesn't help (whereas on x86-gcc2, this allows me to have a stable system). Disabling SMP helps in both cases to get a stable system.

I'm lost what else I could try...

comment:28 in reply to:  27 ; Changed 4 months ago by nzimmermann

Replying to Nikolas Zimmermann:

I'm lost what else I could try...

I just came across an interesting article: http://blog.stuffedcow.net/2015/08/pagewalk-coherence/ Apparently AMD Bulldozer series is special in the sense that the pagewalk behavior is "non-coherent", compared to most other AMD/Intel CPUs.

Both Intel and AMD manuals state that without invalidation, even if the old entry is not cached in the TLB, the pagewalk may still see the old entry. That is, in the following pseudocode, the second instruction (load) can non-deterministically use either the old mapping or new mapping, and that there must be an invalidation or TLB flush in between to guarantee the new page table entry is visible by the second instruction.

mov [page table], new_mapping
mov eax, [linear address using updated mapping]

I wonder if the Haiku paging code guarantees this. Apparently we only call InvalidatePage(..) when the accessed bit is set.

        if ((oldEntry & X86_64_PTE_ACCESSED) != 0) {
                // Note, that we only need to invalidate the address, if the
                // accessed flags was set, since only then the entry could have been
                // in any TLB.
                InvalidatePage(address);
..

X86VMTranslationMap::InvalidatePage() only records that a page was invalid, and upon the next Flush() the TLB is invalidated, when fInvalidPagesCount > 0.

I'm currently traveling, when I'm back I will try to always call invalidatePage() in any case, just to see if it helps with my issue. Obviously I'm still 'fishing in the blind' - trying to understand what makes AMD Bulldozer special.

comment:29 in reply to:  28 ; Changed 4 months ago by nzimmermann

Replying to Nikolas Zimmermann:

Replying to Nikolas Zimmermann:

I'm lost what else I could try...

I just came across an interesting article: http://blog.stuffedcow.net/2015/08/pagewalk-coherence/ Apparently AMD Bulldozer series is special in the sense that the pagewalk behavior is "non-coherent", compared to most other AMD/Intel CPUs.

@korli, what do you think about this?

comment:30 in reply to:  29 Changed 4 months ago by korli

Replying to Nikolas Zimmermann:

@korli, what do you think about this?

I looked some other implementations, and they don't seem to work differently from ours. This article speaks of page table mapping update. The AMD PDF states that the flag X86_64_PTE_ACCESSED is set when speculation occurs. Interesting information, anyway I suppose you can try whatever would make sense and prove a theory by testing.

comment:31 Changed 2 months ago by waddlesplash

Milestone: R1/beta1Unscheduled

comment:32 Changed 8 days ago by nzimmermann

I found out a super easy way to trigger the reboots: find / | xargs md5sum (repeat < 5 times)

Out of curiosity I went back to old hrevs. To make a long story short: hrev45681 reproducible reboots spontaneously, and hrev45225 does not. Unfortunately there are no nightlies in-between, so I will have to check all patches between 45225 .. 45681 that touch the x86_64 kernel and see if I can find the culprit: fun ahead of me.

comment:33 Changed 7 days ago by waddlesplash

Potentially relevant, from that range: hrev45564, hrev45566, hrev45681, hrev45518 (big), hrev45397 (more use of largepages). (not an exhaustive list.)

As bisecting is probably too difficult here, reverting patches individually may be of more use.

Note: See TracTickets for help on using tickets.