Haiku works very poorly as a guest under the Linux Kernel Virtual Machine QEMU KVM

[Don Quixote]

This is most likely several different bugs in the Linux kernel, KVM or QEMU, but there is some possibility that the problems I am experiencing are in Haiku. In any case, it would be helpful for someone who knows more about Haiku's internals than I do to work with the KVM developers to make this work better.

I was planning to submit each of the following problems as a separate Trac ticket, and will do so upon request. But I figured it would be best to bring this to your attention now rather than wait until I had the gumption to type up so many bug reports.

I've spent five or six days trying to get to the point where I can build Haiku when booted off a Micro Center store brand 16 GB USB Flash drive on my Early 2006 MacBook Pro, model identifier MacBookPro1,1.

Haiku mostly works on my unit, but there are some problems that I hoped to fix and submit patches for, but because of all the problems with virtualization I have experienced, I have yet to get as far as booting my laptop with the latest Subversion.

If I do "svn co" for either the buildtools or the haiku source, the checkout will go extremely slowly. If it were to succeed at all, I expect either codebase would take more than a day to completely check out.

However in several days of trying, I was never able to complete the checkout. Always it would fail in some really bad way that required me to reboot Haiku.

To start with, the files are retrieved very slowly at best. Every few dozen files the checkout pauses for a time ranging from a few seconds to a few minutes. Eventually it halts completely.

At that point the svn process cannot be killed by sending it signals, not even signal 9 (SIGKILL) which cannot be caught or ignored.

I had the same thing happen with rm -rf. The rm process got stuck and could not be killed with signals.

More often, svn complains that it could not read the host's response header, then the svn process terminates. I've restarted it a whole bunch of times but have yet to get the whole buildtools tree to check out while running Haiku under KVM. I never had this kind of problem under VirtualBox or Ubuntu. Most exasperating is that after one of my reboots, a whole bunch of the buildtools' .svn directories contained corrupt data. The BFS filesystem itself was not corrupt, but "svn cleanup haiku/buildtools" complained that many of the sources were not under version control. Rather than try to fix this I would "rm -rf" the entire directory where that source file resided. It was while doing that that I found I could not kill rm with SIGKILL.

I was previously using VirtualBox, which worked far better. However full builds of both the buildtools and Haiku took 4 1/2 hours. While VirtualBox guests can be configured to have more than one CPU, only one host CPU is used. The multicore guest CPU is only emulated; I expect it would actually run slower than configuring the guest with just one core.

I hoped to speed up builds under virtualization by using the Linux Kernel Virtual Machine on Ubuntu 11.04. KVM *can* use more than one host core. I have two cores in my Intel Core Duo (32-bit - NOT Core 2 Duo). I have always allocated two cores to the Haiku guest, then avoided using my laptop when Haiku was doing anything CPU-intensive.

Maybe KVM would work better with just one core allocated to the guest. I will try that later and will report my results.

I had the hope that KVM would work better in Ubuntu 11.10 but it does not.

The video and mouse movement are painfully slow. While virt-manager presents the options of using VNC, Spice and SDL for graphics, only VNC actually works. I don't think it is yet possible to use SDL at all. Spice can be gotten to work in Ubuntu 11.10, but Ubuntu 11.10 is only distributing 64-bit packages. I'd be happy to build Spice from source if someone could tell me whether I can expect it to work in 32-bit, but I've asked around and no one has responded yet.

My current /etc/libvirt/qemu guest configuration file is attached. I've tried a whole bunch of different configurations though.

Mapping my USB key to an IDE drive seems to be the most reliable and have the best throughput. Mapping it to SCSI does not work at all as Haiku does not support the emulated SCSI Host Bus Adapter. I would have expected that SCSI Pass-Through to allow Haiku to directly control the USB key would work best, but it works very poorly. Mapping it as a USB drive causes a 100% repeatible crash during boot.

I have a whole bunch of serial logs if you want them, but these problems have been so reproducible for me I don't expect that you'll need them.

There are several problems that are frequently reported in the serial log:

check sense: Hardware error

/dev/net/rtl81xx/0: media change, media 0x800020 quality 1000 speed 1000000000
/dev/net/rtl81xx/0: media change, media 0x900026 quality 1000 speed 10000000

ps2: strange mouse data, x-delta 127, trying resync
ps2: strange mouse data, x/y overflow, trying resync
ps2: bad mouse data, trying resync

virt-manager does not pass SysRq through to the guest, so I have never been able to use the keyboard to enter KDL. virt-manager does have a menu that sends various key combinations to the guest, but Alt-SysRq-D is not one of the listed key combos, and I cannot find a way to customize the menu options.

I'm planning to try Xen next. I've installed the Ubuntu packages but have not yet figured out how to configure it.

Haiku would work best under both KVM or Xen if it had paravirtualized drivers. This would include Virtio for block storage, QXL video for Spice or Xen video, as well as memory ballooning. I'm interested in virtualization, and would love to implement all these myself, but I am a long way from knowing enough about Haiku's internals to write any real code yet.

[Don Quixote]

Linux Kernel Virtual Machine config file with USB Flash drive on /dev/sdb mapped to IDE drive in guest

[Don Quixote]

I would have thought that my Micro Center store brand USB stick was not of good quality, but I've been using it for several years and have never had any trouble with it. The Gnome Disk Utility under Ubuntu 11.04 reports that it has a read speed of 30 MB/s, write speed of 18 MB/s and average access time of 1.4 msec.

I'm planning to buy a faster stick, either a LaCie Fastkey or a Corsair Flash Voyager GT. The FastKey appears to be the fastest stick on the market, but is extremely expensive. The GT is decently fast at less than half the price of the FastKey.

A faster drive might only help when booting natively off my laptop though.

I would be a much happier camper if someone could give me a clue as to how to access the physical USB device from VirtualBox. VB allowed me to configure the drive that way, but gives me an error when I try to start the guest.

[mmlr]

The two CPU thing might very well be the problem here. At least under QEMU emulating more than one virtual CPU always caused the system to crawl. Since this only happened under QEMU I figured it'd be some problem of the APIC emulation, likely being specific/optimized for emulating other OSes. I never came around to actually investigate it though.

The stick performance shouldn't really be all that much of a problem. Keep in mind though that raw throughput is really not a meaningful number. A stick doing 30MB/s burst reads may very well perform poorly when reading non-contiguous or only small data at a time. In either case caching should hide most of the performance hit there as well.

You mention reproducible crashes but don't provide any more detail. You mean it panics because it doesn't find the boot volume if you "attach it as USB drive"? Is it in that case attached to an emulated USB device? If the KDL doesn't merely concern not finding the boot volume, then please file a new bug report for that and attach the corresponding output.

A total build time of 4 and a half hours for buildtools and Haiku isn't so bad for a single emulated core of that generation IMO. Why exactly are you building the buildtools though?

You can enter KDL via the kernel_debugger command line tool as well BTW.

[Don Quixote]

mmlr, when my untar of buildtools completes I will try using just one core. But the whole reason I tried KVM at all was that I hoped to speed up builds by using both cores in my computer.

KVM allows one to configure how any storage unit appears to the guest. In every case the real sorage unit on the host is my Micro Center USB stick. W

When the guest sees the storage as USB passthrough or IDE, it works but is slow. IDE seems to be faster than USB passthrough. I expect this is because the implementation of USB passthrough in KVM is slow.

When the guest sees the storage as SCSI it cannot find the boot device. This panic occurs very early on.

When the guest sees the storage as a USB disk it panics late in the boot, but before the screen turns blue. This is 100% reproducible. I would have thought this option would work the best. I will file a bug specifically for this. I have a saved serial output log.

There is a cascading series of errors before it tries to enter KDL in this case. The very last message is that it cannot enter the debugger, but the actual problem occurs much earlier. It should be possible to enter KDL early one, set a breakpoint to just before the first complaint, then look at where the problem first begins to occur.

A gzip compressed tarball of yesterday's builtools SVN is 210805 kb. The untar process has been running for about nine hours now. I don't want to stop it until it completes, which I hope will be soon.

However it has been running without grinding completely to a halt as always happened when I tried to check out from Subversion running Haiku. While the filesystem performance is slow under KVM, it does seem to work reliably. Perhaps my worst problem was accessing the network under KVM.

Is there a way to configure the build so that one does not build the buildtools as well? That is not documented on the Haiku build instructions page.

In any case, I think it is helpful to build from buildtools I built myself, just to make sure that the latest buildtools SVN actually still works right.

[anevilyak]

Replying to Don Quixote:

When the guest sees the storage as SCSI it cannot find the boot device. This panic occurs very early on.

We don't currently support any SCSI controllers, so that's not surprising.

Is there a way to configure the build so that one does not build the buildtools as well? That is not documented on the Haiku build instructions page.

configure --build-cross-tools only needs to be run once. After that you just use jam to build/rebuild Haiku itself, which does not attempt to rebuild the build tools.

[Don Quixote]

anevilyak, I have only built while running under Haiku, so I have never done "configure --build-cross-tools", only "configure" then "jam -q @alpha-raw".

Is there a configure option, or is there an entry I could put into my UserBuildConfig so that the build tools aren't built at all when running under Haiku?

Does the alpha-raw target include all the necessary build tools or do I really need to do that first build of the tools myself?

[anevilyak]

If that's the case then you aren't building the build tools at all. The only tools you'll see it build are a handful of things used as part of the build process, not the compiler toolchain.

[bonefish]

One question that comes to mind: Why do you want to punish yourself building Haiku on an emulated Haiku? On your Linux host system Haiku can be built a lot faster than on a Haiku installed on the same machine. And an emulated Haiku is dramatically slower even. When you work on individual libraries or applications, building on an emulated Haiku might make sense, but the more has to be built the better are the turn-around times for building on Linux. For kernel or driver work that doesn't need actual hardware, it's the best option anyway.

Anyway, putting that many different issues into one ticket is really not a good idea. Likely some get overlooked and the whole ticket gets really bloated. Most developers don't like to have to read through a lot of text and filter out the information that pertain to the problem they are interested in.

Regarding the unkillable processes/threads, it's rather easy to see what they are blocking at in the kernel debugger. If you follow the lock holders recursively it's usually not that complicated to identify deadlocks. From your description I'd guess there's something fishy with the network driver and the thread is probably blocking on one of its locking primitives and not being woken up. Please open a new ticket for that and attach at least the stack trace for the thread in question and the output of thread -s <thread ID>.

On a general note, virtualization support (particularly virtio) would be very welcome. Please feel encouraged to work on it. :-)

[korli]

Virtio block devices are supported with revisions hrev45714 and upper.

Would it make sense to close this ticket and open different ones for each remaining issue?

[kallisti5]

i'd say this is resolved. We have a virtio scsi driver and a virtio bus driver. There are tickets open for each virtio driver that needs written for the virtio bus. (below)

#9803: enhancement: Virtio balloon driver (new) #9802: enhancement: Virtio SCSI driver (closed: fixed) #9801: enhancement: Virtio entropy driver (new) #9800: enhancement: Virtio network driver (new)