Problems with kvm virtualization on Intel CPU with CentOS

[Ehud Karni]

Hello all,

I've been using virtualization with Xen for quite a while.
2 weeks ago I decided give `kvm' a try, but yesterday I went back to
Xen, after kvm caused panic and misbehaved.

The CPU is Intel Xeon E5335 @ 2.00GHz 4 cores.
The OS is CentOS 5.3/4 2.6.18-164.el5 (for kvm use),
                       2.6.18-164.el5xen (for xen use).

The problems with the kvm started when I tried to install the guest.
The guest panicked immediately after I pressed enter on the 1st screen.
I overcome this problem by adding the `nmi_watchdog=0' kernel parameter
when doing the guest installation.

But than I had 3 problems on the host: It lost its network or part of
it (e.g. NFS mount) occasionally, sometimes it rebooted itself and
sometimes it panicked. It also filled the log with BUG alerts (example
is bellow).

So I added the `nmi_watchdog=0' kernel parameter to the host too.
It made thing better (no reboot or panic) but it still lost the network,
and the log filled with the same BUG reports.

I changed to the Xen kernel (dropping the nmi_watchdog=0), things
quited down and there are no errors.

BTW, I found that I can still run the guests with `qemu-kvm' (not the
qemu-dm that come with Xen) and control them with virt-manager through
the qemu connection (qemu:///system) even though the kvm and kvm-intel
modules are not loaded.

Ehud.


The BUG allert in the system log:

Oct 10 10:00:11 cvhlnx kernel: BUG: soft lockup - CPU#1 stuck for 10s! [qemu-kvm:12659]
Oct 10 10:00:11 cvhlnx kernel: CPU 1:
Oct 10 10:00:11 cvhlnx kernel: Modules linked in: loop tun ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables nfsd exportfs auth_rpcgss nfs fscache nfs_acl lockd sunrpc bridge iscsi_tcp libiscsi_tcp ib_iser libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi ib_srp ib_sdp ib_ipoib ipoib_helper ipv6 xfrm_nalgo crypto_api rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa ib_mad ib_core ext4 jbd2 crc16 dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi acpi_memhotplug ac lp kvm_intel(U) kvm(U) i2c_i801 i2c_core parport_pc ide_cd tg3 sg pcspkr cdrom parport i5000_edac edac_mc serio_raw dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ata_piix libata shpchp aacraid sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Oct 10 10:00:11 cvhlnx kernel: Pid: 12659, comm: qemu-kvm Tainted: G      2.6.18-164.el5 #1
Oct 10 10:00:11 cvhlnx kernel: RIP: 0010:[<ffffffff8827f613>]  [<ffffffff8827f613>] :kvm:kvm_flush_remote_tlbs+0xfb/0x109
Oct 10 10:00:11 cvhlnx kernel: RSP: 0018:ffff81014f567878  EFLAGS: 00000297
Oct 10 10:00:11 cvhlnx kernel: RAX: 0000000000000000 RBX: ffff81014f5678a8 RCX: 0000000000000000
Oct 10 10:00:11 cvhlnx kernel: RDX: 000000000003e0a3 RSI: 00000000000000ff RDI: 00000000000000c0
Oct 10 10:00:11 cvhlnx kernel: RBP: ffffffff8827e000 R08: 0000000000000004 R09: 000000000000003c
Oct 10 10:00:11 cvhlnx kernel: R10: ffff81014f567888 R11: ffffffff8828fba0 R12: 0000000000000000
Oct 10 10:00:11 cvhlnx kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 000000000000000d
Oct 10 10:00:11 cvhlnx kernel: FS:  0000000042577940(0000) GS:ffff810107f107c0(0000) knlGS:00000000f7fc46c0
Oct 10 10:00:11 cvhlnx kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Oct 10 10:00:11 cvhlnx kernel: CR2: 00000000080dd1a0 CR3: 00000001332da000 CR4: 00000000000026e0
Oct 10 10:00:11 cvhlnx kernel:
Oct 10 10:00:11 cvhlnx kernel: Call Trace:
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8827f602>] :kvm:kvm_flush_remote_tlbs+0xea/0x109
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff882833d0>] :kvm:kvm_mmu_pte_write+0x1b0/0x295
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff88280f9e>] :kvm:emulator_write_emulated_onepage+0xd4/0x171
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8828740f>] :kvm:x86_emulate_memop+0x2bd1/0x3e6a
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff88284402>] :kvm:paging64_page_fault+0x102/0x260
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff800114ee>] do_wp_page+0x821/0x918
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8829a3be>] :kvm_intel:vmcs_readl+0x17/0x1c
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8828002f>] :kvm:emulate_instruction+0xf6/0x286
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8829bd6f>] :kvm_intel:handle_exception+0x138/0x21b
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8829cba3>] :kvm_intel:vmx_vcpu_run+0x694/0x775
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff88288d1a>] :kvm:preempt_notifier_register+0x53/0x79
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff88280384>] :kvm:kvm_vcpu_ioctl+0x1c5/0xd0b
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8008a29b>] __wake_up_common+0x3e/0x68
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8002e2f2>] __wake_up+0x38/0x4f
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8005121d>] wake_futex+0x45/0x4f
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff800a1f9d>] futex_wake+0x91/0xc7
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8003e2b0>] do_futex+0x282/0xc3f
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff80008d84>] __handle_mm_fault+0x5f2/0xf99
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8014d83d>] __next_cpu+0x19/0x28
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff80097baf>] __dequeue_signal+0x12d/0x193
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8008be71>] default_wake_function+0x0/0xe
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff80097667>] recalc_sigpending+0xe/0x25
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff800420a5>] do_ioctl+0x21/0x6b
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff800302ce>] vfs_ioctl+0x457/0x4b9
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff800a2d1a>] sys_futex+0x10d/0x12d
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8004c766>] sys_ioctl+0x59/0x78
Oct 10 10:00:11 cvhlnx kernel:  [<ffffffff8005d116>] system_call+0x7e/0x83
Oct 10 10:00:11 cvhlnx kernel:


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry