Monday, June 19, 2017

Installing/running Debian stretsch inside kvm debian jessie

Installing or upgrading to a Debian stretch (Debian 9) edition should be a "no problem" case for most platforms.

Unfortunally running stretch in a KVM/QEMU environment based on Debian jessie isn't "just" working.

We did try to upgrade a existing Debian Jessie VM to stretch with the usual steps.
The upgrade itself works very fine and flawless as known from Debian systems.
But after rebooting the system, we did start getting errors in the guest, most of the time it did not even boot correctly.

In the host logfile we did see these messages in syslog:

kvm: zapping shadow pages for mmio generation wraparound
vcpu0 unhandled rdmsr: 0x34

In the log of the guest, we did see these stack traces:

Jun 19 15:40:18 sv54 kernel: [    0.000000] Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2 (2017-06-12)
Jun 19 15:40:18 sv54 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-3-amd64 root=UUID=fb8d9106-4421-4e85-af6f-8c0561bb2b25 ro quiet
Jun 19 15:40:18 sv54 kernel: [    0.000000] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.000000] WARNING: CPU: 0 PID: 0 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/kernel/fpu/xstate.c:593 fpu__init_system_xstate+0x53b/0x981
Jun 19 15:40:18 sv54 kernel: [    0.000000] XSAVE consistency problem, dumping leaves
Jun 19 15:40:18 sv54 kernel: [    0.000000] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.000000]  0000000000000000 ffffffff87f28634 ffffffff88803e08 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.000000]  ffffffff87c76eae 000000000000000a ffffffff88803e60 0000000000000340
Jun 19 15:40:18 sv54 kernel: [    0.000000]  ffffffff88803e90 ffffffff88803e9c 0000000000000100 ffffffff87c76f2f
Jun 19 15:40:18 sv54 kernel: [    0.000000] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c76f2f>] ? warn_slowpath_fmt+0x5f/0x80
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c72754>] ? xfeature_size+0x5a/0x78
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88947243>] ? fpu__init_system_xstate+0x53b/0x981
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87f586e6>] ? msr_clear_bit+0x36/0xa0
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff889468dc>] ? fpu__init_system+0x208/0x30b
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88942fea>] ? setup_arch+0xb8/0xcc6
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87d7a24e>] ? printk+0x57/0x73
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938120>] ? early_idt_handler_array+0x120/0x120
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938bbf>] ? start_kernel+0xab/0x463
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938120>] ? early_idt_handler_array+0x120/0x120
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938408>] ? x86_64_start_kernel+0x14c/0x170
Jun 19 15:40:18 sv54 kernel: [    0.000000] ---[ end trace 44d08096f31f4f03 ]---


and

Jun 19 15:40:18 sv54 kernel: [    0.387100] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:368 fpu__clear+0x179/0x1b0
Jun 19 15:40:18 sv54 kernel: [    0.387101] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387103] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387104]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387105]  ffffffff87c76eae ffff9ec076158040 ffff9ec076158b40 ffff9ec072329400
Jun 19 15:40:18 sv54 kernel: [    0.387106]  ffffffff88869da0 ffff9ec072354400 ffff9ec076158758 ffffffff87c30fc9
Jun 19 15:40:18 sv54 kernel: [    0.387107] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387110]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.387112]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.387113]  [<ffffffff87c30fc9>] ? fpu__clear+0x179/0x1b0
Jun 19 15:40:18 sv54 kernel: [    0.387114]  [<ffffffff87e0914c>] ? flush_old_exec+0x5bc/0x6b0
Jun 19 15:40:18 sv54 kernel: [    0.387116]  [<ffffffff87e5df52>] ? load_elf_binary+0x3c2/0x1600
Jun 19 15:40:18 sv54 kernel: [    0.387117]  [<ffffffff87e087f0>] ? search_binary_handler+0xa0/0x1c0
Jun 19 15:40:18 sv54 kernel: [    0.387118]  [<ffffffff87e5d734>] ? load_script+0x204/0x230
Jun 19 15:40:18 sv54 kernel: [    0.387118]  [<ffffffff87dfe0da>] ? __check_object_size+0xfa/0x1d8
Jun 19 15:40:18 sv54 kernel: [    0.387119]  [<ffffffff87e093c8>] ? copy_strings.isra.25+0x188/0x450
Jun 19 15:40:18 sv54 kernel: [    0.387120]  [<ffffffff87e087f0>] ? search_binary_handler+0xa0/0x1c0
Jun 19 15:40:18 sv54 kernel: [    0.387121]  [<ffffffff87e09f0a>] ? do_execveat_common.isra.37+0x5aa/0x790
Jun 19 15:40:18 sv54 kernel: [    0.387123]  [<ffffffff881f8d20>] ? rest_init+0x80/0x80
Jun 19 15:40:18 sv54 kernel: [    0.387124]  [<ffffffff87e0a118>] ? do_execve+0x28/0x30
Jun 19 15:40:18 sv54 kernel: [    0.387125]  [<ffffffff881f8d70>] ? kernel_init+0x50/0x100
Jun 19 15:40:18 sv54 kernel: [    0.387126]  [<ffffffff882064f5>] ? ret_from_fork+0x25/0x30
Jun 19 15:40:18 sv54 kernel: [    0.387127] ---[ end trace 44d08096f31f4f07 ]---
Jun 19 15:40:18 sv54 kernel: [    0.387363] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387365] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:353 fpu__copy+0x140/0x190
Jun 19 15:40:18 sv54 kernel: [    0.387366] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387367] CPU: 2 PID: 1 Comm: init Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387368]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387369]  ffffffff87c76eae ffff9ec0723e7ec0 ffff9ec076158b40 ffff9ec0723e7e80
Jun 19 15:40:18 sv54 kernel: [    0.387370]  0000000000000000 0000000000000000 00007f7cfecd79d0 ffffffff87c30a90
Jun 19 15:40:18 sv54 kernel: [    0.387371] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387373]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.387374]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.387375]  [<ffffffff87c30a90>] ? fpu__copy+0x140/0x190
Jun 19 15:40:18 sv54 kernel: [    0.387376]  [<ffffffff87c74490>] ? copy_process.part.33+0x1a0/0x1c00
Jun 19 15:40:18 sv54 kernel: [    0.387377]  [<ffffffff87dfe0da>] ? __check_object_size+0xfa/0x1d8
Jun 19 15:40:18 sv54 kernel: [    0.387379]  [<ffffffff87f56988>] ? strncpy_from_user+0x48/0x160
Jun 19 15:40:18 sv54 kernel: [    0.387379]  [<ffffffff87e0744d>] ? cp_new_stat+0x14d/0x180
Jun 19 15:40:18 sv54 kernel: [    0.387381]  [<ffffffff87c760d3>] ? _do_fork+0xe3/0x3f0
Jun 19 15:40:18 sv54 kernel: [    0.387381]  [<ffffffff87e074b9>] ? SYSC_newstat+0x39/0x60
Jun 19 15:40:18 sv54 kernel: [    0.387383]  [<ffffffff87c03b1c>] ? do_syscall_64+0x7c/0xf0
Jun 19 15:40:18 sv54 kernel: [    0.387384]  [<ffffffff8820632f>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jun 19 15:40:18 sv54 kernel: [    0.387384] ---[ end trace 44d08096f31f4f08 ]---
Jun 19 15:40:18 sv54 kernel: [    0.387420] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387422] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:353 __switch_to+0x66a/0x6c0
Jun 19 15:40:18 sv54 kernel: [    0.387422] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387423] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387425] WARNING: CPU: 3 PID: 0 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:368 __switch_to+0x415/0x6c0
Jun 19 15:40:18 sv54 kernel: [    0.387426] CPU: 2 PID: 1 Comm: init Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387427] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387428]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387430]  ffffffff87c76eae ffff9ec0762401c0 ffff9ec076158040 ffff9ec076240cc0
Jun 19 15:40:18 sv54 kernel: [    0.387431]  ffff9ec0762401c0 0000000000000002 ffff9ec076158a80 ffffffff87c24aaa
Jun 19 15:40:18 sv54 kernel: [    0.387432] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387432] ---[ end trace 44d08096f31f4f09 ]---


After some research we did find out, that there is a bug in the QEMU/KVM packages as provided by jessie stable repositories.

To solve the problem, you have to install more recent QEMU/KVM packages, and the best way to get them for stretch is to add the jessie backports to your sources.list

deb http://ftp.debian.org/debian jessie-backports main

After the usual apt-get update just install the qemu-kvm and related packages in the 2.8xxx version, instead of the 2.1xxxx releases.

After a reboot of your physical server, you can now enjoy debian stretch in your guests