Wednesday, June 21, 2017

JSVC fails with error 11 after latest linux kernel Upgrades on debian/ubuntu

This morning, after doing some apt-get update/upgrades on various debian systems, we noticed that many of our java services did no longer work.
Since we use it to handle all our tomcat instances on many many servers, the impact is heavy.No single tomcat did run this morning...

Looking into the tomcat logs, we did see this message:
Service killed by signal 11

After some more investigations, it looks like the problem is related to the new kernel version installed in the upgrade process.

Google didn't find much about this, but the more important ones are here:
So the solution for now, is either switch to not using jsvc to start your services, or use a older or a "unstable" kernel.

The kernel causing the problems is
3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u1 (2017-06-18) x86_64
 
With the older one it does work
3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64
 
To switch back to the older kernel just do this, and reboot your system:
apt install linux-image-3.16.0-4-amd64=3.16.43-2  
 
Reverting back to a older kernel is discouraged, since this does not solve the security problem.

Fortunately there is a very simple work arround for it.
When you start jsvc, just specify it to use a larger stack.
For tomcat you can put this in your startup file, so the daemon.sh takes the new options for jsvc.
 
export JSVC_OPTS=-Xss1280k

Thanks to https://community.ubnt.com/t5/UniFi-Wireless/IMPORTANT-Debian-Ubuntu-users-MUST-READ-Updated-06-21/td-p/1968252

Monday, June 19, 2017

Installing/running Debian stretsch inside kvm debian jessie

Installing or upgrading to a Debian stretch (Debian 9) edition should be a "no problem" case for most platforms.

Unfortunally running stretch in a KVM/QEMU environment based on Debian jessie isn't "just" working.

We did try to upgrade a existing Debian Jessie VM to stretch with the usual steps.
The upgrade itself works very fine and flawless as known from Debian systems.
But after rebooting the system, we did start getting errors in the guest, most of the time it did not even boot correctly.

In the host logfile we did see these messages in syslog:

kvm: zapping shadow pages for mmio generation wraparound
vcpu0 unhandled rdmsr: 0x34

In the log of the guest, we did see these stack traces:

Jun 19 15:40:18 sv54 kernel: [    0.000000] Linux version 4.9.0-3-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.30-2 (2017-06-12)
Jun 19 15:40:18 sv54 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-3-amd64 root=UUID=fb8d9106-4421-4e85-af6f-8c0561bb2b25 ro quiet
Jun 19 15:40:18 sv54 kernel: [    0.000000] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.000000] WARNING: CPU: 0 PID: 0 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/kernel/fpu/xstate.c:593 fpu__init_system_xstate+0x53b/0x981
Jun 19 15:40:18 sv54 kernel: [    0.000000] XSAVE consistency problem, dumping leaves
Jun 19 15:40:18 sv54 kernel: [    0.000000] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.000000]  0000000000000000 ffffffff87f28634 ffffffff88803e08 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.000000]  ffffffff87c76eae 000000000000000a ffffffff88803e60 0000000000000340
Jun 19 15:40:18 sv54 kernel: [    0.000000]  ffffffff88803e90 ffffffff88803e9c 0000000000000100 ffffffff87c76f2f
Jun 19 15:40:18 sv54 kernel: [    0.000000] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c76f2f>] ? warn_slowpath_fmt+0x5f/0x80
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87c72754>] ? xfeature_size+0x5a/0x78
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88947243>] ? fpu__init_system_xstate+0x53b/0x981
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87f586e6>] ? msr_clear_bit+0x36/0xa0
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff889468dc>] ? fpu__init_system+0x208/0x30b
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88942fea>] ? setup_arch+0xb8/0xcc6
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff87d7a24e>] ? printk+0x57/0x73
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938120>] ? early_idt_handler_array+0x120/0x120
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938bbf>] ? start_kernel+0xab/0x463
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938120>] ? early_idt_handler_array+0x120/0x120
Jun 19 15:40:18 sv54 kernel: [    0.000000]  [<ffffffff88938408>] ? x86_64_start_kernel+0x14c/0x170
Jun 19 15:40:18 sv54 kernel: [    0.000000] ---[ end trace 44d08096f31f4f03 ]---


and

Jun 19 15:40:18 sv54 kernel: [    0.387100] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:368 fpu__clear+0x179/0x1b0
Jun 19 15:40:18 sv54 kernel: [    0.387101] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387103] CPU: 2 PID: 1 Comm: swapper/0 Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387104]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387105]  ffffffff87c76eae ffff9ec076158040 ffff9ec076158b40 ffff9ec072329400
Jun 19 15:40:18 sv54 kernel: [    0.387106]  ffffffff88869da0 ffff9ec072354400 ffff9ec076158758 ffffffff87c30fc9
Jun 19 15:40:18 sv54 kernel: [    0.387107] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387110]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.387112]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.387113]  [<ffffffff87c30fc9>] ? fpu__clear+0x179/0x1b0
Jun 19 15:40:18 sv54 kernel: [    0.387114]  [<ffffffff87e0914c>] ? flush_old_exec+0x5bc/0x6b0
Jun 19 15:40:18 sv54 kernel: [    0.387116]  [<ffffffff87e5df52>] ? load_elf_binary+0x3c2/0x1600
Jun 19 15:40:18 sv54 kernel: [    0.387117]  [<ffffffff87e087f0>] ? search_binary_handler+0xa0/0x1c0
Jun 19 15:40:18 sv54 kernel: [    0.387118]  [<ffffffff87e5d734>] ? load_script+0x204/0x230
Jun 19 15:40:18 sv54 kernel: [    0.387118]  [<ffffffff87dfe0da>] ? __check_object_size+0xfa/0x1d8
Jun 19 15:40:18 sv54 kernel: [    0.387119]  [<ffffffff87e093c8>] ? copy_strings.isra.25+0x188/0x450
Jun 19 15:40:18 sv54 kernel: [    0.387120]  [<ffffffff87e087f0>] ? search_binary_handler+0xa0/0x1c0
Jun 19 15:40:18 sv54 kernel: [    0.387121]  [<ffffffff87e09f0a>] ? do_execveat_common.isra.37+0x5aa/0x790
Jun 19 15:40:18 sv54 kernel: [    0.387123]  [<ffffffff881f8d20>] ? rest_init+0x80/0x80
Jun 19 15:40:18 sv54 kernel: [    0.387124]  [<ffffffff87e0a118>] ? do_execve+0x28/0x30
Jun 19 15:40:18 sv54 kernel: [    0.387125]  [<ffffffff881f8d70>] ? kernel_init+0x50/0x100
Jun 19 15:40:18 sv54 kernel: [    0.387126]  [<ffffffff882064f5>] ? ret_from_fork+0x25/0x30
Jun 19 15:40:18 sv54 kernel: [    0.387127] ---[ end trace 44d08096f31f4f07 ]---
Jun 19 15:40:18 sv54 kernel: [    0.387363] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387365] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:353 fpu__copy+0x140/0x190
Jun 19 15:40:18 sv54 kernel: [    0.387366] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387367] CPU: 2 PID: 1 Comm: init Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387368]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387369]  ffffffff87c76eae ffff9ec0723e7ec0 ffff9ec076158b40 ffff9ec0723e7e80
Jun 19 15:40:18 sv54 kernel: [    0.387370]  0000000000000000 0000000000000000 00007f7cfecd79d0 ffffffff87c30a90
Jun 19 15:40:18 sv54 kernel: [    0.387371] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387373]  [<ffffffff87f28634>] ? dump_stack+0x5c/0x78
Jun 19 15:40:18 sv54 kernel: [    0.387374]  [<ffffffff87c76eae>] ? __warn+0xbe/0xe0
Jun 19 15:40:18 sv54 kernel: [    0.387375]  [<ffffffff87c30a90>] ? fpu__copy+0x140/0x190
Jun 19 15:40:18 sv54 kernel: [    0.387376]  [<ffffffff87c74490>] ? copy_process.part.33+0x1a0/0x1c00
Jun 19 15:40:18 sv54 kernel: [    0.387377]  [<ffffffff87dfe0da>] ? __check_object_size+0xfa/0x1d8
Jun 19 15:40:18 sv54 kernel: [    0.387379]  [<ffffffff87f56988>] ? strncpy_from_user+0x48/0x160
Jun 19 15:40:18 sv54 kernel: [    0.387379]  [<ffffffff87e0744d>] ? cp_new_stat+0x14d/0x180
Jun 19 15:40:18 sv54 kernel: [    0.387381]  [<ffffffff87c760d3>] ? _do_fork+0xe3/0x3f0
Jun 19 15:40:18 sv54 kernel: [    0.387381]  [<ffffffff87e074b9>] ? SYSC_newstat+0x39/0x60
Jun 19 15:40:18 sv54 kernel: [    0.387383]  [<ffffffff87c03b1c>] ? do_syscall_64+0x7c/0xf0
Jun 19 15:40:18 sv54 kernel: [    0.387384]  [<ffffffff8820632f>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jun 19 15:40:18 sv54 kernel: [    0.387384] ---[ end trace 44d08096f31f4f08 ]---
Jun 19 15:40:18 sv54 kernel: [    0.387420] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387422] WARNING: CPU: 2 PID: 1 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:353 __switch_to+0x66a/0x6c0
Jun 19 15:40:18 sv54 kernel: [    0.387422] ------------[ cut here ]------------
Jun 19 15:40:18 sv54 kernel: [    0.387423] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387425] WARNING: CPU: 3 PID: 0 at /build/linux-FT3UnK/linux-4.9.30/arch/x86/include/asm/fpu/internal.h:368 __switch_to+0x415/0x6c0
Jun 19 15:40:18 sv54 kernel: [    0.387426] CPU: 2 PID: 1 Comm: init Tainted: G        W       4.9.0-3-amd64 #1 Debian 4.9.30-2
Jun 19 15:40:18 sv54 kernel: [    0.387427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
Jun 19 15:40:18 sv54 kernel: [    0.387427] Modules linked in:
Jun 19 15:40:18 sv54 kernel: [    0.387428]  0000000000000000 ffffffff87f28634 0000000000000000 0000000000000000
Jun 19 15:40:18 sv54 kernel: [    0.387430]  ffffffff87c76eae ffff9ec0762401c0 ffff9ec076158040 ffff9ec076240cc0
Jun 19 15:40:18 sv54 kernel: [    0.387431]  ffff9ec0762401c0 0000000000000002 ffff9ec076158a80 ffffffff87c24aaa
Jun 19 15:40:18 sv54 kernel: [    0.387432] Call Trace:
Jun 19 15:40:18 sv54 kernel: [    0.387432] ---[ end trace 44d08096f31f4f09 ]---


After some research we did find out, that there is a bug in the QEMU/KVM packages as provided by jessie stable repositories.

To solve the problem, you have to install more recent QEMU/KVM packages, and the best way to get them for stretch is to add the jessie backports to your sources.list

deb http://ftp.debian.org/debian jessie-backports main

After the usual apt-get update just install the qemu-kvm and related packages in the 2.8xxx version, instead of the 2.1xxxx releases.

After a reboot of your physical server, you can now enjoy debian stretch in your guests

Tuesday, June 6, 2017

Purge and remove backup files in Bareos / Bacula

Bareos / Bacula are very powerfull backup solutions for enterprises

When using backup to disk, one of the more nasty maintenance tasks, is removing old backup files.
To help you with this, you can use this script to remove expired backup files, and also backup files which contain error-only backups.

It is based on the code from Kjetil and has been modified to also woth with bareos 15 and mysql databases.

Patches and enhancements are welcome on the github platform.