Wednesday, June 21, 2017

JSVC fails with error 11 after latest linux kernel Upgrades on debian/ubuntu

This morning, after doing some apt-get update/upgrades on various debian systems, we noticed that many of our java services did no longer work.
Since we use it to handle all our tomcat instances on many many servers, the impact is heavy.No single tomcat did run this morning...

Looking into the tomcat logs, we did see this message:
Service killed by signal 11

After some more investigations, it looks like the problem is related to the new kernel version installed in the upgrade process.

Google didn't find much about this, but the more important ones are here:
So the solution for now, is either switch to not using jsvc to start your services, or use a older or a "unstable" kernel.

The kernel causing the problems is
3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u1 (2017-06-18) x86_64
 
With the older one it does work
3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64
 
To switch back to the older kernel just do this, and reboot your system:
apt install linux-image-3.16.0-4-amd64=3.16.43-2  
 
Reverting back to a older kernel is discouraged, since this does not solve the security problem.

Fortunately there is a very simple work arround for it.
When you start jsvc, just specify it to use a larger stack.
For tomcat you can put this in your startup file, so the daemon.sh takes the new options for jsvc.
 
export JSVC_OPTS=-Xss1280k

Thanks to https://community.ubnt.com/t5/UniFi-Wireless/IMPORTANT-Debian-Ubuntu-users-MUST-READ-Updated-06-21/td-p/1968252