GRIZZLY0023: Interrupting idle Thread issue
We run a production environment with over 30 Glassfish servers, providing web services that are invoked by our client applications. We have recently upgraded our servers to Glassfish 3, previously we were running Glassfish 2. Since doing this we have been experiencing an intermittent issue where the server logs the error containing:
GRIZZLY0023: Interrupting idle Thread: http-thread-pool....
It repeatedly logs this every 2ms and the server no longer serves any requests. It is not possible to stop the domain, to resolve the problem the JVM has to be forced to close and then the domain started again. The issue mainly seems to happen overnight when we suspect the servers are experiencing no load. Some nights none of the servers experience the issue but this is rare (last night 3 servers failed).
We are running JVM 1.6.0_33, Glassfish 188.8.131.52 (build 5) and the OS is Linux centos (2.6.32-279.2.1.el6.x86_64) which is running in a virtual environment. There have been various threads on forums about this issue and we have tried the patches suggested but with little success. On Glassfish 2 we had to switch off the Grizzly connector and use the Coyote one because of an epoll selector bug but there does not seem to be this option on v3.
Perhaps someone could be so kind as to suggest the areas where we can enable logging and possibly help determine the cause? An obvious starting place to enable logging would appear to be com.sun.grizzly.config.GrizzlyServiceListener.