Skip to main content

What causes GRIZZLY to flood the Glassfish-Log with messages like this 'Interrupting idle Thread: http-thread-pool-80(3)'

24 replies [Last post]
GrhrdBlmln
Offline
Joined: 2012-07-26

Hey hey @all,

my glassfish-server-log will be flooded with the following message:

[#|2012-08-18T16:24:11.461+0200|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=11;_ThreadName=Thread-2;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-80(3).|#]

... after 15 minutes this message starts repeating ervery second, which floods my server.log.

By the way, I don't use GZIP compression, which was indicated in serveral other topis for the same GF-behavior, here in this forum.

Does anyone have idea what may cause GRIZZLY not to finalize / delete the http-request, which is obviously timed-out?

Thank you in advance for your response.
Gerhard

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Open Indiana

How big is your http-thread-pool?

-----Original Message-----
From: admin@java.net [mailto:admin@java.net] On Behalf Of forums@java.net
Sent: maandag 3 september 2012 13:41
To: users@glassfish.java.net
Subject: What causes GRIZZLY to flood the Glassfish-Log with messages like this 'Interrupting idle Thread: http-thread-pool-80(3)'

Hey hey @all, my glassfish-server-log will be flooded with the following
message:
[#|2012-08-18T16:24:11.461+0200|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=11;_ThreadName=Thread-2;|GRIZZLY0023:
Interrupting idle Thread: http-thread-pool-80(3).|#] ... after 15 minutes this message starts repeating ervery second, which floods my server.log. By the way, I don't use GZIP compression, which was indicated in serveral other topis for the same GF-behavior, here in this forum. Does anyone have idea what may cause GRIZZLY not to finalize / delete the http-request, which is obviously timed-out? Thank you in advance for your response. Gerhard

--

[Message sent by forum member 'GrhrdBlmln']

View Post: http://forums.java.net/node/889790

GrhrdBlmln
Offline
Joined: 2012-07-26

hey hey forumguest,

actually we use the standard-pool-size-values (min 2, max 5, timeout 900sec) which seems to be OK for us. the application will run for a couple of hours or days, sometimes weeks without any problem. and after an undefined timeperiod more and more http-pool-threads stuck in something like an infinite loop or something else.

jstack will show something like this for the correspondent threads:
"http-thread-pool-80(1)" daemon prio=10 tid=0x0000000043408800 nid=0x6d2c runnable [0x00007f04cb3f1000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00000000f8292228> (a java.lang.InterruptedException)
at java.lang.Throwable.(Throwable.java:181)
at java.lang.Exception.(Exception.java:29)
at java.lang.InterruptedException.(InterruptedException.java:38)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:735)
at com.sun.corba.ee.impl.oa.poa.POAImpl.acquireLock(POAImpl.java:390)
at com.sun.corba.ee.impl.oa.poa.POAImpl.readLock(POAImpl.java:422)
at com.sun.corba.ee.impl.oa.poa.POAImpl.enter(POAImpl.java:1743)
at com.sun.corba.ee.impl.protocol.FullServantCacheLocalCRDImpl.internalPreinvoke(FullServantCacheLocalCRDImpl.java:73)
at com.sun.corba.ee.impl.protocol.LocalClientRequestDispatcherBase.servant_preinvoke(LocalClientRequestDispatcherBase.java:240)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.servant_preinvoke(CorbaClientDelegateImpl.java:574)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:218)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
...
(some application classes)
...
at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

The interesting thing is, that the thread hasn't the state BLOCKING. Can anyone explain this to me?

oleksiys
Offline
Joined: 2006-01-25

Hi,

are you using Glassfish 3.1.2.2? If not - can you pls. try it?

Thanks.

WBR,
Alexey.

On 09/03/2012 01:40 PM, forums@java.net wrote:
> Hey hey @all, my glassfish-server-log will be flooded with the following
> message:
> [#|2012-08-18T16:24:11.461+0200|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=11;_ThreadName=Thread-2;|GRIZZLY0023:
>
> Interrupting idle Thread: http-thread-pool-80(3).|#] ... after 15 minutes
> this message starts repeating ervery second, which floods my
> server.log. By
> the way, I don't use GZIP compression, which was indicated in serveral
> other
> topis for the same GF-behavior, here in this forum. Does anyone have idea
> what may cause GRIZZLY not to finalize / delete the http-request,
> which is
> obviously timed-out? Thank you in advance for your response. Gerhard
>
> --
>
> [Message sent by forum member 'GrhrdBlmln']
>
> View Post: http://forums.java.net/node/889790
>
>

GrhrdBlmln
Offline
Joined: 2012-07-26

hey Alexey,

we already updated the glassfish to the newest stable version (GlassFish Server Open Source Edition 3.1.2.2 (build 5)) but this didn't fix our problem. :-(

Any other ideas?

Thank you in advance for your response.
Gerhard

oleksiys
Offline
Joined: 2006-01-25

Hey Gerhard,
can you pls. try this patch w/ Glassfish 3.1.2.2 (copy the attached jar to gfv3/glassfish/modules).
Pls. let me know if it works.
Also wanted to ask you to create an issue here [1].

Thank you.

WBR,
Alexey.

[1] http://java.net/jira/browse/GLASSFISH

GrhrdBlmln
Offline
Joined: 2012-07-26

Hey Alexey,
thnaxalot for your quick answer and for the patch. I will let you know if this patch fixes our issues. The first tests in our development respectively virtual testing environment looks good, but unfortunatly this is not new to us.
The problems rises only on the productive machines which i will patch next week.
When i have more informations about the situation, i'll give you a more detailed feedback and create an issue on http://java.net/jira/browse/GLASSFISH as requested.

Have a nice weekend,
Gerhard

oleksiys
Offline
Joined: 2006-01-25

Hi Gerhard,

looks like this forum doesn't work quite well at the moment.
I see your messages on glassfish mailing list (users [-at-] glassfish.java.net), but not on this forum.
I filed an issue [1], hope the forum will be fixed soon. Meanwhile, can I ask you to post your messages to the glassfish users mailing list directly [2]?

Thanks.

WBR,
Alexey.

[1] http://kenai.com/jira/browse/KENAI-3713
[2] http://glassfish.java.net/public/mailing-lists.html

oleksiys
Offline
Joined: 2006-01-25

Hi Gerhard,

any updates?

Thanks.
Alexey.

GrhrdBlmln
Offline
Joined: 2012-07-26

Hey Alexey,

good news from the bad world !!! :-)
The patch which was provided by Harshad Vilekar on the thread [1] works fine for our problem with GF.
Last Friday we had again the situation, that the GF detects an idle thread and tries to interrupt it. … Now with success !!! :-)
Obviously the patched GF is now able to interrupt the faulty thread.
Now we see our implemented error-messages in the GF-log which gives us more informations about the main cause of the problem.

In our case we tried to read from the InputStream of the request, which obviously crashes when the length of the submitted XML-File is 0.

I think that our problem was reading from an InputStream which has a length of 0 and was caused by the following lines of code:


try {
    xmlString = getValue(request.getInputStream());
                   
} catch (Exception e) {
    ...
    System.out.println("Error in Methode : processRequest");
    System.out.println("Error Message : " + e.getMessage());
}



private String getValue(InputStream is)throws IOException {
        BufferedInputStream bis = new BufferedInputStream(is);
        ByteArrayOutputStream buf = new ByteArrayOutputStream();
        int result = bis.read();
        while(result != -1) {
          byte b = (byte)result;
          buf.write(b);
          result = bis.read();
        }       
        return buf.toString();
}

Apart from that the patch of Harshad Vilekar will make a good job, we modified our source code with the following lines:

try {
    if(request.getInputStream().available() > 0){
        xmlString = getValue(request.getInputStream());
    }else{
        xmlString = "";
    }
   
} catch (Exception e) {
...
    System.out.println("Error in Methode : processRequest");
    System.out.println("Error Message : " + e.getMessage());
}

Now we first check the InpuStream of the request, if there is something in to read.
If not, we can react on that situation and will not have to rely on the GF error-handling. ;-)

I keep observing GF-logs for the rest of the week, but i think we made it.
Many many thanks for your support right now. If we reach the end of this week/month without any hanging threads, i call you my personal hero !!! ;-)

Cheers,
Gerhard

[1] http://java.net/jira/browse/GLASSFISH-16217

HolgerD
Offline
Joined: 2012-10-02

Hey Alexey,

sorry for the long time without any response of Gerhard, I am Holger a colleague of Gerhard.

So what has happend since his last response?
Yesterday we uploaded your patch fileto our prod. system and it works for more than 24 hours without any problems. Now we recognized at one prod. system that one http-thread stacks into something like a loop (like before).

The difference since patching the server is that just one log message appears in the glassfish log with "idle thread". But the server load is still high. One thread consumes 100% CPU power.

Bye Holger and Gerhard

HolgerD
Offline
Joined: 2012-10-02

Hey Alexey,

sorry for the long time without any response of Gerhard, I am Holger a colleague of Gerhard.

So what has happend since his last response?
1. we checked our source code again and implemented some little modifications in the two critical servlet algorithms, so that we can be sure that every servlet ends with a response.flushBuffer();
2. After the modification we updated the prod. system and for one week it seems to be the right solution. The server works without any problems.
3. During this time we patched our local GF in the netbeans IDE. It works without any problems.
4. On week 38 the prod. system raises the same problems as mentioned before. In the week 39 we had to stop the GF manually, because it was unavailable for the regular http-requests.

Yesterday we uploaded your patch fileto our prod. system and it works for more than 24 hours without any problems. Now we recognized at one prod. system that one http-thread stacks into something like a loop (like before).

The difference since patching the server is that just one log message appears in the glassfish log with "idle thread". But the server load is still high. One thread consumes 100% CPU power.

Bye Holger and Gerhard

GrhrdBlmln
Offline
Joined: 2012-07-26

Hi Alexey,

sorry for the few words, but this forum spam-protection drives me crazy !!! ;-)

So just a short description of the actual situation:

Your patch works for abording messages like 'Interrupting idle Thread: http-thread-pool-80(X)' in the glassfish-LOGfile.
Unfortunatly it doesn't solve the problem, that a thread will stuck in a loop.
Actually we have such a thread in one of our productive-systems which consumes 100% CPU power of one core. :-(

Any other suggestions ?

GrhrdBlmln
Offline
Joined: 2012-07-26

System-load graph

GrhrdBlmln
Offline
Joined: 2012-07-26

Hey Alexey,

we uploaded your patch to the productive systems, but it didn't fix our main problem.
Just the log messages like this 'Interrupting idle Thread: http-thread-pool-80(3)' after 15 min won't come up and didn't flood the LOG-File any more.
That's cool but didn't solve the main-problem.

GrhrdBlmln
Offline
Joined: 2012-07-26

just a test-reply caused by massive problems, posting a reply

GrhrdBlmln
Offline
Joined: 2012-07-26

Hey Alexey,

sorry for the long time without any response of me. (I was in vacation for the last two weeks :-) )

So what has happend since my last response ...
(1) We checked our source code again and implemented some little modifications in the two critical servlet algorithms, so that we can be sure that every servlett ends with an 'response.flushBuffer();'
(2) After the modification we updated the productive systems and for one week it seems to be the solution. The server works without any problems. (See attached picture of the CPU load graph for week 37)
(3) During this time i patched my local glassfish in the netbeans environment. It works without any problems.
(4) Unfortunatly on week 38 the productive systems raises the same problems as mentioned before. In week 39 we had to stop the glassfish manually, because it was unavailable for the regular http-requests.
Actually the system has again 3 http-thread-pool threads which seems to stuck in an infinite loop.

In my next step i do upload your patch to the productive systems and restart them manually.

I think it will take some time to get you some more feedback if it solves our issues or not.

Do you need some more informations about our source code to understand the big picture?
Do you have any suggestions what we can do or check in our source code to react on those bad threads?

Thanks in advance,
Gerhard

adam_dennis
Offline
Joined: 2012-09-18

Hi Gerhard,

We are having the same problem as you describe - I was wondering how the patch worked for you?

Also did you raise an issue?

many thanks!

Adam.

baze985
Offline
Joined: 2009-05-07

Hi all,

we are getting the same problem. The server runs ok for some period, can be hours days, and then just starts going in a infinite loop where this message is logged in the server log.
Restart can help but very fast goes in a same state again...

It realy funny that on a quadcore machine eats up all cpu power, runs on a 400% :)

Any help..or fix?!

We downgrade to 3.0.1 for now...

Blaze

oleksiys
Offline
Joined: 2006-01-25

Hi,

if you use GF 3.1.2.2, you may want to apply this patch [1] to fix inifinite logging.
And, if your app loops inside corba (see the stack trace at the top of the thread), take a look at this issue [2].

WBR,
Alexey.

[1] http://forums.java.net/forum/topic/glassfish/glassfish/what-causes-grizz...
[2] http://java.net/jira/browse/GLASSFISH-16217

baze985
Offline
Joined: 2009-05-07

Hi Alexey,

first thank you for the first replay!
I replaced the grizzly and after 3-4 hours, I started getting this exception at firt:

[#|2012-10-04T15:53:52.118+0200|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-3;|GRIZZLY0006: Exception accepting channel
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at com.sun.grizzly.TCPSelectorHandler.acceptWithoutRegistration(TCPSelectorHandler.java:745)
at com.sun.enterprise.v3.services.impl.monitor.MonitorableSelectorHandler.acceptWithoutRegistration(MonitorableSelectorHandler.java:99)
at com.sun.grizzly.http.SelectorThreadHandler.onAcceptInterest(SelectorThreadHandler.java:99)
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:301)
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263)
at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
|#]

this was thrown a lot of times, and then everything that was trying to use a socket was throwing the same exception, for example if server try to send email...

java.net.SocketException: Too many open files
at javax.mail.MessagingException: Could not connect to SMTP host: localhost, port: 25
at at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1934)
at at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:638)
at at javax.mail.Service.connect(Service.java:295)
at at javax.mail.Service.connect(Service.java:176)
at at javax.mail.Service.connect(Service.java:125)
at at javax.mail.Transport.send0(Transport.java:194)
at at javax.mail.Transport.send(Transport.java:124)
...
Caused by: java.net.SocketException: Too many open files
at at java.net.Socket.createImpl(Socket.java:397)
at at java.net.Socket.connect(Socket.java:527)
at at java.net.Socket.connect(Socket.java:478)
at at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:288)
at at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:231)
at at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1900)

As you can see the bougth use a different package for IO( one Uses NIO and other IO )
Any idea?!

Regards,
Blaze

oleksiys
Offline
Joined: 2006-01-25

Hi,

looks like you hit max open files limitation of your OS, most probably it's because you have too many clients connected at the same time.

Using some OS tools like netstat, you can try to figure out which service (HTTP, SMTP ...) accepts that many client connections.
If it's expected- you can just raise the limit.

WBR,
Alexey.

baze985
Offline
Joined: 2009-05-07

Hi Alexey,

Yes you are right, but Its crazy in any way because restart of the server didnt help(again I was getting in the log to much open files, even the admin console was not opening ) shutting down this one(3.1.2 build5) and starting up on the same machine 3.0.1 continue working with no problem at all:) Ofcourse bought are having same apps deployed...

Blaze

oleksiys
Offline
Joined: 2006-01-25

Hi Blaze,

can you pls. try GF 3.1.2.2, cause all the patches I suggested work w/ 3.1.2.2 only.
If you still have the same problem - try to find out what kind of files (may be network connections) consume all the file handles, otherwise it's difficult to narrow down the problem.

Thanks.

WBR,
Alexey.

baze985
Offline
Joined: 2009-05-07

Hi Alexey,

Yes this all that Im talking about is for 3.1.2.2 build5.
The glassfish crashed totally, any action related to opening socket was trowing a to many open files so I had to reinstall it from scratch(trying to load the admin console was also throwing the exception). After the clean install all works ok. Ill run it now again without the grizzly patch just to see how long will work..when I get the exception Ill replace the jar with the one from the patch..

Ill post a replay on my state!
Tnx for your help again,
Blaze