Skip to main content

RIZZLY0023: Interruptin g idle Thread glassfish 3.1.2.2 build5

18 replies [Last post]
baze985
Offline
Joined: 2009-05-07
Points: 0

Hi all,

I know that this is a old very known problem but from all the topics I saw its saying should be not present(fixed) in glassfish 3.1.2.2 build5!

I re-produce the problem over a 5 times in my test/dev enviroment...

What happens is that glassfish tryies to stop-interrupt a idle thread which is all ok..but it dose this in a starvesion lock flowting the log file with this output and eating up maximum(100%) of the machine cpu...

[#|2012-11-13T20:23:52.137+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.Gr
izzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0023: Interruptin
g idle Thread: admin-thread-pool-4848(5).|#]
[#|2012-11-13T20:23:53.138+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.Gr
izzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0023: Interruptin
g idle Thread: admin-thread-pool-4848(4).|#]

Any idea somebody?!

Like I said before I reproduce the problem 5-6 times from a fresh instaled version with no app deployed

Tnx
B

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Anonymous

Hi, unfortunately it hasn't been fixed in 3.1.2.2, you need to apply this
patch [1] in order to fix it. WBR, Alexey. [1]
http://forums.java.net/forum/topic/glassfish/glassfish/what-causes-grizz...

--

[Message sent by forum member 'oleksiys']

View Post: http://forums.java.net/node/892403

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

thank you for the fast replay!

I swap the patch for grizzly-http.jar but I see no changes at all. Do i have to clean some cache?!
What I did is just stop glassfish remove the jar and add the new one. But the outcome is the same!!

log file filled with this warning every second...

[#|2012-11-17T16:08:33.568+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0
023: Interrupting idle Thread: admin-thread-pool-4848(15).|#]

[#|2012-11-17T16:08:33.568+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0
023: Interrupting idle Thread: admin-thread-pool-4848(24).|#]

Like I said before, I use out of the box glassfish 3.1.2.2 build5
my process of testing the bug is this one:

1. instal glassfish
2. start the admin web ui
3. create one jdbc pool and one resource for the this pool pointing to a mysql database which has a table with about 2gb of data in it.
4. created a single ejb bean application annotated this ejb bean as @Startup and created one method in it annotated with @PostConstract, in this method i open a connection to mysql and I fire a add constraint to the big table. This add constraint in mysql takes about 45 mins on my machine
5. deploy the app from the aadmin web ui. On deployment the app stops(goes Idle) waithing for my sql to finish the add of constraint, and after 15 mins(default time for idle handling in glassfish) the logfile starts getting the warnning message every half second. During this period I dont see a big cpu usage from the glassfish process, on my 2core dev machine mysql uses about 50% of one core and glassgish not more then a 1%. But after the 45mins when mysql recreation of the table is done, mysql goes to 0% cpu usage and then glassfish takes over with a 100%(one core) usage....

Restart ofcourse helps:) but if I go prod with this version Ill get in the problem after couple of hours. In my applicatiosn I do a restfull calls every 10sec for a small update of data...and some of this call will end in this kind of loop for sure...

Regards,
Blaze

p.s. Ill give it a secound try with this patch jar...just to be sure that its again the same out come as the first try...

Anonymous

Hi Blaze, do you see lots of [1]-like messages or usually it's just 2 of
them? Cause the patch supposed to fix the problem, when the log [1] is being
reported over and over on the same thread. Looks like issue you see is a bit
different, I'd say it might be a problem in the admin gui code, which
swallows InterruptedIOException, but doesn't reset Thread's "interrupted"
state, so it goes into infinite loop and consumes 100% of CPU. I'd recommend
to file an admin-gui issue. As a workaround you might want to disable
request-timeout for admin-listener: /$ asadmin set
server-config.network-config.protocols.protocol.admin-listener.http.request-timeout-seconds=-1/
Thanks. WBR, Alexey. [1]
[#|2012-11-17T16:08:33.568+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0
023: Interrupting idle Thread: admin-thread-pool-4848(15).|#]

--

[Message sent by forum member 'oleksiys']

View Post: http://forums.java.net/node/892403

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

I did more testing and here is what I find out!

The replace of the jar in the module dir is not enough..some cache stuff needs to cleaned also,
just like the other guy said.

I did a 2 times test on a out of the box glassfish where before very first start I replace the jar, all went fine. This time I saw just 2 messages like you said, and after the add of the new constraint to the table glassfish come down to a 0% cpu usage not like before where was going to 100%cpu usage(one core)

but,

on a two day running on my dev env where I did some deploys from the admin gui, and run just one user making calles to my rest service, the bug happend again. What is interesting now is that I see no logging in the logfile but the cpu goes again to 100%, and I had to bounce the GF process cos was totally not responding...

Ma thinking is that the bug is still there..something moves glassfish to a starvation lock but now just the warning is missing...

Also you mantioned a way to disable the idle check for the admin http treads
request-timeout for admin-listener: /$ asadmin set
server-config.network-config.protocols.protocol.admin-listener.http.request-timeout-seconds=-1/

is this posible to be done for a normal one?!
also if this is posible what is the scenario then?
this idle thread is lost forever and the available pool threads decrese?

This same application runs about 2 year now on a GF 3.0.1 with no problems at all, I like to move to last version cos its really much faster:) And i know why is having this kind of problems. You rewrite a lot of stuff I guess...:)

Once again thank you for your replay!

BR,
Blaze

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi Blaze,

regarding the 100% CPU consumption problem, it might (and most probably is) be a different problem. When you observe this just take couple of jstack snapshots w/ say 5-15 seconds inverval, so we can guess which thread consumes it.

> Also you mantioned a way to disable the idle check for the admin http treads
> request-timeout for admin-listener: /$ asadmin set
> server-config.network-config.protocols.protocol.admin-listener.http.request-timeout-seconds=-1
> is this posible to be done for a normal one?!
Yes, absolutely, just replace "admin-listener" w/ the proper listener name.

> this idle thread is lost forever and the available pool threads decrese?
If the code, which makes the thread idle doesn't have its own timeout detection algorithm - then yes.

> This same application runs about 2 year now on a GF 3.0.1 with no problems at all, I like to
> move to last version cos its really much faster:) And i know why is having this kind of
> problems. You rewrite a lot of stuff I guess...:)
:)) Let's figure out what causes 100% CPU consumption, because if you don't see the logged messages it might be unrelated issue.

Thanks.

WBR,
Alexey.

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

I went productive today and after 2hrs glassfish went in 100%spu usage. I made couple of snapshots with jstack, some of them with -F some with -m.

Hope the files will help!

BR,
Blaze

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi Blaze,
the dump is not that helpful :(
BTW, do you see the "GRIZZLY0023:..." message in the log?
If not - it has to be different issue. I see only one busy Grizzly thread, which is running some GWT RPC task, but there are lots of EJB threads, so it might be one of them.

Let's try to figure out which exactly thread is consuming 100% CPU. I found a blog [1], which looks useful.

Let me know if it works for you.

Thanks.

WBR,
Alexey.

[1] http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is-c...

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

thank you for the replay!
This is bad news:(

To give more info on the app...would be a good idea now!

I have one ear.file using jpa to contact the database and I have one war file(gwt as u see) calling the ear via remote interfaces. The client dose a restcall to the war every 10sec and this transfer the call via the remote interfaces to the related beans.

I did couple of more snapshots...it went to 100% after a couple of hours again.
Maybe this will help...

One last thing ill like to try when its going to 100% again is to undeploy the war file. But this is not possible because glassfish is not responding at all.

Check the snapshots..maybe you see some more info!

Thanks again
Blaze

last, when i try to connect with jstack it throws this exceptions...lots of them this is just a part..

sun.jvm.hotspot.oops.UnknownOopException
at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:393)
at sun.jvm.hotspot.runtime.JavaThread.getThreadObj(JavaThread.java:331)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentParkBlocker(JavaThread.java:383)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:82)
at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)

java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:244)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:451)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:288)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:213)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:218)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:235)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
sun.jvm.hotspot.utilities.AssertionFailure: overflow
at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)
at sun.jvm.hotspot.compiler.OopMapSet.updateRegisterMap(OopMapSet.java:267)
at sun.jvm.hotspot.runtime.x86.X86Frame.senderForCompiledFrame(X86Frame.java:411)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:283)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:213)
at sun.jvm.hotspot.runtime.VFrame.newVFrame(VFrame.java:83)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:228)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)

java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:244)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:451)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:288)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:213)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:218)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:235)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:244)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:451)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:288)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:213)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:218)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:119)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:235)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
at sun.tools.jstack.JStack.main(JStack.java:102)

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi Blaze,

again, difficult to say. If you compare these snapshots with the previous ones - they are absolutely different. I'd really suggest to try the "top" utility as described in the blog i mentioned in my prev. post.

Thanks.

WBR,
Alexey.

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

I did the 'top' search for a thread and I cannot find a thread with this pid inside the dump.
maybe I miss something..

Check the attachment dump!

you will find inside a top from a process showing that the process pid is 14269,
and in the thread top ull find that the thread using 100% is the one with pid 14274.
But in the dump it self I cant see(find) this kind of thread, just threads with dump ids higher then 14274....(if the numbers are in dec. system)

One thing im sure is that the problems lays between the war and the ear call or directly in the war...because all the other stuff when no war is present worked with no problem at all
When the war is added then with 5-6 clients I get the problem..

the war file package names start with mk.icelabs.vtcustomer.....

Once again thank you a lot!!!

BR,
Blaze

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi Blaze,

in thread3.out I found this [1], the id 14274 corresponds to the one reported by top. From the first glance looks like [1] is GC thread. So I'm starting to think that 100% CPU is caused by GC. You can try to monitor GC cycles and memory consumption (for ex. using jvisualvm).

WBR,
Alexey.

[1] ----------------- 14274 -----------------
0x00a1f9c9 _ZN13instanceKlass19oop_follow_contentsEP7oopDesc + 0x119
0x00b1aa21 _ZN9MarkSweep12follow_stackEv + 0x31
0x00c3ef87 _ZN8Universe7oops_doEP10OopClosureb + 0x17
0x00bc2752 _ZN10SharedHeap20process_strong_rootsEbbNS_14ScanningOptionEP10OopClosureP15CodeBlobClosureP16OopsInGenClosure + 0x2a2
0x009f4cfb _ZN16GenCollectedHeap24gen_process_strong_rootsEibbbN10SharedHeap14ScanningOptionEP16OopsInGenClosurebS3_ + 0x5b
0x009f796f _ZN12GenMarkSweep17mark_sweep_phase1Eib + 0x7f
0x009f7f65 _ZN12GenMarkSweep19invoke_at_safepointEiP18ReferenceProcessorb + 0x105

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

I tried to connect with VisualVm on 8686,
but because GF is not responsive it cannot be done..

Here is more of the jstack dumps, they are available only in a mix mode,
jstack -m pid, must be a low level native thread...

----------------- 14274 -----------------
0x00b5e931 _ZN13objArrayKlass18oop_oop_iterate_nvEP7oopDescP15FastScanClosure + 0x1
0x009998e9 _ZN16DefNewGeneration31oop_since_save_marks_iterate_nvEP15FastScanClosure + 0x39
0x009f5779 _ZN16GenCollectedHeap28oop_since_save_marks_iterateEiP15FastScanClosureS1_ + 0x29
0x0099bbd5 _ZN16DefNewGeneration7collectEbbjb + 0x245
0x009f6a95 _ZN16GenCollectedHeap13do_collectionEbbjbi + 0x4a5
0x0094a6c7 _ZN18GenCollectorPolicy25satisfy_failed_allocationEjb + 0xe7
0x00c62024 _ZN26VM_GenCollectForAllocation4doitEv + 0x74
0x00c6b211 _ZN12VM_Operation8evaluateEv + 0x41
0x00c69af8 _ZN8VMThread18evaluate_operationEP12VM_Operation + 0x78
0x00c6a088 _ZN8VMThread4loopEv + 0x1e8
0x00c6a725 _ZN8VMThread3runEv + 0x85
0x00b7a8f1 _ZL10java_startP6Thread + 0x111
0x00552852 start_thread + 0xe2

----------------- 14274 -----------------
0x00a1f9b8 _ZN13instanceKlass19oop_follow_contentsEP7oopDesc + 0x108
0x00b1aa21 _ZN9MarkSweep12follow_stackEv + 0x31
0x00c3ef87 _ZN8Universe7oops_doEP10OopClosureb + 0x17
0x00bc2752 _ZN10SharedHeap20process_strong_rootsEbbNS_14ScanningOptionEP10OopClosureP15CodeBlobClosureP16OopsInGenClosure + 0x2a2
0x009f4cfb _ZN16GenCollectedHeap24gen_process_strong_rootsEibbbN10SharedHeap14ScanningOptionEP16OopsInGenClosurebS3_ + 0x5b
0x009f796f _ZN12GenMarkSweep17mark_sweep_phase1Eib + 0x7f
0x009f7f65 _ZN12GenMarkSweep19invoke_at_safepointEiP18ReferenceProcessorb + 0x105
0x00a00fa1 _ZN28OneContigSpaceCardGeneration7collectEbbjb + 0x41
0x009f6a95 _ZN16GenCollectedHeap13do_collectionEbbjbi + 0x4a5
0x0094a6c7 _ZN18GenCollectorPolicy25satisfy_failed_allocationEjb + 0xe7
0x00c62024 _ZN26VM_GenCollectForAllocation4doitEv + 0x74
0x00c6b211 _ZN12VM_Operation8evaluateEv + 0x41
0x00c69af8 _ZN8VMThread18evaluate_operationEP12VM_Operation + 0x78
0x00c6a088 _ZN8VMThread4loopEv + 0x1e8
0x00c6a725 _ZN8VMThread3runEv + 0x85
0x00b7a8f1 _ZL10java_startP6Thread + 0x111
0x00552852 start_thread + 0xe2

----------------- 14274 -----------------
0x00bdde50 _ZN16CompactibleSpace7forwardEP7oopDescjP12CompactPointP8HeapWord
0x00a006ec _ZN10Generation22prepare_for_compactionEP12CompactPoint + 0x2c
0x009f456c _ZN16GenCollectedHeap22prepare_for_compactionEv + 0x3c
0x009f7f9c _ZN12GenMarkSweep19invoke_at_safepointEiP18ReferenceProcessorb + 0x13c
0x00a00fa1 _ZN28OneContigSpaceCardGeneration7collectEbbjb + 0x41
0x009f6a95 _ZN16GenCollectedHeap13do_collectionEbbjbi + 0x4a5
0x0094a6c7 _ZN18GenCollectorPolicy25satisfy_failed_allocationEjb + 0xe7
0x00c62024 _ZN26VM_GenCollectForAllocation4doitEv + 0x74
0x00c6b211 _ZN12VM_Operation8evaluateEv + 0x41
0x00c69af8 _ZN8VMThread18evaluate_operationEP12VM_Operation + 0x78
0x00c6a088 _ZN8VMThread4loopEv + 0x1e8
0x00c6a725 _ZN8VMThread3runEv + 0x85
0x00b7a8f1 _ZL10java_startP6Thread + 0x111
0x00552852 start_thread + 0xe2

----------------- 14274 -----------------
0x00a1f9c9 _ZN13instanceKlass19oop_follow_contentsEP7oopDesc + 0x119
0x00b1aa21 _ZN9MarkSweep12follow_stackEv + 0x31
0x00c3ef87 _ZN8Universe7oops_doEP10OopClosureb + 0x17
0x00bc2752 _ZN10SharedHeap20process_strong_rootsEbbNS_14ScanningOptionEP10OopClosureP15CodeBlobClosureP16OopsInGenClosure + 0x2a2
0x009f4cfb _ZN16GenCollectedHeap24gen_process_strong_rootsEibbbN10SharedHeap14ScanningOptionEP16OopsInGenClosurebS3_ + 0x5b
0x009f796f _ZN12GenMarkSweep17mark_sweep_phase1Eib + 0x7f
0x009f7f65 _ZN12GenMarkSweep19invoke_at_safepointEiP18ReferenceProcessorb + 0x105
0x00a00fa1 _ZN28OneContigSpaceCardGeneration7collectEbbjb + 0x41
0x009f6a95 _ZN16GenCollectedHeap13do_collectionEbbjbi + 0x4a5
0x0094a6c7 _ZN18GenCollectorPolicy25satisfy_failed_allocationEjb + 0xe7
0x00c62024 _ZN26VM_GenCollectForAllocation4doitEv + 0x74
0x00c6b211 _ZN12VM_Operation8evaluateEv + 0x41
0x00c69af8 _ZN8VMThread18evaluate_operationEP12VM_Operation + 0x78
0x00c6a088 _ZN8VMThread4loopEv + 0x1e8
0x00c6a725 _ZN8VMThread3runEv + 0x85
0x00b7a8f1 _ZL10java_startP6Thread + 0x111
0x00552852 start_thread + 0xe2

0x00bdf42a _ZN15ContiguousSpace22prepare_for_compactionEP12CompactPoint + 0x2aa
0x00a006ec _ZN10Generation22prepare_for_compactionEP12CompactPoint + 0x2c
0x009f7fbc _ZN12GenMarkSweep19invoke_at_safepointEiP18ReferenceProcessorb + 0x15c
0x00a00fa1 _ZN28OneContigSpaceCardGeneration7collectEbbjb + 0x41
0x009f6a95 _ZN16GenCollectedHeap13do_collectionEbbjbi + 0x4a5
0x0094a6c7 _ZN18GenCollectorPolicy25satisfy_failed_allocationEjb + 0xe7
0x00c62024 _ZN26VM_GenCollectForAllocation4doitEv + 0x74
0x00c6b211 _ZN12VM_Operation8evaluateEv + 0x41
0x00c69af8 _ZN8VMThread18evaluate_operationEP12VM_Operation + 0x78
0x00c6a088 _ZN8VMThread4loopEv + 0x1e8
0x00c6a725 _ZN8VMThread3runEv + 0x85
0x00b7a8f1 _ZL10java_startP6Thread + 0x111
0x00552852 start_thread + 0xe2

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

Just to give a small feedback, I found the problem for the gc thread, It was a gwt class which had a dual implement..cross compile and server side. I fleatern the class for the server side use and all worked ok.

But after a one day work I got again(looks like this) the grizzly problem.
What happend was that the log had about 6-7 warning logs tring to stop idle thread, after this glassfish went to high cpu for a couple of minutes and then come down, but then mysql went sky high 4-5 mysql threads wore consuming maximum cpu.

But what is really interesting is that, because od the small changes I did in the apps, now the app is not working on 3.0.1. Its deploing and the ear works fine, but when I try to call a remote bean to get a object which has member field of type enum, Im getting this exception..

Caused by: java.io.InvalidObjectException: can't deserialize enum
at java.lang.Enum.readObject(Enum.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.corba.ee.impl.io.IIOPInputStream.invokeObjectReader(IIOPInputStream.java:1965)
at com.sun.corba.ee.impl.io.IIOPInputStream.inputObject(IIOPInputStream.java:1300)
at com.sun.corba.ee.impl.io.IIOPInputStream.simpleReadObject(IIOPInputStream.java:449)
at com.sun.corba.ee.impl.io.ValueHandlerImpl.readValueInternal(ValueHandlerImpl.java:364)
at com.sun.corba.ee.impl.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:320)
at com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.readRMIIIOPValueType(CDRInputStream_1_0.java:1066)
... 80 more
|#]

[#|2012-11-29T02:30:15.218+0100|WARNING|glassfish3.0.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.encoding|_ThreadID=26;_ThreadName=Thread-1;|"IOP00810211: (MARSHAL) Exception from readValue on ValueHandler in CDRInputStream"
org.omg.CORBA.MARSHAL: vmcid: SUN minor code: 211 completed: Maybe

Caused by: java.io.InvalidObjectException: can't deserialize enum
at java.lang.Enum.readObject(Enum.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.corba.ee.impl.io.IIOPInputStream.invokeObjectReader(IIOPInputStream.java:1965)
at com.sun.corba.ee.impl.io.IIOPInputStream.inputObject(IIOPInputStream.java:1300)
at com.sun.corba.ee.impl.io.IIOPInputStream.simpleReadObject(IIOPInputStream.java:449)
at com.sun.corba.ee.impl.io.ValueHandlerImpl.readValueInternal(ValueHandlerImpl.java:364)
at com.sun.corba.ee.impl.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:320)
at com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.readRMIIIOPValueType(CDRInputStream_1_0.java:1066)
... 80 more

at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
at mk.icelabs.vtcore.advertisement.__HistoryEntryUC_Remote_DynamicStub.findOldHistoryEntries(mk/icelabs/vtcore/advertisement/__History
EntryUC_Remote_DynamicStub.java)
at mk.icelabs.vtcore.advertisement._HistoryEntryUC_Wrapper.findOldHistoryEntries(mk/icelabs/vtcore/advertisement/_HistoryEntryUC_Wrapp
er.java)

|#]

[#|2012-11-29T02:30:15.218+0100|WARNING|glassfish3.0.1|javax.enterprise.resource.corba.ee.S1AS-ORB.rpc.encoding|_ThreadID=26;_ThreadName=Thread-1;|"IOP00810211: (MARSHAL) Exception from readValue on ValueHandler in CDRInputStream"
org.omg.CORBA.MARSHAL: vmcid: SUN minor code: 211 completed: Maybe

I cannot belive it...:)
This same ear and war have no problem like this on 3.1.2.2, but there I have the cpu usage problem...

Any idea maybe?
Im becaming speachless:)

BR,
Blaze

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi Blaze,

the exception looks like corba related, not sure what it exactly means.

Regarding the logs - it doesn't look like Grizzly issue as well, 6 messages (only) sounds like a proper request-timeout interruption. Do that messages notify about different threads interruption or the same one? It might be MySQL connection pool problem.

Thanks.

WBR,
Alexey.

baze985
Offline
Joined: 2009-05-07
Points: 0

Hi Alexey,

tnx for the replay once again.
Ill observe it when the next time happens and try to get the shapshots. I havent used jstack that much, do I have to call it with some special parameters or the normal one: jstack -m PID

BR,
Blaze

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi Blaze,

do you see lots of [1]-like messages or usually it's just 2 of them? Cause the patch supposed to fix the problem, when the log [1] is being reported over and over on the same thread.

Looks like issue you see is a bit different, I'd say it might be a problem in the admin gui code, which swallows InterruptedIOException, but doesn't reset Thread's "interrupted" state, so it goes into infinite loop and consumes 100% of CPU.

I'd recommend to file an admin-gui issue. As a workaround you might want to disable request-timeout for admin-listener:
$ asadmin set server-config.network-config.protocols.protocol.admin-listener.http.request-timeout-seconds=-1

Thanks.

WBR,
Alexey.

[1] [#|2012-11-17T16:08:33.568+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0
023: Interrupting idle Thread: admin-thread-pool-4848(15).|#]

Korbinian Bachl

afaik all caches in Domain directory and the generated one needs to be emptied ( especially osgi cache dir)

Best

Von meinem iPhone gesendet

Am 17.11.2012 um 15:52 schrieb :

> Hi Alexey, thank you for the fast replay! I swap the patch for
> grizzly-http.jar but I see no changes at all. Do i have to clean some cache?!
> What I did is just stop glassfish remove the jar and add the new one. But the
> outcome is the same!! log file filled with this warning every second...
> [#|2012-11-17T16:08:33.568+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0
> 023: Interrupting idle Thread: admin-thread-pool-4848(15).|#]
> [#|2012-11-17T16:08:33.568+0100|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0
> 023: Interrupting idle Thread: admin-thread-pool-4848(24).|#] Like I said
> before, I use out of the box glassfish 3.1.2.2 build5 my process of testing
> the bug is this one: 1. instal glassfish 2. start the admin web ui 3. create
> one jdbc pool and one resource for the this pool pointing to a mysql database
> which has a table with about 2gb of data in it. 4. created a single ejb bean
> application annotated this ejb bean as @Startup and created one method in it
> annotated with @PostConstract, in this method i open a connection to mysql
> and I fire a add constraint to the big table. This add constraint in mysql
> takes about 45 mins on my machine 5. deploy the app from the aadmin web ui.
> On deployment the app stops(goes Idle) waithing for my sql to finish the add
> of constraint, and after 15 mins(default time for idle handling in glassfish)
> the logfile starts getting the warnning message every half second. During
> this period I dont see a big cpu usage from the glassfish process, on my
> 2core dev machine mysql uses about 50% of one core and glassgish not more
> then a 1%. But after the 45mins when mysql recreation of the table is done,
> mysql goes to 0% cpu usage and then glassfish takes over with a 100%(one
> core) usage.... Restart ofcourse helps:) but if I go prod with this version
> Ill get in the problem after couple of hours. In my applicatiosn I do a
> restfull calls every 10sec for a small update of data...and some of this call
> will end in this kind of loop for sure... Regards, Blaze p.s. Ill give it a
> try one more time with the new patch jar...just to be sure...but I think its
> going to be the same
>
> --
>
> [Message sent by forum member 'baze985']
>
> View Post: http://forums.java.net/node/892403
>
>

oleksiys
Offline
Joined: 2006-01-25
Points: 0

Hi,

unfortunately it hasn't been fixed in 3.1.2.2, you need to apply this patch [1] in order to fix it.

WBR,
Alexey.

[1] http://forums.java.net/forum/topic/glassfish/glassfish/what-causes-grizz...