Skip to main content

EJBClassLoader - huge number of streams

12 replies [Last post]
imranbohoran
Offline
Joined: 2008-02-20

We've been running our application and have been having a high number of requests (more than usual) based on a new functionality in place.

This piece of functionality is served through it's own web-application which is packaged in a ear deployment amongst other war files. Since we've got this out, we've noticed our heap usage growing quite rapidly and pretty much tops up the old generation. I've taken a heap dump and realised that the EJBClassLoader had a huge number objects (447mb out of 480mb) were down to the 'streams' Vector.

Can any one shed some light on what gets held up as streams and any possible reasons why this might happen?

Cheers
-- Imran

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
preston001
Offline
Joined: 2008-11-19

Are you doing any JAXB stuff?
If you are make sure you're only creating one instance of your context and share it among the multiple requests.

366 Guest
Offline
Joined: 2010-11-02

Are you doing any JAXB stuff?

If you are make sure you're only creating one instance of your context and
share it among the multiple requests.

Richard Kolb

Hi Imran

On 29 October 2010 14:17, wrote:

> Thanks Richard and Tim for your responses.
>
> @Richard
> The technology stack I have for this is spring, dwr and hibernate.
> The web-application is packaged into a ear file amongst other web-apps.
> The JDK been used is 1.6.0_15
>

My first guess is try upgrade that JDK to _22.
It's a shot in the dark, but I know _16 had a memory leak that effects
GlassFish.
I think this only effected redeployment, not sure.

I can create a sample application, that represents the functionality, but
> unfortunately not quick enough amongst the other things going on. In terms
> of reproducing this a high load needs to be applied though. If this is the
> last resort to getting to the bottom of it, I guess I'll have to find
> sometime and knock something out for you. Although you would have to apply a
> high load (at least 20 per second) to reproduce it.
>

If you write the application , I can try create the load app.

regards
Richard.

>
[att1.html]

imranbohoran
Offline
Joined: 2008-02-20

Thanks Richard and Tim for your responses.

@Richard
The technology stack I have for this is spring, dwr and hibernate.
The web-application is packaged into a ear file amongst other web-apps.
The JDK been used is 1.6.0_15

I can create a sample application, that represents the functionality, but unfortunately not quick enough amongst the other things going on. In terms of reproducing this a high load needs to be applied though. If this is the last resort to getting to the bottom of it, I guess I'll have to find sometime and knock something out for you. Although you would have to apply a high load (at least 20 per second) to reproduce it.

@Tim
Getting on to No.2 of your comments, my application doesn't open an explicit stream to the jar file I've seen in these streams. This requests are issued against a specific servlet (A DWR servlet) which is in the jar file I see been opened.
And from what I've seen in the DWR servlet code, there's nothing that's opening any resource to this jar file. So I would imagine this to be standard servlet execution.
I see a method getResourceAsStream(final String name) in the EJBClassLoader which seems to be creating new SentinelInputStream objects. But I can't claim I've figured out where and who actually calls this method - perhaps you might be able to shed some light on that.
And then there is a closeOpenStreams() method which is getting called on the done() method of EJBClassLoader which says (from the javadoc) to be assisting garbage collection. Again, not sure where and what calls this method. A check on the doneCalled variable seems to return from the method without going onto closing streams. Not sure if this has any wired behaviour.
I'm just thinking out loud here looking only at the EJBClassLoader. So bare with me if this sounds more like a rant.
I've been trying to get my hands on the glassfish code for the version I'm using, and I must be looking at the wrong places because I cant find out where I can get it from. Would you be able to point me to any svn repositories (if any) where I can find it.
Let me know if what I've mentioned above makes any sense to you from your know-how of the EJBClassLoader. Thanks!!

Oh also, as an aside - I've done some profiling of this user journey (admittedly on tomcat and not with the same load) and haven't noticed any memory leaks.

Cheers
-- Imran

tjquinn
Offline
Joined: 2005-03-30

I could not tell from your latest post whether you tried setting a breakpoint at the SentinelInputStream constructor and then look at the whole stack at that point.

I know that EJBClassLoader will be the immediate caller to the constructor. What we want to find out is the call structure upstream from there. What code is causing the class loader to create the sentinel stream? That should begin to point in the right direction. So if you can set a breakpoint there and examine the call stack for the suspended thread that might help.

The done method is typically invoked by the container when the application is unloaded (disabled) or undeployed or during app server shutdown. I cannot remember if done is also invoked if the EJBClassLoader's finalize method is invoked. But in any case the intent is to clean up any streams that were opened by the class loader on behalf of the application but had not been closed already.

- Tim

imranbohoran
Offline
Joined: 2008-02-20

I was looking at the EJBClassLoader from the web. And my earlier post was based on a perusal of the EJBClassLoader alone.

So basically what we are looking for is some code (be it in Glassfish or any framework surrounding my application) calling the getResourceAsStream method on the classloader.

So looks like this is not straightforward as 'ah I've had that one' or 'the config is probably wrong'. I guess I'll have to hookup the IDE to a glassfish container to find out the call stack. Thanks for your inputs. And will post again when I have got to this.

If anyone else can easily get to this before me, or has time to spare to find out this call stack before me please add on to this.

Cheers
-- Imran

tjquinn
Offline
Joined: 2005-03-30

No, this is not something that others have reported, which is why it will be useful to see the stack trace of calls into the SentinelInputStream constructor. My suspicion is that the application or a framework it uses (or perhaps GlassFish itself) is invoking getResource or getResourceAsStream, directly or indirectly, which ultimately result in the creation of SentinelInputStreams. The stack traces will confirm or refute my suspicion.

- Tim

imranbohoran
Offline
Joined: 2008-02-20

Hi Tim
I've been distracted away from this for a couple of weeks and managed to squeeze some time to look at it again.
I attached my debugger on to a Glassfish container running the piece of functionality in concern and stuck a break point in the EJBClassLoader.getResourceAsStream() method (as that seemed to be the place a new SentinelInputStream was built). I got quite a few paths hitting this break point during the start up of the application. Mostly (if not all) to do with properties files that are managed in spring. But I never got anything that showed the file which was showing up in the heap dump which had the 10k plus SentinelInputSteam objects.
I ran the functionality in concern (although not under load), and still didn't have any thing hitting my break point.
The file that shows up in all of the SentinelInputStream objects is one jar file (dwr.jar), and the GC Roots (through the yourkit profiler) shows the originating path to be the web-application in concern. I've got other web-apps using the same framework mix (dwr, spring, hibernate) from which I haven't had an issue with. Admittedly they never go under the type of load this particular web application goes through. My deployment is done as a single .ear which has all libraries in one place to which all web-apps refer to.
Given that the SentinelInputStream was holding on to dwr.jar, I'm wondering if this is in fact created by Glassfish and not getting closed. But I don't have anything solid to back that up. So I'm not quite sure where to go from here.
Since I have to run this web-app under this load and since the heap dump was pointing to the EJBClassLoader I've pulled this out of my uber ear deployment artifact and deployed as a separate war file in a completely different cluster. And interestingly I've not see any issues since this was done. I've been keeping an eye on the memory utilisations and everything seems to be collecting properly. I took a heap dump to check what's going on inside and don't see any references to SentinelInputStream.
So I'm not sure why I saw this when it was deployed under the ear, which has a different class loader hierarchy. Is it possible that there is something to do with the EJBClassLoader that's causing this?

Also @preston - No I don't have any JAXB stuff. I checked the DWR code (as I use DWR) to see if it has any JAXB, but nothing that I could find.

Thanks for sticking with me so far.

Cheers
-- Imran

tjquinn
Offline
Joined: 2005-03-30

It has been a long time since I looked at the logic in EJBClassLoader, so I am not sure how helpful this will be. But I'll try.

The basic purpose of the sentinel streams is twofold.

1. When an app is unloaded or undeployed, its classloader (an EJBClassLoader even if the app has nothing to do with EJBs) is notified to clean up. Part of that clean-up work is to close any streams that GlassFish or the application opened through that class loader (for example, through getResourceAsStream) but has not closed explicitly. To do that the class loader records all such streams.

2. When a stream is closed in an unusual way - for example, either as the class loader is cleaning up or when the stream is finalized (which of course is not guaranteed to happen in a timely way if ever) - then that means GlassFish or the app has not closed the stream explicitly itself. That's a bug, and the sentinel stream reports where it was opened from to help track down the leak.

It is possible you have found a bug in this logic, although I do not think that GlassFish will by itself use the class loader to open a stream for each request. It's also possible as your app is running that it is opening streams - perhaps indirectly (via getResource and then opening a stream using the URL, or via getResourceAsStream) - but not closing them. This would cause the accumulation of streams you describe.

To track this down further you might try starting GlassFish with the debugger. Then attach to it using a debugger (from an IDE for example). Run a request or two through your app, then set a method breakpoint at the SentinelInputStream constructor. SentinelInputStream is an inner class inside EJBClassLoader. Then submit another request and see if and from where the constructor is invoked. That might provide some hints where to look.

- Tim

imranbohoran
Offline
Joined: 2008-02-20

Also, looking at the elements of the 'streams' Vector, the SentinelInputStream (around 10k of them) point to the same jar file (a dwr jar file). The constant requests (about 20 per second) to the new web-app are dwr calls which tunnels through the dwr servlet within this dwr jar file that is opened. Is there any reason why there would be so many elements in this steams Vector that doesn't seem to be getting closed?

Appreciate any responses.

Cheers
-- Imran

imranbohoran
Offline
Joined: 2008-02-20

Any one with anything on this please? I was hoping someone from the Glassfish team will be able to give me some sought of indication on what this might be.

As I said, the multiple SentinelInputStream objects are crowding the old generation on the heap. And I'm not sure why the EJBClassLoader is keeping a reference to it. I'm assuming that a stream is opened for every request that comes in. But I cant think why? Even if it does, I don't know why the streams stay opened.

Any inputs from the Glassfish team or anyone else who has seen this problem would be very much appreciated. Even if its to say that the way I'm looking at it is bonkers (ofcourse with reason to say why that would be).

I've now disabled the requests hitting my new web-application, and my heap usage is a happy camper now. Which tells me that it's definetely down to the huge amount of requests I was taking in from this new feature.

Would love to hear if it's some configuration I've got wrong, or if I got the deployment packaging wrong or if there's some bug in the GF version I'm using which is v2.1 (9.1.1) (build b60e-fcs)

Thanks!!

Cheers
-- Imran

Richard Kolb

Hi Imran

Sorry no one has got back to you.
Firstly can you indicate which technologies you are using eg. JSF, Remote
EJB, Local EJB, JPA, JMS, WebServices etc.

Then is you JDK up to date ?
(There are memory leaks in the JDK that effect GlassFish)

Lastly can you create a small application to reproduce this.
In my experience , if you can create a small test app , the GlassFish
developers can more easily trace the issue.

regards
Richard

On 28 October 2010 13:48, wrote:

> Any one with anything on this please? I was hoping someone from the
> Glassfish team will be able to give me some sought of indication on what
> this might be.
>
> As I said, the multiple SentinelInputStream objects are crowding the old
> generation on the heap. And I'm not sure why the EJBClassLoader is keeping a
> reference to it. I'm assuming that a stream is opened for every request that
> comes in. But I cant think why? Even if it does, I don't know why the
> streams stay opened.
>
> Any inputs from the Glassfish team or anyone else who has seen this problem
> would be very much appreciated. Even if its to say that the way I'm looking
> at it is bonkers (ofcourse with reason to say why that would be).
>
> I've now disabled the requests hitting my new web-application, and my heap
> usage is a happy camper now. Which tells me that it's definetely down to the
> huge amount of requests I was taking in from this new feature.
>
> Would love to hear if it's some configuration I've got wrong, or if I got
> the deployment packaging wrong or if there's some bug in the GF version I'm
> using which is v2.1 (9.1.1) (build b60e-fcs)
>
> Thanks!!
>
> Cheers
> -- Imran
> [Message sent by forum member 'imranbohoran']
>
> http://forums.java.net/jive/thread.jspa?messageID=486413
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>
>
[att1.html]