Skip to main content

"Too many open files" problem

10 replies [Last post]
edek234
Offline
Joined: 2007-03-24

Hello,

after leaving the glassfish (9.1 (build b33e-beta)) running for a few hours it runs into "too many open files" problem. After that there is a mess.

I suspect it has something to do with JMS, because this is what I recently added. Every 60 seconds a timer generates a message into a queue, which is handled. I always close the producer, the session and the connection (when sending). Handling the message is currently doing nothing (or almost nothing, depending on configuration), but always succeeds.

What I find suspicious is that every few minutes there is a following trace, always twice (does it mean it generations new and new connections?):
[#|2007-05-03T23:32:54.107+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=34;_ThreadName=Timer-4;_RequestID=a17dfebf-c2c
c-4850-a198-e22cfe7776ac;|
MQRA:MC:Warning:createConnection API used w/o username, password for Application Auth|#]

[#|2007-05-03T23:32:54.136+0200|INFO|sun-appserver9.1|javax.resourceadapter.mqjmsra.outbound.connection|_ThreadID=34;_ThreadName=Timer-4;|MQJMSRA_MC
1101: constructor:Created mcId=43:xacId=6560357433579922176:Using xacf config={imqDefaultUsername=guest, imqBrokerHostName=localhost, imqSetJMSXUser
ID=false, imqJMSExpiration=0, imqQueueBrowserRetrieveTimeout=60000, imqSSLProviderClassname=com.sun.net.ssl.internal.ssl.Provider, imqConfiguredClie
ntID=, imqSSLIsHostTrusted=false, imqSetJMSXProducerTXID=false, imqJMSDeliveryMode=PERSISTENT, imqAddressListBehavior=RANDOM, imqEnableSharedClientI
D=false, imqAckOnProduce=, imqConnectionURL=http://localhost/imq/tunnel, imqConnectionFlowLimit=1000, imqSetJMSXAppID=false, imqAddressListIteration
s=3, imqBrokerServicePort=0, imqSetJMSXRcvTimestamp=false, imqReconnectInterval=5000, imqConnectionType=TCP, imqLoadMaxToServerSession=true, imqSetJ
MSXConsumerTXID=false, imqAddressList=localhost:7676, imqOverrideJMSDeliveryMode=false, imqPingInterval=30, imqDefaultPassword=guest, imqAckOnAcknow
ledge=, imqJMSPriority=4, imqQueueBrowserMaxMessagesPerRetrieve=1000, imqOverrideJMSPriority=false, imqConsumerFlowLimit=1000, imqBrokerHostPort=767
6, imqReconnectEnabled=true, imqConnectionFlowCount=100, imqReconnectAttempts=3, imqConnectionFlowLimitEnabled=false, imqAckTimeout=0, imqBrokerServ
iceName=, imqOverrideJMSExpiration=false, imqConnectionHandler=com.sun.messaging.jmq.jmsclient.protocol.tcp.TCPStreamHandler, imqDisableSetClientID=
false, imqOverrideJMSHeadersToTemporaryDestinations=false, imqConsumerFlowThreshold=50}|#]

Any ideas?

Thanks,
Edek

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
johanvos
Offline
Joined: 2005-07-05

I am using JDK 1.5, and an application that ran on pre-b33 builds without problems, had the "Too many open files" problems after about 48 hours.

On my Linux distribution, the max number of open files is 1024 by default (for root and non-root). Changing that to 2048 solves the problem (at least for 72 hours and counting ;)).

I wonder if something has changed in b33, causing more file descriptors to be used?

- Johan

Alan Frechette

Solution to the Too many open files problem.

I have further input on the too many open files issue. On SOLARIS the
soft and hard limit for number of open files is 256 for a non root user.

For a root user the soft limit is 256 and the hard limit is 65536.

So if you start the GF Appserver as root you will have a hard limit of
65536. You will not get too many open files.

If on the other hand you start the GF Appserver as a non root user you
will have a hard limit of 256. This will most likely cause too many open
files from appserver.

WORKAROUNDS
-----------

1) Run the GF Appserver as root. You will get the 65536 (64K) hard limit

2) If you run the GF Appserver as a yourself do the following. From your
terminal window do an "rlogin back into your system". This will give
you a "in.rlogind" process that is owned by root. You can then start
the GF Appserver as yourself. This will give you a hard limit of 64K
because the "in.rlogind" process is owned by root. I have tried this
and it works. So to summarize:

% rlogin host -l yourself
% asadmin start-domain //Your GF Appserver will now have 64K
//open file limit

glassfish@javadesktop.org wrote:
> Hello,
>
> after leaving the glassfish (9.1 (build b33e-beta)) running for a few hours it runs into "too many open files" problem. After that there is a mess.
>
> I suspect it has something to do with JMS, because this is what I recently added. Every 60 seconds a timer generates a message into a queue, which is handled. I always close the producer, the session and the connection (when sending). Handling the message is currently doing nothing (or almost nothing, depending on configuration), but always succeeds.
>
> What I find suspicious is that every few minutes there is a following trace, always twice (does it mean it generations new and new connections?):
> [#|2007-05-03T23:32:54.107+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=34;_ThreadName=Timer-4;_RequestID=a17dfebf-c2c
> c-4850-a198-e22cfe7776ac;|
> MQRA:MC:Warning:createConnection API used w/o username, password for Application Auth|#]
>
> [#|2007-05-03T23:32:54.136+0200|INFO|sun-appserver9.1|javax.resourceadapter.mqjmsra.outbound.connection|_ThreadID=34;_ThreadName=Timer-4;|MQJMSRA_MC
> 1101: constructor:Created mcId=43:xacId=6560357433579922176:Using xacf config={imqDefaultUsername=guest, imqBrokerHostName=localhost, imqSetJMSXUser
> ID=false, imqJMSExpiration=0, imqQueueBrowserRetrieveTimeout=60000, imqSSLProviderClassname=com.sun.net.ssl.internal.ssl.Provider, imqConfiguredClie
> ntID=, imqSSLIsHostTrusted=false, imqSetJMSXProducerTXID=false, imqJMSDeliveryMode=PERSISTENT, imqAddressListBehavior=RANDOM, imqEnableSharedClientI
> D=false, imqAckOnProduce=, imqConnectionURL=http://localhost/imq/tunnel, imqConnectionFlowLimit=1000, imqSetJMSXAppID=false, imqAddressListIteration
> s=3, imqBrokerServicePort=0, imqSetJMSXRcvTimestamp=false, imqReconnectInterval=5000, imqConnectionType=TCP, imqLoadMaxToServerSession=true, imqSetJ
> MSXConsumerTXID=false, imqAddressList=localhost:7676, imqOverrideJMSDeliveryMode=false, imqPingInterval=30, imqDefaultPassword=guest, imqAckOnAcknow
> ledge=, imqJMSPriority=4, imqQueueBrowserMaxMessagesPerRetrieve=1000, imqOverrideJMSPriority=false, imqConsumerFlowLimit=1000, imqBrokerHostPort=767
> 6, imqReconnectEnabled=true, imqConnectionFlowCount=100, imqReconnectAttempts=3, imqConnectionFlowLimitEnabled=false, imqAckTimeout=0, imqBrokerServ
> iceName=, imqOverrideJMSExpiration=false, imqConnectionHandler=com.sun.messaging.jmq.jmsclient.protocol.tcp.TCPStreamHandler, imqDisableSetClientID=
> false, imqOverrideJMSHeadersToTemporaryDestinations=false, imqConsumerFlowThreshold=50}|#]
>
> Any ideas?
>
> Thanks,
> Edek
> [Message sent by forum member 'edek234' (edek234)]
>
> http://forums.java.net/jive/thread.jspa?messageID=215579
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

--

Regards,

______
/_____/\
/_____\\ \
/_____\ \\ / Alan E. Frechette
/_____/ \/ / / Sun Microsystems Computer Company
/_____/ / \//\ One Network Drive
\_____\//\ / / Burlington, MA 01803-0902
\_____/ / /\ /
\_____/ \\ \ Email: alan.frechette@East.Sun.COM
\_____\ \\ Phone (781) 442-0723 Fax (781) 442-1610
\_____\/ Web site...http://www.sun.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Binod

Are you on a linux machine? Are you using JDK 5?

- Binod.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

edek234
Offline
Joined: 2007-03-24

java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode)

on Fedora 6

I can try i586 version of 1.6.0, or x86_64 of 1.5.0 (these I have installed). Which one should I try?

Edek

edek234
Offline
Joined: 2007-03-24

Hello,

I filed a bug report
https://glassfish.dev.java.net/issues/show_bug.cgi?id=2979

What I tested: I set the timer (see my first post) to 1 second (it fired actually every 7 seconds), and both the problem and the traces (see first post) did not appear for about 24 hours. I set the timer back to 60 sec, and the traces and the problem appeared again.
What the timer does: it puts a message into queue; the message is handled, by reading a few configuration entities and currently it does nothing more.

Edek

Binod

Can you please also use "lsof" and see what the leak is?
You can use "lsof -p " to find out the files opened by the
appserver process.

You can use "lsof -p -r " to periodically check
whats happening,

Thanks for the issue report!

- Binod.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

Alan Frechette

Binod all you need to do is grab the cts and webservices subset bundle
and run the webservices12 tree of tests on the nightly GF and the
condition will happen right away.

alan

Binod wrote:
> Can you please also use "lsof" and see what the leak is?
> You can use "lsof -p " to find out the files opened by the
> appserver process.
>
> You can use "lsof -p -r " to periodically check
> whats happening,
>
> Thanks for the issue report!
>
> - Binod.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
> For additional commands, e-mail: users-help@glassfish.dev.java.net
>

--

Regards,

______
/_____/\
/_____\\ \
/_____\ \\ / Alan E. Frechette
/_____/ \/ / / Sun Microsystems Computer Company
/_____/ / \//\ One Network Drive
\_____\//\ / / Burlington, MA 01803-0902
\_____/ / /\ /
\_____/ \\ \ Email: alan.frechette@East.Sun.COM
\_____\ \\ Phone (781) 442-0723 Fax (781) 442-1610
\_____\/ Web site...http://www.sun.com

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@glassfish.dev.java.net
For additional commands, e-mail: users-help@glassfish.dev.java.net

edek234
Offline
Joined: 2007-03-24

Anyway, I attached lsof output to bug report.

To me it does not tell much: some pipes + plus one tcp connection (all repeated).

Edek

>> Can you please also use "lsof" and see what the leak is?
>> You can use "lsof -p " to find out the files opened by the
>> appserver process.

rampsarathy
Offline
Joined: 2005-11-23

Hi Edek,

Could you try using JDK 1.5.0,

This JDK 1.6 CR is of interest here
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6533291

Also i was able to run the webservices12 tree of CTS tests with 79 pass and 23 failures across b46 of glassfish using jdk 1.5.

Thanks
-Ramesh

edek234
Offline
Joined: 2007-03-24

Hi Ramesh,

no problem with jdk 1.5.0.

I tried to extract a testcase to reproduce the problem and I did not manage to reproduce it.

If I set open files limit to 2048, I get no problem with my application using jdk 1.6.0, and lsof does not report constantly growing resource usage. So, it probably not a leak, but just being too close to the limit, and the CR you mention is probably one of the factors causing the problem.

Thanks,
Edek