Skip to main content

Glassfish_v2 DoS while using webservices

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
4 replies [Last post]
supererwin
Offline
Joined: 2012-02-27

Hello!

I am no Java-Expert, but still I need to cope with a problem on a Glassfish-Server running Glassfishv2 or - as far as I understood so far - ApplicationServer 9.1.

We are using the Glassfish-Server to provide database access via a webservice interface, meaning that another server can send datapackets containing webservice requests and the glassfish server then examines the mysql database that it is connected to and sends the data received to the other server as an answer to its request. So far, so good.

The problem that I see is, that after some time, the glassfish-server's webservice is not available anymore. The glassfish-server receives webservice-requests quite often(~130 times per second) and after some minutes(less than 5 minutes in general), the webservice is not available anymore. Anyway, I can still do a "/opt/glassfish/bin/asadmin ping-connection-pool", which returns that the ping was done successfully.

I am only able to reproduce this error with one specific webservice-questioning program. If I use another one, that also asks for the webservices available, I can't make it crash - even after one full weekend of testing.

My question is rather where to search for an error than that I'd be waiting for a solution of my problem. I simply don't know, where to look and what to check. I'd also like to know, if there are any "common" points, that should be adjusted if a glassfish server denies to answer to webservice-requests...

I'd be really happy to receive any single hint.

Best regards from Germany,

Dennis

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Kumar Jayanti Guest
Offline
Joined: 2011-04-02

when you say not available anymore what does that mean, is that a hang or does the client receive a 404 not-found ?. If its a hang, then you should take a thread dump and send it to this list.
On 27-Feb-2012, at 7:05 PM, forums@java.net wrote:

> Hello!
>
> I am no Java-Expert, but still I need to cope with a problem on a
> Glassfish-Server running Glassfishv2 or - as far as I understood so far -
> ApplicationServer 9.1.
>
> We are using the Glassfish-Server to provide database access via a webservice
> interface, meaning that another server can send datapackets containing
> webservice requests and the glassfish server then examines the mysql database
> that it is connected to and sends the data received to the other server as an
> answer to its request. So far, so good.
>
> The problem that I see is, that after some time, the glassfish-server's
> webservice is not available anymore. The glassfish-server receives
> webservice-requests quite often(~130 times per second) and after some
> minutes(less than 5 minutes in general), the webservice is not available
> anymore. Anyway, I can still do a "/opt/glassfish/bin/asadmin
> ping-connection-pool", which returns that the ping was done successfully.
>
> I am only able to reproduce this error with one specific
> webservice-questioning program. If I use another one, that also asks for the
> webservices available, I can't make it crash - even after one full weekend of
> testing.
>
> My question is rather where to search for an error than that I'd be waiting
> for a solution of my problem. I simply don't know, where to look and what to
> check. I'd also like to know, if there are any "common" points, that should
> be adjusted if a glassfish server denies to answer to webservice-requests...
>
>
>
> I'd be really happy to receive any single hint.
>
> Best regards from Germany,
>
> Dennis
>
>
> --
>
> [Message sent by forum member 'supererwin']
>
> View Post: http://forums.java.net/node/883810
>
>

oleksiys
Offline
Joined: 2006-01-25

Hi Dennis,

just to make sure, when you say webservice becomes unavailable, what
about other service(s)? Can you still access them at the same time?
Can you pls. take a thread dump (using jstack) at time you see the
webservice is not available, so we can check the threads status?

Thanks.

WBR,
Alexey.

On 02/27/2012 05:35 AM, forums@java.net wrote:
> Hello!
>
> I am no Java-Expert, but still I need to cope with a problem on a
> Glassfish-Server running Glassfishv2 or - as far as I understood so far -
> ApplicationServer 9.1.
>
> We are using the Glassfish-Server to provide database access via a
> webservice
> interface, meaning that another server can send datapackets containing
> webservice requests and the glassfish server then examines the mysql
> database
> that it is connected to and sends the data received to the other
> server as an
> answer to its request. So far, so good.
>
> The problem that I see is, that after some time, the glassfish-server's
> webservice is not available anymore. The glassfish-server receives
> webservice-requests quite often(~130 times per second) and after some
> minutes(less than 5 minutes in general), the webservice is not available
> anymore. Anyway, I can still do a "/opt/glassfish/bin/asadmin
> ping-connection-pool", which returns that the ping was done successfully.
>
> I am only able to reproduce this error with one specific
> webservice-questioning program. If I use another one, that also asks
> for the
> webservices available, I can't make it crash - even after one full
> weekend of
> testing.
>
> My question is rather where to search for an error than that I'd be
> waiting
> for a solution of my problem. I simply don't know, where to look and
> what to
> check. I'd also like to know, if there are any "common" points, that
> should
> be adjusted if a glassfish server denies to answer to
> webservice-requests...
>
>
>
> I'd be really happy to receive any single hint.
>
> Best regards from Germany,
>
> Dennis
>
>
> --
>
> [Message sent by forum member 'supererwin']
>
> View Post: http://forums.java.net/node/883810
>
>

supererwin
Offline
Joined: 2012-02-27

Hey Alexey,

first off thank you for your cooperation. I put the output of jstack <pid> to this pastebin-link:

pastebin.com/fPsjA9WU

All other processes are still avaiable - if you mean such things like ssh or mysql, but I am not able to reach the glassfish-welcome link (not at port 4848 nor at port 8080). I don't know, if this is related to the apache not responding anymore, or if the glassfish has ended working.

One more thing, that I observed: "netstat -an | grep 8080" gives LOTS(~120) of close_wait states and "lsof | grep glass | wc -l" gives me around 260 files everytime I observe the error. I already tried to tuned the apache2 a little(by setting the "KeepAlive Off"-option in its config) and I set some options of the glassfish(idle timeout and maxwaittime of the connection pool), but nothing influenced the behaviour. Of course, I did each step one-by-one in order to find, if any option helped out an not all at once.

Hmmm - so far from my side.

Best regards,

Dennis

garamirez
Offline
Joined: 2012-05-08

Hi !

I'm facing the same issue with no luck so far.. Did you had any luck solving thus issue ?

Thanks!!