Skip to main content

Need some help and advice on tuning an apache - to - glassfish proxy

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
2 replies [Last post]
Meatwad
Offline
Joined: 2012-01-03

I have a java web application running in Glassfish 3.1.1 on port 8080. I have Apache running on the same server on port 80 acting as a reverse proxy (using ajp) to Glassfish. Here is my configuration (and forgive the formatting but the pre tags don't seem to work on this forum):

In glassfish:
network-listener port="8009" protocol="http-listener-1" transport="tcp" name="jk-connector" jk-enabled="true" thread-pool="http-thread-pool"

In apache:
ProxyRequests Off
ProxyPass /client ajp://10.72.8.167:8009/client timeout=1200 ttl=1200

The problem occurs when I have around 10-15 concurrent connections I start seeing server unavailable messages in client browsers and this in the logs:

In glassfish domain log:
[#|2013-03-28T10:02:15.742-0500|SEVERE|glassfish3.1.1|org.apache.tomcat.util.threads.ThreadPool|_ThreadID=22;_ThreadName=Thread-2;|threadpool.busy|#]

In apache error_log:
[Thu Mar 28 10:13:01 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:13:01 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:13:01 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)
[Thu Mar 28 10:13:13 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:13:13 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:13:13 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)

[Thu Mar 28 10:13:35 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:13:35 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:13:35 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.72.8.167:8009 (10.72.8.167)

[Thu Mar 28 10:15:14 2013] [error] ap_proxy_connect_backend disabling worker for (10.72.8.167)
[Thu Mar 28 10:15:14 2013] [error] proxy: AJP: failed to make connection to backend: 10.72.8.167
[Thu Mar 28 10:15:15 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:15:15 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:15:15 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)
[Thu Mar 28 10:15:17 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:15:17 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:15:17 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)
[Thu Mar 28 10:15:18 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:18 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:15:18 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:15:18 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)
[Thu Mar 28 10:15:18 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:19 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:15:19 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:15:19 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:15:24 2013] [error] ajp_read_header: ajp_ilink_receive failed
[Thu Mar 28 10:15:24 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from (null) (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for (10.72.8.167)
[Thu Mar 28 10:15:28 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
[Thu Mar 28 10:15:28 2013] [error] ajp_read_header: ajp_ilink_receive failed

Can anyone help me figure this out or where to go from here?

Thanks

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
amyroh
Offline
Joined: 2004-05-06

You should be able to resolve this issue by increasing the
max-thread-pool-size to be at least the value of Apache maximum number
of connections. Or you can also set Apache's maximum number of
connections that will be allowed to the backend server by using the max
parameter.

asadmin set
configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=20
ProxyPass / ajp://mydomain:8009/ max=20

Amy

On 3/29/13 6:10 AM, forums@java.net wrote:
> I have a java web application running in Glassfish 3.1.1 on port 8080.
> I have
> Apache running on the same server on port 80 acting as a reverse proxy
> (using
> ajp) to Glassfish. Here is my configuration (and forgive the
> formatting but
> the pre tags don't seem to work on this forum): In glassfish: In apache:
> ProxyRequests Off ProxyPass /client ajp://10.72.8.167:8009/client
> timeout=1200 ttl=1200 The problem occurs when I have around 10-15
> concurrent
> connections I start seeing server unavailable messages in client
> browsers and
> this in the logs: In glassfish domain log:
> [#|2013-03-28T10:02:15.742-0500|SEVERE|glassfish3.1.1|org.apache.tomcat.util.threads.ThreadPool|_ThreadID=22;_ThreadName=Thread-2;|threadpool.busy|#]
>
> In apache error_log: [Thu Mar 28 10:13:01 2013] [error]
> (104)Connection reset
> by peer: ajp_ilink_receive() can't receive header [Thu Mar 28 10:13:01
> 2013]
> [error] ajp_read_header: ajp_ilink_receive failed [Thu Mar 28 10:13:01
> 2013]
> [error] (120006)APR does not understand this error code: proxy: read
> response
> failed from (null) (10.72.8.167) [Thu Mar 28 10:13:13 2013] [error]
> (104)Connection reset by peer: ajp_ilink_receive() can't receive
> header [Thu
> Mar 28 10:13:13 2013] [error] ajp_read_header: ajp_ilink_receive
> failed [Thu
> Mar 28 10:13:13 2013] [error] (120006)APR does not understand this error
> code: proxy: read response failed from (null) (10.72.8.167) [Thu Mar 28
> 10:13:35 2013] [error] (104)Connection reset by peer: ajp_ilink_receive()
> can't receive header [Thu Mar 28 10:13:35 2013] [error] ajp_read_header:
> ajp_ilink_receive failed [Thu Mar 28 10:13:35 2013] [error]
> (120006)APR does
> not understand this error code: proxy: read response failed from
> 10.72.8.167:8009 (10.72.8.167) [Thu Mar 28 10:15:14 2013] [error]
> ap_proxy_connect_backend disabling worker for (10.72.8.167) [Thu Mar 28
> 10:15:14 2013] [error] proxy: AJP: failed to make connection to backend:
> 10.72.8.167 [Thu Mar 28 10:15:15 2013] [error] (104)Connection reset
> by peer:
> ajp_ilink_receive() can't receive header [Thu Mar 28 10:15:15 2013]
> [error]
> ajp_read_header: ajp_ilink_receive failed [Thu Mar 28 10:15:15 2013]
> [error]
> (120006)APR does not understand this error code: proxy: read response
> failed
> from (null) (10.72.8.167) [Thu Mar 28 10:15:17 2013] [error]
> (104)Connection
> reset by peer: ajp_ilink_receive() can't receive header [Thu Mar 28
> 10:15:17
> 2013] [error] ajp_read_header: ajp_ilink_receive failed [Thu Mar 28
> 10:15:17
> 2013] [error] (120006)APR does not understand this error code: proxy:
> read
> response failed from (null) (10.72.8.167) [Thu Mar 28 10:15:18 2013]
> [error]
> proxy: AJP: disabled connection for (10.72.8.167) [Thu Mar 28 10:15:18
> 2013]
> [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive
> header [Thu Mar 28 10:15:18 2013] [error] ajp_read_header:
> ajp_ilink_receive
> failed [Thu Mar 28 10:15:18 2013] [error] (120006)APR does not understand
> this error code: proxy: read response failed from (null) (10.72.8.167)
> [Thu
> Mar 28 10:15:18 2013] [error] proxy: AJP: disabled connection for
> (10.72.8.167) [Thu Mar 28 10:15:19 2013] [error] (104)Connection reset by
> peer: ajp_ilink_receive() can't receive header [Thu Mar 28 10:15:19 2013]
> [error] ajp_read_header: ajp_ilink_receive failed [Thu Mar 28 10:15:19
> 2013]
> [error] (120006)APR does not understand this error code: proxy: read
> response
> failed from (null) (10.72.8.167) [Thu Mar 28 10:15:24 2013] [error]
> (104)Connection reset by peer: ajp_ilink_receive() can't receive
> header [Thu
> Mar 28 10:15:24 2013] [error] ajp_read_header: ajp_ilink_receive
> failed [Thu
> Mar 28 10:15:24 2013] [error] (120006)APR does not understand this error
> code: proxy: read response failed from (null) (10.72.8.167) [Thu Mar 28
> 10:15:24 2013] [error] proxy: AJP: disabled connection for
> (10.72.8.167) [Thu
> Mar 28 10:15:24 2013] [error] proxy: AJP: disabled connection for
> (10.72.8.167) [Thu Mar 28 10:15:24 2013] [error] proxy: AJP: disabled
> connection for (10.72.8.167) [Thu Mar 28 10:15:24 2013] [error] proxy:
> AJP:
> disabled connection for (10.72.8.167) [Thu Mar 28 10:15:24 2013] [error]
> proxy: AJP: disabled connection for (10.72.8.167) [Thu Mar 28 10:15:24
> 2013]
> [error] proxy: AJP: disabled connection for (10.72.8.167) [Thu Mar 28
> 10:15:28 2013] [error] (104)Connection reset by peer: ajp_ilink_receive()
> can't receive header [Thu Mar 28 10:15:28 2013] [error] ajp_read_header:
> ajp_ilink_receive failed Can anyone help me figure this out or where
> to go
> from here? Thanks
>
> --
>
> [Message sent by forum member 'Meatwad']
>
> View Post: http://forums.java.net/node/896003
>
>

Meatwad
Offline
Joined: 2012-01-03

Thank you, amy. I increased the max thread pool to 300 based on what I read in the documentation here: http://docs.oracle.com/cd/E18930_01/html/821-2431/abehk.html. By the way I found it very strange that the documentation indicated a suitable size was between 100 - 500 and yet the default is 5?!

I also set the max number of proxy connections in apache to 300 like this:
ProxyPass /client ajp://10.72.8.167:8009/client timeout=1200 ttl=1200 max=300

Those 2 changes definitely improved performance and sent my CPU utilization way up. I also found that I was getting OutOfMemory exceptions in my glassfish domain log so I changed the starting memory size from 512m to 1024m through the JVM Options of the admin console.