Skip to main content

ssl_error_rx_malformed_cert_request with two-way ssl authentication

9 replies [Last post]
gabosu
Offline
Joined: 2012-09-06

On Glassfish 3.1.1 I have two-way ssl authentication and cacerts.jks has 498 certificates now.
When I have 516 entries (size 487KB) the server is starting but when I try to connect is rising
Secure Connection Failed
SSL received a malformed Certificate Request handshake message
Error code: ssl_error_rx_malformed_cert_request
I tried ti increase the allocated memory - the same result.
The only way to make it to run is to delete one certificate.
Any idea?
Regards,
Gabriel

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
gabosu
Offline
Joined: 2012-09-06

Attached is the server.log at the moment of the error. I can't see any details in order to be able to solve the problem...

oleksiys
Offline
Joined: 2006-01-25

Hi Gabriel,

can you pls. send me certificates and client code (privately), so I can
try to reproduce the problem locally and see if it's something related
to GF server?

Thanks.

WBR,
Alexey.

On 09/09/2012 12:39 PM, forums@java.net wrote:
> Attached is the server.log at the moment of the error. I can't see any
> details in order to be able to solve the problem...
>
> --
>
> [Message sent by forum member 'gabosu']
>
> View Post: http://forums.java.net/node/889917
>
>
>

gabosu
Offline
Joined: 2012-09-06

I made 1000 more dummy CAs and now I have 1500 CAs and cacerts.jks has 1.19 MB - I am receiving the same error.

gabosu
Offline
Joined: 2012-09-06

Seems till ServerHelloDone is ok but after that the trouble is coming - see the log below:

[#|2012-09-08T01:28:38.215+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;||#]
[#|2012-09-08T01:28:38.226+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), WRITE: TLSv1 Handshake, length = 16384|#]
[#|2012-09-08T01:28:38.233+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), WRITE: TLSv1 Handshake, length = 16384|#]
[#|2012-09-08T01:28:38.240+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), WRITE: TLSv1 Handshake, length = 16384|#]
[#|2012-09-08T01:28:38.248+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), WRITE: TLSv1 Handshake, length = 16384|#]
[#|2012-09-08T01:28:38.249+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|*** ServerHelloDone|#]
[#|2012-09-08T01:28:38.252+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), WRITE: TLSv1 Handshake, length = 5579|#]
[#|2012-09-08T01:28:38.400+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), READ: TLSv1 Alert, length = 2|#]
[#|2012-09-08T01:28:38.401+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1)|#]
[#|2012-09-08T01:28:38.402+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|, RECV TLSv1 ALERT: |#]
[#|2012-09-08T01:28:38.402+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|fatal, |#][#|2012-09-08T01:28:38.404+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|decode_err
or|#]
[#|2012-09-08T01:28:38.405+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: decode_error|#]
[#|2012-09-08T01:28:38.406+0300|INFO|glassfish3.1.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=20;_ThreadName=Thread-2;|http-thread-pool-8181(1), fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: decode_error|#]

gabosu
Offline
Joined: 2012-09-06

A situation like this one you can find at
https://savannah.cern.ch/bugs/?69554
when CAs are between 500 and 1000.

gabosu
Offline
Joined: 2012-09-06

Hi,
With 516 CAs the server is returning ssl_error_rx_malformed_cert_request.
Does not matter which CA is deleted, with 515 CAs also is not working.
With 514 CAs is working.
I made several tests deleting randomly 2 CAs and is always working.
I will put a detailed log on SSL to go further into the problem.
I found out a thread for TomCat and they had the same problem for 500-1000 certificates - a problem with java-util which is generating a memory leak. I will put more CAs to see if over 1000 is doing the same.
Regards,
Gabriel

wetmore
Offline
Joined: 2005-04-29

The root cause is a limitation of TLS unfortunately. It can only transport up to 2^16-1 bytes of CA Distinguished name information. http://www.rfc-editor.org/rfc/rfc2246.txt section 7.4.4.

SunJSSE is not checking this when it is encoding, and is wrapping the value sent if the list gets above 64KB. i.e. it is sending a truncated length value. It needs to throw an exception here if it wraps.

If the number of bytes read doesn't match up with the advertised header, the peer will fail on reads.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7200295

HTH.

mgainty
Offline
Joined: 2004-05-21

The certificate you are referencing in your truststore is either
self-signed which will work ONLY on localhost
fraudulent which means the attributes used to create the cert do not match the credentials of either the CA client or server

You should contact your CA-Authority (which should be verisign or thawte) and re-acquire a valid certificate for 2way SSL authentication

Mit Freundlichen Gruben
Martin
______________________________________________
Verzicht und Vertraulichkeitanmerkung

Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.

> To: users@glassfish.java.net
> Subject: ssl_error_rx_malformed_cert_request with two-way ssl authentication
> From: forums@java.net
> Date: Thu, 6 Sep 2012 08:45:17 -0500
>
> On Glassfish 3.1.1 I have two-way ssl authentication and cacerts.jks has 498
> certificates now. When I have 516 entries (size 487KB) the server is starting
> but when I try to connect is rising Secure Connection Failed SSL received a
> malformed Certificate Request handshake message Error code:
> ssl_error_rx_malformed_cert_request I tried ti increase the allocated memory
> - the same result. The only way to make it to run is to delete one
> certificate. Any idea? Regards, Gabriel
>
> --
>
> [Message sent by forum member 'gabosu']
>
> View Post: http://forums.java.net/node/889917
>
>

gabosu
Offline
Joined: 2012-09-06

I have a GoDaddy.com Standard SSL Certificate valid for one site and I can't see any restriction or a more advanced type of certificate for 2way ssl.
Anyway, I have big doubts your answer is good - it is not explaining why this certificate is
working till at a moment and is not working after a threshold.
Regards,
Gabriel