Skip to main content

GlassFish 2.1.1 - Application client jar is generated corruptly in clustered environment / Java Web Start

6 replies [Last post]
squibber
Offline
Joined: 2012-08-22
Points: 0

We have an EAR about 25 MB in size.

This EAR contains an application client, two EJB projects and two Web projects.

When the EAR is deployed in a non-clustered environment (standalone) everything is fine: the application client jar is generated correctly (size about 20MB). You can launch the client via Java Web Start.

However, when we deploy the EAR in a clustered environment (consists of two nodes with one instance on each node) the generated application client jar -
which can be found under GF_PATH/nodeagents/nodeagent1/instance1/generated/xml/j2ee-apps/EAR-folder.. - shows that the generated "EAR-name"Client.jar is always in a corrupted state. You cannot even open the file with a zip tool etc. This file corruption happens on both instances and is always reproducible.

If you delete the corrupted file on both instances and restart the cluster, the application client jar gets generated again and this time, the file is always generated correctly.

I started to test this further by shutting down one instance in the cluster to see if has something to do with the sync process for each instance during deployment. But the problem remains, it does not matter if on instance is down or not - the generated file is corrupt.

Does anyone else have this problem in a clustered environment? As I mentioned above in a standalone app server we do not have this problem.

Any help appreciated. Thank you.

squibber

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
mgainty
Offline
Joined: 2004-05-21
Points: 0

This is where being in a large organisation can have advantages specifically where development is separate from operations
If the dev guys create an EAR that passses all testcases and works standalone then they might consider that theire code is ready to be promoted
When operational guys find a problem when deploying to a clustered environment they toss it back to the dev guys and say your code doesnt work (when another node is used)

If the dev guys put in a hardcoded resource (hardcoded filename) that Node1 has access to and Node2 does not have access to well then the devs need to place that resource on
a Virtual File or some universally accessible location

if the devs are providing any code that is specific to an environment then they need to make those environment attributes a configurable option in either
*.properties
*.xml
Notice that domain.xml under element name you have the ability to supply multiple properties with name and value attributes e.g.
-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/

autostart folder would contain
org.apache.felix.fileinstall-autodeploy-bundles.cfg
which would contain
# Directory being watched by fileinstall.
felix.fileinstall.dir=${com.sun.aas.instanceRoot}/autodeploy/bundles/
# Time period fileinstaller thread in ms.
felix.fileinstall.poll=5000
# log level
felix.fileinstall.log.level=3
# should new bundles be started or installed only? true => start, false => only install
felix.fileinstall.bundles.new.start=true
# should bundles be started transiently?
felix.fileinstall.bundles.startTransient=true

any new properties files conaining new attributes that should be bundled by felix are located at
${com.sun.aas.instanceRoot}/autodeploy/bundles/

BTW: the rebuild process should also generate fresh jars wars and ears so any problems with corrupt jar/war/ear will not be go away

An interesing deployment challenge,
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité

Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.

> To: users@glassfish.java.net
> Subject: GlassFish 2.1.1 - Application client jar is generated corruptly in clustered environment / Java Web Start
> From: forums@java.net
> Date: Wed, 22 Aug 2012 06:09:30 -0500
>
> We have an EAR about 25 MB in size. This EAR contains an application client,
> two EJB projects and two Web projects. When the EAR is deployed in a
> non-clustered environment (standalone) everything is fine: the application
> client jar is generated correctly (size about 20MB). You can launch the
> client via Java Web Start. However, when we deploy the EAR in a clustered
> environment (consists of two nodes with one instance on each node) the
> generated application client jar - which can be found under
> GF_PATH/nodeagents/nodeagent1/instance1/generated/xml/j2ee-apps/EAR-folder..
> - shows that the generated "EAR-name"Client.jar is always in a corrupted
> state. You cannot even open the file with a zip tool etc. This file
> corruption happens on both instances and is always reproducible. *If you
> delete the corrupted file on both instances and restart the cluster, the
> application client jar gets generated again and this time, the file is always
> generated correctly.* I started to test this further by shutting down one
> instance in the cluster to see if has something to do with the sync process
> for each instance during deployment. But the problem remains, it does not
> matter if on instance is down or not - the generated file is corrupt. Does
> anyone else have this problem in a clustered environment? As I mentioned
> above in a standalone app server we do not have this problem. Any help
> appreciated. Thank you. squibber
>
> --
>
> [Message sent by forum member 'squibber']
>
> View Post: http://forums.java.net/node/889421
>
>

squibber
Offline
Joined: 2012-08-22
Points: 0

@Martin thanks for you feedback, so we do not know what you actually tried to say specific to our case..

1. we use GF 2.1.1, so no felix container is involved
2. we do not think this has anything to do with our infrastructure
3. there is no conflict between our devs or operational guys

squibber

squibber
Offline
Joined: 2012-08-22
Points: 0

I am still struggling on this. Wondering what could impose on the generation of the Client jar so it gets corrupted on each initial deployment.

In the server logs I see

[#|2012-09-10T19:13:31.231+0200|FINE|sun-appserver2.1.1|javax.enterprise.system.tools.deployment|_ThreadID=65;_ThreadName=RMI TCP Connection(255)-172.28.52.245;ClassName=com.sun.enterprise.appclient.jws.AppclientJWSSupportInfo;MethodName=startJWSServicesForApplication;_RequestID=21f07ad8-cf0f-4bb0-8dd1-aec55d8070cb;|Starting Java Web Start services for application NextGenServer-ear-1.2.2-SNAPSHOT|#]
</pre>
<pre>[#|2012-09-10T19:13:31.239+0200|INFO|sun-appserver2.1.1|javax.enterprise.system.tools.deployment|_ThreadID=65;_ThreadName=RMI TCP Connection(255)-172.28.52.245;|Registering ad hoc servlet: WebPathPath: context root = "/jp42", path = "'|#]

</pre>
<pre>[#|2012-09-10T19:13:31.265+0200|INFO|sun-appserver2.1.1|javax.enterprise.system.stream.out|_ThreadID=65;_ThreadName=RMI TCP Connection(255)-172.28.52.245;|jarsigner: unable to open jar file: /home/dm/development/server/glassfish/glassfish-2.1.1-business/nodeagents/jp42-nodeagent-1/jp42-instance-1/generated/xml/j2ee-apps/NextGenServer-ear-1.2.2-SNAPSHOT/NextGenServer-ear-1.2.2-SNAPSHOTClient.jar|#]

</pre>
<pre>[#|2012-09-10T19:13:31.266+0200|INFO|sun-appserver2.1.1|javax.enterprise.system.stream.out|_ThreadID=65;_ThreadName=RMI TCP Connection(255)-172.28.52.245;|jarsigner error: java.security.AccessControlException: System.exit|#]

</pre>
<pre>[#|2012-09-10T19:13:31.266+0200|SEVERE|sun-appserver2.1.1|javax.enterprise.system|_ThreadID=65;_ThreadName=RMI TCP Connection(255)-172.28.52.245;_RequestID=21f07ad8-cf0f-4bb0-8dd1-aec55d8070cb;|Error updating Java Web Start information for application NextGenServer-ear-1.2.2-SNAPSHOT

</pre>
<pre>java.lang.RuntimeException: java.lang.Exception: Error attempting to create signed jar /home/dm/development/server/glassfish/glassfish-2.1.1-business/nodeagents/jp42-nodeagent-1/jp42-instance-1/java-web-start/NextGenServer-ear-1.2.2-SNAPSHOT/NextGenServer-ear-1.2.2-SNAPSHOTClient.jar; see other log messages for more information

        at com.sun.enterprise.appclient.jws.SignedStaticContent.getRelativeURI(SignedStaticContent.java:123)
        at com.sun.enterprise.appclient.jws.StaticContent.toString(StaticContent.java:84)
        at java.lang.String.valueOf(String.java:2826)
        at java.lang.StringBuilder.append(StringBuilder.java:115)
        at java.util.AbstractMap.toString(AbstractMap.java:490)
        at com.sun.enterprise.appclient.jws.ContentOrigin.toLongString(ContentOrigin.java:155)
        at com.sun.enterprise.appclient.jws.AppclientJWSSupportInfo.startJWSServicesForApplication(AppclientJWSSupportInfo.java:487)
        at com.sun.enterprise.appclient.jws.AppclientJWSSupportManager.handleApplicationEvent(AppclientJWSSupportManager.java:156)
        at com.sun.enterprise.server.event.ApplicationLoaderEventNotifier.notifyListeners(ApplicationLoaderEventNotifier.java:154)
        at com.sun.enterprise.server.AbstractLoader.notifyAppEvent(AbstractLoader.java:894)
        at com.sun.enterprise.server.ApplicationLoader.doLoad(ApplicationLoader.java:192)
        at com.sun.enterprise.server.TomcatApplicationLoader.doLoad(TomcatApplicationLoader.java:126)
        at com.sun.enterprise.server.ExtendedApplicationLoader.doLoad(ExtendedApplicationLoader.java:134)
        at com.sun.enterprise.server.AbstractLoader.load(AbstractLoader.java:240)
        at com.sun.enterprise.server.ApplicationManager.applicationDeployed(ApplicationManager.java:336)
        at com.sun.enterprise.server.ApplicationManager.applicationDeployed(ApplicationManager.java:210)
        at com.sun.enterprise.server.ApplicationManager.applicationDeployed(ApplicationManager.java:648)
        at com.sun.enterprise.admin.event.AdminEventMulticaster.invokeApplicationDeployEventListener(AdminEventMulticaster.java:959)
        at com.sun.enterprise.admin.event.AdminEventMulticaster.handleApplicationDeployEvent(AdminEventMulticaster.java:943)
        at com.sun.enterprise.admin.event.AdminEventMulticaster.processEvent(AdminEventMulticaster.java:467)
        at com.sun.enterprise.admin.event.AdminEventMulticaster.multicastEvent(AdminEventMulticaster.java:182)
        at com.sun.enterprise.ee.admin.mbeans.ServerRuntimeMBean.forwardEvent(ServerRuntimeMBean.java:95)
        at sun.reflect.GeneratedMethodAccessor335.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.admin.MBeanHelper.invokeOperationInBean(MBeanHelper.java:393)
        at com.sun.enterprise.admin.MBeanHelper.invokeOperationInBean(MBeanHelper.java:376)
        at com.sun.enterprise.admin.runtime.BaseRuntimeMBean.invoke(BaseRuntimeMBean.java:471)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.enterprise.admin.util.proxy.ProxyClass.invoke(ProxyClass.java:90)
        at $Proxy1.invoke(Unknown Source)
        at com.sun.enterprise.admin.server.core.jmx.SunoneInterceptor.invoke(SunoneInterceptor.java:304)
        at com.sun.enterprise.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:170)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
        at sun.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.Exception: Error attempting to create signed jar /home/dm/development/server/glassfish/glassfish-2.1.1-business/nodeagents/jp42-nodeagent-1/jp42-instance-1/java-web-start/NextGenServer-ear-1.2.2-SNAPSHOT/NextGenServer-ear-1.2.2-SNAPSHOTClient.jar; see other log messages for more information
        at com.sun.enterprise.appclient.jws.ASJarSigner.sign(ASJarSigner.java:178)
        at com.sun.enterprise.appclient.jws.ASJarSigner.signJar(ASJarSigner.java:116)
        at com.sun.enterprise.appclient.jws.SignedStaticContent.signJar(SignedStaticContent.java:164)
        at com.sun.enterprise.appclient.jws.SignedStaticContent.ensureSignedFileUpToDate(SignedStaticContent.java:144)
        at com.sun.enterprise.appclient.jws.SignedStaticContent.getRelativeURI(SignedStaticContent.java:120)
        ... 54 more
Caused by: java.security.AccessControlException: System.exit
        at com.sun.enterprise.appclient.jws.ASJarSigner$NoExitSecurityManager.checkExit(ASJarSigner.java:573)
        at java.lang.Runtime.exit(Runtime.java:88)
        at java.lang.System.exit(System.java:904)
        at sun.security.tools.JarSigner.run(JarSigner.java:210)
        at sun.security.tools.JarSigner.main(JarSigner.java:74)
        at com.sun.enterprise.appclient.jws.ASJarSigner.sign(ASJarSigner.java:166)
        ... 58 more

AccessControlException is the consequence of the System.exit in the catch block of ASJarSigner.. the error happens earlier, exactly when the app Client jar is generated.

Does anyone can point me to the class where Client jar gets generated? I have to debug this and find the reason.. it's driving me nuts..

squibber

tjquinn
Offline
Joined: 2005-03-30
Points: 0

It's not clear exactly why this is not working, but I can explain what the system is trying to do.

Because Java Web Start enforces strict security (because the apps it launches are downloaded over the network) GlassFish automatically signs the JARs used for launching app clients using Java Web Start. In GlassFish 2.1.1 it does this by, basically, running the jarsigner utility. (It does not really create another OS process and run that command, but it uses the same code.)

The jarsigner utility is complaining that it cannot locate the (unsigned) generated app client JAR file to sign it for use in Java Web Start. (One of the early log messages you posted from the server.log file identifies the exact file.)

(Do not worry about the security errors about the System.exit calls. The jarsigner code invokes System.exit if it detects an error. We do not want the entire server to exit if the jarsigner fails to sign a JAR as it does in your example, so the server traps the attempt to invoke System.exit by using a security manager for just the jarsigner call that prohibits System.exit calls.)

Here is one possible cause. Back in the 2.x days, the app client JAR was generated asynchronously if possible, to allow the deployment itself to be reported as completed more quickly so the admin user could continue working sooner. It's possible that the async generation of the app client JAR has not completed by the time the system is trying to sign the JAR.

IIRC the app client JAR generation is synchronous if the deploy command specifies "--retrieve localdir" so you might try that - even though you do not really need the retrieved JAR itself - and see if that helps.

Let us know.

By the way, do you have plans to move to 3.x soon? Deployment and client generation have changed moving from 2.x to 3.x and this problem should not appear in 3.x.

- Tim

squibber
Offline
Joined: 2012-08-22
Points: 0

Hi Tim,

tjquinn wrote:

Here is one possible cause. Back in the 2.x days, the app client JAR was generated asynchronously if possible, to allow the deployment itself to be reported as completed more quickly so the admin user could continue working sooner. It's possible that the async generation of the app client JAR has not completed by the time the system is trying to sign the JAR.

IIRC the app client JAR generation is synchronous if the deploy command specifies "--retrieve localdir" so you might try that - even though you do not really need the retrieved JAR itself - and see if that helps.

Let us know.

Option --retrieve did the trick. You are the hero of the day of our department! :)

So hmm.. seems that this behaviour is related to some bug in the GlassFish 2.1.1 deployment process? Though we have commercial support from Oracle I can live with the --retrieve option. Or should I file this bug anyway?

tjquinn wrote:

By the way, do you have plans to move to 3.x soon? Deployment and client generation have changed moving from 2.x to 3.x and this problem should not appear in 3.x.

Yes we plan to move to 3.x in 2013, probably in March or April. JWS in GF 3 has been improved a lot and I like especially the jnlp template option. I miss that in GF 2..

We already have a test system with the current 3.1.2.2 release and encountered some class loading problems so far with our application. When we got commercial support (around May this year) we asked whether we should use GF 3 instead of GF 2. Oracle support said we should stick with GF 2 because we make use of the cluster functionality and GF 3.1 was reported not production ready with this feature.

squibber

tjquinn
Offline
Joined: 2005-03-30
Points: 0

Squibber,

I'm glad to hear that the --retrieve workaround works.

Feel free to file a bug if you want to. Honestly, I doubt that it will be fixed in the 2.x code base because the workaround is relatively easy, but having the bug might make it easier for other people to find the workaround if they run into the same problem you had.

- Tim