Skip to main content

A lock could not be obtained within the time requested

Please note these java.net forums are being decommissioned and use the new and improved forums at https://community.oracle.com/community/java.
10 replies [Last post]
jcite
Offline
Joined: 2011-07-03

Hello,
I have a Problem with glassfish 3.1.1 and the ejb-timer-service.

First when I deployed my application everything worked fine, but after some days, logging of the server immediatly stopped. I investigated the logs and saw, that there are error messages when trying to restart the application.


Internal Exception: java.sql.SQLTransactionRollbackException: A lock could not be obtained within the time requested
Error Code: 30000
Call: SELECT "TIMERID", "APPLICATIONID", "BLOB", "CONTAINERID", "CREATIONTIMERAW", "INITIALEXPIRATIONRAW", "INTERVALDURATION", "LASTEXPIRATIONRAW", "OWNERID", "PKHASHCODE", "SCHEDULE", "STATE" FROM "EJB__TIMER__TBL" WHERE (("OWNERID" = ?) AND ("STATE" = ?))
bind => [2 parameters bound]
Query: ReadAllQuery(name="findTimersByOwnerAndState" referenceClass=TimerState sql="SELECT "TIMERID", "APPLICATIONID", "BLOB", "CONTAINERID", "CREATIONTIMERAW", "INITIALEXPIRATIONRAW", "INTERVALDURATION", "LASTEXPIRATIONRAW", "OWNERID", "PKHASHCODE", "SCHEDULE", "STATE" FROM "EJB__TIMER__TBL" WHERE (("OWNERID" = ?) AND ("STATE" = ?))")

So I undeployed my application and redeployed it with the result

Exception while loading the app. EJB Timer Service is not available Please see server logs for more details.

My thaughts were, that it has something to do with the built in jdbc__TimerPool and the derby database, so I stopped the Glassfish-3.1.1 Instance.
because I found this article http://www.lyonlabs.org/howto/howdoi-glassfish.html „Problems with Persistent Timers“ and tried to access the interal derby database, to delete the persitent timers, because I use persistent timers in my application and sometimes they are not deleted and attempts to redeliever expired timers are made at server startup.


/opt/glassfish3/javadb/bin/ij
ij> connect 'jdbc:derby:/opt/glassfish3/glassfish/domains/domain1/lib/databases/ejbtimer';
ij> show tables;
TABLE_SCHEM |TABLE_NAME |REMARKS
------------------------------------------------------------------------
SYS |SYSALIASES |
SYS |SYSCHECKS |
SYS |SYSCOLPERMS |
SYS |SYSCOLUMNS |
SYS |SYSCONGLOMERATES |
SYS |SYSCONSTRAINTS |
SYS |SYSDEPENDS |
SYS |SYSFILES |
SYS |SYSFOREIGNKEYS |
SYS |SYSKEYS |
SYS |SYSPERMS |
SYS |SYSROLES |
SYS |SYSROUTINEPERMS |
SYS |SYSSCHEMAS |
SYS |SYSSEQUENCES |
SYS |SYSSTATEMENTS |
SYS |SYSSTATISTICS |
SYS |SYSTABLEPERMS |
SYS |SYSTABLES |
SYS |SYSTRIGGERS |
SYS |SYSVIEWS |
SYSIBM |SYSDUMMY1 |
APP |EJB__TIMER__TBL |

23 rows selected
ij> select * from EJB__TIMER__TBL;
TIMERID |APPLICATIONID |BLOB |CONTAINERID |CREATIONTIMERAW |INITIALEXPIRATIONRAW|INTERVALDURATION |LASTEXPIRATIONRAW |OWNERID |PKHASHCODE |SCHEDULE |STATE
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ERROR 40XL1: A lock could not be obtained within the time requested
ij>

So obviously the problem is, that there are locks on the javaDB that prevent accessing and using it for the timer-service, but despite serveral reboots the locks remain. EJB__TIMER__TBL obviously exist, there is no glassfish instance running, because if glassfish is running connection attempts to the javaDB are refused.

Just to be sure a call to
http://xxx.xxx.xxx.xxx:8080/ejb-timer-service-app/timer

delivers
HTTP Status 404 " so there is no timer service running.
Altough this is a non clustered Environment, I tried to use the jdbc/__default for the server config with no success.

So there is no ejb-timer-service running and can't be reactivated. The only thing I can do from this point is to reinstall whole glassfish-3.1.1 and everything will be fine again for the next few days. I had this issue couple of times and we are looking for a way out of this problem, because we can't cope with this issue in our live environment.

Any help would be apreciated, thanks
regards j.

AttachmentSize
log.txt13.82 KB

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
badbob.nsk
Offline
Joined: 2011-12-04

Have the same issue. Did you found the solution?

jcite
Offline
Joined: 2011-07-03

I only can recommed to every one how is facing this problem and does not depend necessary on persistent timers, to switch to non persistent one.

I did and since the problem never ocurred again:

Timer timer;
TimerService timerService;
TimerConfig timCfg = new TimerConfig();
timCfg.setPersistent(false);
this.timer = timerService.createSingleActionTimer(10000,timCfg);

you can check if you are still using persistent timers by calling
http://localhost:8080/ejb-timer-service-app/timer
in your browser while GF is running

regards, j.

mvatkina
Offline
Joined: 2005-04-04

Why would an application being restarted?

There can be an unfinished XA transaction that needs to be recovered to
remove the locks on the database, or there might be another thread
trying to access the same instance (and by default, it's an embedded
derby server, so no 2 VMs can access it at the same time).

-marina

forums@java.net wrote:
> Hello, I have a Problem with glassfish 3.1.1 and the
> ejb-timer-service. First
> when I deployed my application everything worked fine, but after some
> days,
> logging of the server immediatly stopped. I investigated the logs and
> saw,
> that there are error messages when trying to restart the application.
> Internal Exception:
> java.sql.SQLTransactionRollbackException:
> A lock could not be obtained within the time requested Error Code: 30000
> Call: SELECT "TIMERID", "APPLICATIONID", "BLOB", "CONTAINERID",
> "CREATIONTIMERAW", "INITIALEXPIRATIONRAW", "INTERVALDURATION",
> "LASTEXPIRATIONRAW", "OWNERID", "PKHASHCODE", "SCHEDULE", "STATE" FROM
> "EJB__TIMER__TBL" WHERE (("OWNERID" = ?) AND ("STATE" = ?)) bind => [2
> parameters bound] Query: ReadAllQuery(name="findTimersByOwnerAndState"
> referenceClass=TimerState sql="SELECT "TIMERID", "APPLICATIONID", "BLOB",
> "CONTAINERID", "CREATIONTIMERAW", "INITIALEXPIRATIONRAW",
> "INTERVALDURATION",
> "LASTEXPIRATIONRAW", "OWNERID", "PKHASHCODE", "SCHEDULE", "STATE" FROM
> "EJB__TIMER__TBL" WHERE (("OWNERID" = ?) AND ("STATE" = ?))")

> So I
> undeployed my application and redeployed it with the result
>
> Exception while loading the app. EJB Timer Service is not available
> Please
> see server logs for more details.
My thaughts were, that it has
> something to do with the built in jdbc__TimerPool and the derby
> database, so
> I stopped the Glassfish-3.1.1 Instance. because I found this article
> http://www.lyonlabs.org/howto/howdoi-glassfish.html „Problems with
> Persistent Timers“ and tried to access the interal derby database, to
> delete the persitent timers, because I use persistent timers in my
> application and sometimes they are not deleted and attempts to redeliever
> expired timers are made at server startup. [=Courier]
> /opt/glassfish3/javadb/bin/ij ij> connect
> 'jdbc:derby:/opt/glassfish3/glassfish/domains/domain1/lib/databases/ejbtimer';
>
> ij> show tables; TABLE_SCHEM |TABLE_NAME |REMARKS

jcite
Offline
Joined: 2011-07-03

Hi Marina,

thanx for your answer. Yes there is a lock on the database, but I can not figure out why. I dont know about the interal mechanism of locking on the dery DB, but because it remains and can not be finished, it might be a bug. I made sure, that every VM on the system was down when I tried, to query the database. Because logging stops at the time, Glassfish gets into troubles with the timer services I am not able to examine the log files. The soulution might be, to get rid of the derby database for timer services and utilize a mysql database via JDBC, in this case I hopefully will be able to take a look at the database.

j.

satoc
Offline
Joined: 2011-06-06

Hi,

Your post and work around not using persistent timer helped me a lot. I also have just experienced the same problem with v3.1.2 Open Source Edition.
Here is how to recover the corrupted EJB__TIMER__TBL.
I am posting this for everyone's interest who come accross this problem.

1) Delete EJB Timer application from the GlassFish(GF) server.

2) Create a new JavaDB database with the same name as GlassFish uses.
Put the following code anywhere in your GlassFish server(J2EE) application. Any server application running on GF should recognize the "jdbc:derby" scheme.

Connection conn;
try {
conn = DriverManager.getConnection(
"jdbc:derby:C:/temp/ejbtimer;create=true"
);
} catch(SQLException se) {
for (SQLException e = se; e != null; e = e.getNextException()) {
System.err.printf("%s: %s%n", e.getSQLState(), e.getMessage());
}
}

3) Stop GF server and delete the original ejb-timer JavaDB from the following location. Be sure to make backup before deletion!
[GlassFIsh Intall Dir]\glassfish\domains\domain1\lib\databases\ejbtimer

4) Then using 'ij' program located in the directory [GlassFIsh Intall Dir]\javadb\bin, create the EJB__TIMER__TBL in the newly created JavaDB that the default set up of GF uses for EJB Timer.

>ij
ij>connect 'jdbc:derby:c:/glassfish3/glassfish/domains/domain1/lib/databases/ejbtimer';
ij>CREATE TABLE "APP"."EJB__TIMER__TBL" ("TIMERID" VARCHAR(255) NOT NULL, "APPLICATIONID" BIGINT, "BLOB" BLOB(2147483647),

"CONTAINERID" BIGINT, "CREATIONTIMERAW" BIGINT, "INITIALEXPIRATIONRAW" BIGINT, "INTERVALDURATION" BIGINT,

"LASTEXPIRATIONRAW" BIGINT, "OWNERID" VARCHAR(255), "PKHASHCODE" INTEGER,"SCHEDULE" VARCHAR(255), "STATE" INTEGER);
ij>exit;

5) Now you can restart GF and should have no locking (or persistence of it) problem.

petino
Offline
Joined: 2006-10-18

You could as well skip step 2 by defining datasource with property create=true. However I would prefer if EJB timer implementation on Glassfish automatically tries to create the table on startup if it does not exist..

mvatkina
Offline
Joined: 2005-04-04

The table is created automatically since GF 3.0 when the Timer Service is started the 1st time on a specific instance. If the table is to be recreated after that, the marker file needs to be removed (see documentation). The previous poster tried to recreate the table with all the data.

-marina

petino
Offline
Joined: 2006-10-18

Great news Marina, thanks.. I am still on 2.1.1 :-)

dalaitama
Offline
Joined: 2013-01-17

May be a bit late, but i got the same problem this morning, it could work also for 2.1.1.

. To solve the problem you should delete from your domain folder the following file and folder:

File:
…..\glassfish-v3\glassfish\domains\domain1\generated\ ejb-timer-service-app
Folder:
….. \glassfish-v3\glassfish\domains\domain1\lib\databases\ ejbtimer\

Then it works again. Obviously you loose the persisted timers,...

Check if your EJB Timer Service is running then after the deployement of your application.
http://localhost:8080/ejb-timer-service-app/timer

mvatkina
Offline
Joined: 2005-04-04

Are you by any chance deleting all timers and creating them again in some kind of a loop operation on startup? We saw this to be a problem some time ago...

-marina