Skip to main content

[load tests] activity created timeout problem

7 replies [Last post]
scottjg
Offline
Joined: 2008-08-21
Points: 0

Hi,

I am during load tests. We are using Parlay Adaptor in our solution.
The main problem is between RA and SLEE. Our service has to responde with maximum delay of 500ms. Service execution takes maximum between 10ms - 200ms (two queries to external resources). That would be acceptable. But there are long timeouts when new activity context is created and when new event is fired during high volume of traffic.
e.g. timeouts:

org.mobicents.slee.resource.SleeEndpointImpl#activityCreated(Object) - 1042ms
org.mobicents.slee.resource.SleeEndpointImpl#fireEvent(javax.slee.resource.ActivityHandle, Object, int, javax.slee.Address) - 479ms

and there are not single exceptions.

Traffic parameters:
avarage caps: 40
maximum caps: 60

Event flow:
thread-1: corba request received (CallEventNotify) -> parlay handler
thread-2: parlay handler -> create activity context per call id -> fire event to slee (here are these timeouts)
thread-3: event router (EventRouterImpl#routeQueuedEvent) -> service

After 15 seconds there are sended another event RouteReqRes, that is mean that per one call are two events: new CallEventNotify and RouteReqRes from existing call, so there are 120 events/s from resource adaptor.

Question: do you now source of so long timeouts during org.mobicents.slee.resource.SleeEndpointImpl#activityCreated?

I can send more details if needed.

Best regards,
Gregory

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
scottjg
Offline
Joined: 2008-08-21
Points: 0

Sorry, I wrote previos post before check it.

Correct is

executorIndex = ++executorIndex%EXECUTOR_POOL_SIZE;
return this.execs[executorIndex];

pocketbikemike
Offline
Joined: 2010-02-27
Points: 0

Wow! That was all it took to fix? Trying to figure that out would have taken me days. I was overlooking the main elements that were wrong and trying to complicate the whole problem. Thanks for helping!!

---------------------

Pocket Bikes

scottjg
Offline
Joined: 2008-08-21
Points: 0

We change this lines like this:
private ExecutorService pickupExecutor() {
executorIndex++;
return this.execs[executorIndex % EXECUTOR_POOL_SIZE];
}

and it helps.

Gregory

scottjg
Offline
Joined: 2008-08-21
Points: 0

OK , It was my fault. I have been using JBoss Profiler and which has too many class to loop up.
But still I have problem.
Our test call has time duration equals 20 seconds (5 seconds before answer and 15 seconds before end). 20 seconds after test is started we have almost equals number of activities, e.g. say for 20 CAPS and 20 seconds per one call we have about 400 activites. This number is almost constant during test. I noticed that after that 20 seconds I have first losts (lost is call that takes 500ms before sending first event).
I added log line to [b]class org.mobicents.slee.runtime.EventRouterImpl in method private ExecutorService getExecutor(Object activity)[/b]. What I noticed is that when number of activites is stabilised this method assign many new activites to one executor. Probably because of that line:
[b]return this.execs[executors.size() % EXECUTOR_POOL_SIZE];[/b]
So last calls which are assigned to one executor fails.

Does anybody noticed similar problem ?
Is this shouldn't be change for something like that :
return this.execs[somethinRandom % EXECUTOR_POOL_SIZE];

Thanks for help in advance,

Best regards
Gregory

eduardomartins
Offline
Joined: 2005-10-10
Points: 0

Indeed, makes sense, if the number of activities becomes constant it will always use the same executor, but it still doesn't explain why you are getting those failures. Mobicents SLEE 1.x has several latency issues within its data source (xacache), due to excessive locking, but I guess your event handling methods are also not helping, probably take too much time processing the event.

Anyway, we will think about the distribution function, perhaps make other algorithms available through configuration.

scottjg
Offline
Joined: 2008-08-21
Points: 0

Hi,

Yes, unfortunately we make synchronous call which takes unexpectedly time between 100-150ms. When it is happened it affect few next calls in regular periods. It is our design fault. But changing services now could take to much time and we cannot afford it.
We solved it changing one method:

private ExecutorService pickupExecutor() {
executorIndex = executorIndex++%EXECUTOR_POOL_SIZE;
return this.execs[executorIndex % EXECUTOR_POOL_SIZE];
}

(we modified assiging this value to avoid negative values).

I posted it here because I didn't investigate transaction cases. I am not sure is this solution doesn't violent the rest of components in EventRouter.
But I must admit it helps us because for test cases based on call with constant duration and constant call per second, every next call has assigned next executor, they didn't queue in this situation many per one executor.

Thanks for reply,
Gregory

scottjg
Offline
Joined: 2008-08-21
Points: 0

By synchronous calls I mean method calls to external resource.