Skip to main content

Strange 503 (service unavailable) error of "call controller 2"

9 replies [Last post]
vola
Offline
Joined: 2008-05-20

Hello every experts,

I am doing some load testing of "call controller 2", and found a really strange problem.

I am using "mobicents-all-1_2_0_BETA1" and Sipp 3.1.

The problem is:

If I run "sipp -sf uac_pcap.xml -r 1 -rp 200" (Sipp will generate 1 call per 200 ms), after receiving about 2200 INVITE requests, mobicents server will start to return 503 (service unavailable) errors and not process most of the new coming INVITE requests.

That is, for a new starting mobicents server, it can correctly process about first 2200 calls without returning any errors. But after that, it rejects to provide services for most of new coming requests (it will correctly process some of them) and return 503 errors.

There are no exceptions showing up on the JBoss console and it can receive those INVITE requests with no problem. However, it just does not redirect them to the UAS.

I tried to stop the loading test at 2100, and then restart Sipp again. However, when Sipp reached about 100, the mobicents server will behave like I described above again.

The more surprising thing is, if I change the Sipp generator to "sipp -sf uac_pcap.xml -r 1 -rp 300" (Sipp will generate 1 call per 300 ms), the mobicents server will not start to return errors after the first 2200 calls, but after 4000 calls.

And then I tried to change the "call per ms" to 1/400ms, 1/500/ms, 1/1s, 1/3s. The mobicents will always go crazy at about 4000th calls.

Another interesting point is, to make the mobicents be able to serve request again, I can not just "undeploy-all" and then "deploy-all". I have to terminate JBoss, and restart the whole thing again!!!

I have been testing it for all day. So I am pretty sure it is not an incident case. Could anyone please explain to me if I did anything wrong, or this is a limitation of mobicents/call controller 2"?

Thank you very much~

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
vola
Offline
Joined: 2008-05-20

Thanks Jean and Martins~~~

Our team is building up an adaptive system to control the application. "Call controller 2" is our case study. Could you please give me more information about:

1. How to configure "gov.nist.javax.sip.MAX_SERVER_TRANSACTIONS" or other parameters? I search the whole "mobicents-all-1_2_0_BETA1" folder but can not find the place. The only possible place is in "sipStack.properties" under "..\examples\call-controller2\src\org\mobicents\test". I tried to change this file, but it has no effect for the 503 error problem.

2. Is that possible we can modify "Call controller 2" source code so that it can handle the load testing? It seems like "Call controller 2" can not cancel transactions or clean memory properly. Could we add some code to solve this problem?

Thank you~~~

eduardomartins
Offline
Joined: 2005-10-10

> Thanks Jean and Martins~~~
>
> Our team is building up an adaptive system to control
> the application. "Call controller 2" is our case
> study. Could you please give me more information
> about:
>
> 1. How to configure
> "gov.nist.javax.sip.MAX_SERVER_TRANSACTIONS" or other
> parameters? I search the whole
> "mobicents-all-1_2_0_BETA1" folder but can not find
> the place. The only possible place is in
> "sipStack.properties" under
> "..\examples\call-controller2\src\org\mobicents\test".
> I tried to change this file, but it has no effect for
> the 503 error problem.
>

the sip stack properties are configured on the sip ra @ servers/jain-slee/resources/sip/ra/src/main/resources/...

> 2. Is that possible we can modify "Call controller 2"
> source code so that it can handle the load testing?
> It seems like "Call controller 2" can not cancel
> transactions or clean memory properly. Could we add
> some code to solve this problem?
>
> Thank you~~~

Sure, contributions to improve examples are more than welcome :-)

eduardomartins
Offline
Joined: 2005-10-10

Anyway, Beta1 is a kind of old release, many issues fixed and performance improved since it was released, you should use the last version available.

In the next few days 1.2.0.CR1 should be out, which will include current svn HEAD code for jain-slee, media, xdm and presence servers; and 0.4.x version of sip-servlets server.

vola
Offline
Joined: 2008-05-20

That is cool.

We have found the properties file that contains sip stack parameters.

One more question. If we want to fix the problem of "call controller 2", how should we get starting? Could you please give us some advice about doing what can fix the problem?

eduardomartins
Offline
Joined: 2005-10-10

Well the examples were never thought to become high performance or leaks free applications, and call controller just examples call controlling with our media server, but first thing I think should be profiling it and see if the issues come from call controller or from the underlying sip services examples (proxy and registrar) used, and then start digging and fixing...

vola
Offline
Joined: 2008-05-20

Thanks alot~~~~That was really helpful :-)

vola
Offline
Joined: 2008-05-20

I have found the easiest way to solve this problem...............Using BETA3 instead of BEAT1....

:-)

deruelle_jean
Offline
Joined: 2003-06-24

I'm not sure this example was built to handle performance tests and it may have some problems regarding this, someone else may confirm.
503 responses are typically sent by the jsip stack when it is overloaded I think (cannot create a new STX to handle the request because they are all already used).
You may consider setting the gov.nist.javax.sip.MAX_SERVER_TRANSACTIONS property to a higher value than the default value also and maybe try to tune the jsip stack to help resolve the problem : see properties of the nist sip stack at http://snad.ncsl.nist.gov/proj/iptel/jain-sip-1.2/javadoc/gov/nist/javax...

Best regards
Jean

eduardomartins
Offline
Joined: 2005-10-10

Jean is right on both, cc2 example was never checked for performance or leaks and 503 usually means all stack threads get busy or stuck.

For load testing take a look at http://groups.google.com/group/mobicents-public/web/mobicents-load-testing