All calls failing!

Discussion:

All calls failing!

Sven Evensen

2012-07-03 13:36:03 UTC

We just had a serious fault at a customer where suddenly all calls were
failing. They have sipx 4.4 and a bit over 100 users. Normally they have
5-10 simultaneous calls, but they could see in the "SIP Trunk SBC
Statistics" that there were over 40 connected calls and rising. I had to
recover service quickly but saw the "top cpu" looked normal, disk space ok
and and "free" memory ok.

A quick look in wireshark and I saw many "DECLINE" coming back from SIP
trunk. Well I restarted the three main services "siptrunking, sipxproxy and
sipregistrar" and this recovered the fault. The SIP trunk is IP
authenticated so there was no re-register that could recover a faulty SIP
trunk. Therefore I believe the issue was in sipX.

I have a capture of one of the first calls that was failing and it does
look very strange indeed. I also have the sipxbridge.log for that
call, luckily it was in debug mode.

In the capture 192.168.3.44 is ip of SNOM phone, 193.246.242.135 is SIP
trunk (ACME packet to be precise).

Customer said this had happened one time before, at least same symptoms.

Sven
--
*Sven Evensen, Operations Consultant*

*OnRelay*

Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902 8123 â
mailto:***@onrelay.com <***@onrelay.com> â www.onrelay.com

This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is intended
solely for the recipient and use by any other party is not authorised. If
you are not the intended recipient, be aware that any disclosure, copying,
distribution or use of the contents of this information or any attachment,
is prohibited. If you have received this electronic transmission in error,
please notify us immediately by electronic mail (***@onrelay.com) and
delete this message, along with any attachments, from your computer.
Registered in England No 04006093 ÂŠ Registered Office 1st Floor, 236 Gray's
Inn Road, London WC1X 8HB

Matt White

2012-07-03 15:38:27 UTC

Permalink

do you have the proxy log for the same time period?

sipxbridge is throwing a timer already cancelled. which I havent seen
before. Normally this sounds like a typical SIP DoS but your log isn't
showing any sip vicious attempts that I can see, unless they occurred
earlier and java has already ran out of process and the attack has
ended.

Maybe a bit of the log prior to the onset of failed calls with give more
insight. Can you upload them (bridge and proxy) somewhere for us to
look at?

-M
We just had a serious fault at a customer where suddenly all calls were
failing. They have sipx 4.4 and a bit over 100 users. Normally they have
5-10 simultaneous calls, but they could see in the "SIP Trunk SBC
Statistics" that there were over 40 connected calls and rising. I had to
recover service quickly but saw the "top cpu" looked normal, disk space
ok and and "free" memory ok.

A quick look in wireshark and I saw many "DECLINE" coming back from SIP
trunk. Well I restarted the three main services "siptrunking, sipxproxy
and sipregistrar" and this recovered the fault. The SIP trunk is IP
authenticated so there was no re-register that could recover a faulty
SIP trunk. Therefore I believe the issue was in sipX.

I have a capture of one of the first calls that was failing and it does
look very strange indeed. I also have the sipxbridge.log for that call,
luckily it was in debug mode.

In the capture 192.168.3.44 is ip of SNOM phone, 193.246.242.135 is SIP
trunk (ACME packet to be precise).

Customer said this had happened one time before, at least same symptoms.

Sven
--
Sven Evensen, Operations Consultant
OnRelay
Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902
8123 â mailto:***@onrelay.com â www.onrelay.com

This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is
intended solely for the recipient and use by any other party is not
authorised. If you are not the intended recipient, be aware that any
disclosure, copying, distribution or use of the contents of this
information or any attachment, is prohibited. If you have received this
electronic transmission in error, please notify us immediately by
electronic mail (***@onrelay.com) and delete this message, along with
any attachments, from your computer. Registered in England No 04006093
| Registered Office 1st Floor, 236 Gray's Inn Road, London WC1X 8HB

Sven Evensen

2012-07-03 16:19:56 UTC

Permalink

Hi Matt,

Here is link to the full capture and logs. sipxproxy was not in
debug unfortunately.
https://docs.google.com/open?id=0B6OtrM5F9wLsZFFrd2thYXBEVm8

Sven

Post by Matt White
do you have the proxy log for the same time period?
sipxbridge is throwing a timer already cancelled. which I havent seen
before. Normally this sounds like a typical SIP DoS but your log isn't
showing any sip vicious attempts that I can see, unless they occurred
earlier and java has already ran out of process and the attack has ended.
Maybe a bit of the log prior to the onset of failed calls with give more
insight. Can you upload them (bridge and proxy) somewhere for us to look
at?
-M
We just had a serious fault at a customer where suddenly all calls were
failing. They have sipx 4.4 and a bit over 100 users. Normally they have
5-10 simultaneous calls, but they could see in the "SIP Trunk SBC
Statistics" that there were over 40 connected calls and rising. I had to
recover service quickly but saw the "top cpu" looked normal, disk space ok
and and "free" memory ok.
A quick look in wireshark and I saw many "DECLINE" coming back from SIP
trunk. Well I restarted the three main services "siptrunking, sipxproxy and
sipregistrar" and this recovered the fault. The SIP trunk is IP
authenticated so there was no re-register that could recover a faulty SIP
trunk. Therefore I believe the issue was in sipX.
I have a capture of one of the first calls that was failing and it does
look very strange indeed. I also have the sipxbridge.log for that
call, luckily it was in debug mode.
In the capture 192.168.3.44 is ip of SNOM phone, 193.246.242.135 is SIP
trunk (ACME packet to be precise).
Customer said this had happened one time before, at least same symptoms.
Sven
--
*Sven Evensen, Operations Consultant*
*OnRelay*
Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902 8123â
This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is intended
solely for the recipient and use by any other party is not authorised. If
you are not the intended recipient, be aware that any disclosure, copying,
distribution or use of the contents of this information or any attachment,
is prohibited. If you have received this electronic transmission in error,
delete this message, along with any attachments, from your computer.
Registered in England No 04006093 ÂŠ Registered Office 1st Floor, 236 Gray's
Inn Road, London WC1X 8HB
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
*Sven Evensen, Operations Consultant*

*OnRelay*

Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902 8123 â
mailto:***@onrelay.com <***@onrelay.com> â www.onrelay.com

This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is intended
solely for the recipient and use by any other party is not authorised. If
you are not the intended recipient, be aware that any disclosure, copying,
distribution or use of the contents of this information or any attachment,
is prohibited. If you have received this electronic transmission in error,
please notify us immediately by electronic mail (***@onrelay.com) and
delete this message, along with any attachments, from your computer.
Registered in England No 04006093 ÂŠ Registered Office 1st Floor, 236 Gray's
Inn Road, London WC1X 8HB

Matt White

2012-07-03 17:38:17 UTC

Permalink

Have you verified DNS is resolving properly? Proxy is getting an error resolving the following

_sip._udp.cogestim.swisscom.vpbx
_sip._tcp.cogestim.swisscom.vpbx

-M
Hi Matt,

Here is link to the full capture and logs. sipxproxy was not in debug unfortunately.
https://docs.google.com/open?id=0B6OtrM5F9wLsZFFrd2thYXBEVm8

Sven

Sven Evensen

2012-07-03 17:43:21 UTC

Permalink

It has always been like that. sipx, SIP trunk and all phones are on their
internal network. They explained me once why it was like that, but I dont
quite remember. It has been running like this for months and months, so I
do beleive there must be some other cause.

Do you still think it could be an issue?

Sven

Post by Matt White
Have you verified DNS is resolving properly? Proxy is getting an error
resolving the following
_sip._udp.cogestim.swisscom.vpbx
_sip._tcp.cogestim.swisscom.vpbx
-M
Hi Matt,
Here is link to the full capture and logs. sipxproxy was not in debug unfortunately.
https://docs.google.com/open?id=0B6OtrM5F9wLsZFFrd2thYXBEVm8
Sven
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Matt White

2012-07-03 18:36:57 UTC

Permalink

Normally in the proxy logs we see the following error:

_sip._tls.cogestim.swisscom.vpbx
cogestim.swisscom.vpbx

Those are normal as the TLS and base domain do not typically have SRV
records added.

But unless I'm just being braindead, I've never seen it fail with the
main _sip._udp and _sip._tcp

Maybe I should go check some our logs to make sure i'm not being stupid.

-m
It has always been like that. sipx, SIP trunk and all phones are on
their internal network. They explained me once why it was like that, but
I dont quite remember. It has been running like this for months and
months, so I do beleive there must be some other cause.

Do you still think it could be an issue?

Sven

On Tue, Jul 3, 2012 at 6:38 PM, Matt White <***@thesummit-grp.com>
wrote:
Have you verified DNS is resolving properly? Proxy is getting an error
resolving the following

_sip._udp.cogestim.swisscom.vpbx
_sip._tcp.cogestim.swisscom.vpbx

-M
Hi Matt,

Here is link to the full capture and logs. sipxproxy was not in debug
unfortunately.
https://docs.google.com/open?id=0B6OtrM5F9wLsZFFrd2thYXBEVm8

Sven

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
Sven Evensen, Operations Consultant
OnRelay
Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902
8123 â mailto:***@onrelay.com â www.onrelay.com

This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is
intended solely for the recipient and use by any other party is not
authorised. If you are not the intended recipient, be aware that any
disclosure, copying, distribution or use of the contents of this
information or any attachment, is prohibited. If you have received this
electronic transmission in error, please notify us immediately by
electronic mail (***@onrelay.com) and delete this message, along with
any attachments, from your computer. Registered in England No 04006093
| Registered Office 1st Floor, 236 Gray's Inn Road, London WC1X 8HB

Sven Evensen

2012-07-04 16:05:19 UTC

Permalink

Anyone else who can help. Matt thinks it might be a DNS issue, which does
look strange, but it has always been like that at this customer. Why would
suddenly all calls start failing with lots of timer errors in sipXbridge?

Sven

Post by Matt White
_sip._tls.cogestim.swisscom.vpbx
cogestim.swisscom.vpbx
Those are normal as the TLS and base domain do not typically have SRV
records added.
But unless I'm just being braindead, I've never seen it fail with the main
_sip._udp and _sip._tcp
Maybe I should go check some our logs to make sure i'm not being stupid.
-m
It has always been like that. sipx, SIP trunk and all phones are on their
internal network. They explained me once why it was like that, but I dont
quite remember. It has been running like this for months and months, so I
do beleive there must be some other cause.
Do you still think it could be an issue?
Sven

--
*Sven Evensen, Operations Consultant*
*OnRelay*
Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902 8123â
This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is intended
solely for the recipient and use by any other party is not authorised. If
you are not the intended recipient, be aware that any disclosure, copying,
distribution or use of the contents of this information or any attachment,
is prohibited. If you have received this electronic transmission in error,
delete this message, along with any attachments, from your computer.
Registered in England No 04006093 ÂŠ Registered Office 1st Floor, 236 Gray's
Inn Road, London WC1X 8HB
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Tony Graziano

2012-07-04 16:48:53 UTC

Permalink

Well, I have to ask an obscure question... are your virtual kernels up to
date and not having any leap second issues.

It all boils down to timing (I think). I have a suspicion the DNS is timing
out/can't find the information its looking for (Matt is keen for pointing
DNS out), but it may be "artificially failing" due to the kernel being hung
with how it handles (or didn't handle) leap seconds. This is more prevalent
in virtual environments too I think.

I'm not sure what the official fix is, because I areboot seems to fix this.
If so, it's not a sipx issue and it has plagued a lot of folks lately with
various service(s) (Amazon too).

Post by Sven Evensen
Anyone else who can help. Matt thinks it might be a DNS issue, which does
look strange, but it has always been like that at this customer. Why would
suddenly all calls start failing with lots of timer errors in sipXbridge?
Sven

Post by Matt White
_sip._tls.cogestim.swisscom.vpbx
cogestim.swisscom.vpbx
Those are normal as the TLS and base domain do not typically have SRV
records added.
But unless I'm just being braindead, I've never seen it fail with the
main _sip._udp and _sip._tcp
Maybe I should go check some our logs to make sure i'm not being stupid.
-m
It has always been like that. sipx, SIP trunk and all phones are on their
internal network. They explained me once why it was like that, but I dont
quite remember. It has been running like this for months and months, so I
do beleive there must be some other cause.
Do you still think it could be an issue?
Sven

--
*Sven Evensen, Operations Consultant*
*OnRelay*
Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902 8123
www.onrelay.com
This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is intended
solely for the recipient and use by any other party is not authorised. If
you are not the intended recipient, be aware that any disclosure, copying,
distribution or use of the contents of this information or any attachment,
is prohibited. If you have received this electronic transmission in error,
delete this message, along with any attachments, from your computer.
Registered in England No 04006093 ÂŠ Registered Office 1st Floor, 236 Gray's
Inn Road, London WC1X 8HB
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013! <http://sipxcolab2013.eventbrite.com/?discount=tony2013%22>

--
LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net

Sven Evensen

2012-07-04 16:57:53 UTC

Permalink

I have read what you wrote a few days ago about leap second. Amazing how
that can affect systems.But didnt the leap happen july 1st, while our
customer had issues yesterday, july 3rd. If this is the case, I am just
worried about using this as explanation to customer, might seem a bit far
fetched for them :)

I am a bit worried about the
SIP/sipfrag Status: 500 Server Internal Error, with Sipfrag(Exception Info
Timer already cancelled. at Timer.java:354)
which happens half a second after call starts. Could this point to a timing
or leap second error or is this more of a logical error?

Sven

On Wed, Jul 4, 2012 at 5:48 PM, Tony Graziano

Post by Tony Graziano
Well, I have to ask an obscure question... are your virtual kernels up to
date and not having any leap second issues.
It all boils down to timing (I think). I have a suspicion the DNS is
timing out/can't find the information its looking for (Matt is keen for
pointing DNS out), but it may be "artificially failing" due to the kernel
being hung with how it handles (or didn't handle) leap seconds. This is
more prevalent in virtual environments too I think.
I'm not sure what the official fix is, because I areboot seems to fix
this. If so, it's not a sipx issue and it has plagued a lot of folks lately
with various service(s) (Amazon too).

Post by Matt White
_sip._tls.cogestim.swisscom.vpbx
cogestim.swisscom.vpbx
Those are normal as the TLS and base domain do not typically have SRV
records added.
But unless I'm just being braindead, I've never seen it fail with the
main _sip._udp and _sip._tcp
Maybe I should go check some our logs to make sure i'm not being stupid.
-m
It has always been like that. sipx, SIP trunk and all phones are on
their internal network. They explained me once why it was like that, but I
dont quite remember. It has been running like this for months and months,
so I do beleive there must be some other cause.
Do you still think it could be an issue?
Sven

--
*Sven Evensen, Operations Consultant*
*OnRelay*
Elizabeth House â 39 York Road, London SE1 7NQ, UK â +44 (0) 207 902
www.onrelay.com
This electronic message transmission contains information from OnRelay,
Ltd., that may be confidential or privileged. The information is intended
solely for the recipient and use by any other party is not authorised. If
you are not the intended recipient, be aware that any disclosure, copying,
distribution or use of the contents of this information or any attachment,
is prohibited. If you have received this electronic transmission in error,
delete this message, along with any attachments, from your computer.
Registered in England No 04006093 ÂŠ Registered Office 1st Floor, 236 Gray's
Inn Road, London WC1X 8HB
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013! <http://sipxcolab2013.eventbrite.com/?discount=tony2013%22>
Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.**net<http://myhelp.myitdepartment.net>
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Tony Graziano

2012-07-04 18:00:06 UTC

Permalink

It so happens the kernel issues affecting more recent of linux are JAVA
related. So yeah it sort of points there to me.

https://bugzilla.redhat.com/show_bug.cgi?id=479765

Scroll down and look at the posts dated just after July 1, 2012.

Post by Sven Evensen
I have read what you wrote a few days ago about leap second. Amazing how
that can affect systems.But didnt the leap happen july 1st, while our
customer had issues yesterday, july 3rd. If this is the case, I am just
worried about using this as explanation to customer, might seem a bit far
fetched for them :)
I am a bit worried about the
SIP/sipfrag Status: 500 Server Internal Error, with Sipfrag(Exception
Info Timer already cancelled. at Timer.java:354)
which happens half a second after call starts. Could this point to a
timing or leap second error or is this more of a logical error?
Sven
On Wed, Jul 4, 2012 at 5:48 PM, Tony Graziano <

Post by Sven Evensen
Anyone else who can help. Matt thinks it might be a DNS issue, which
does look strange, but it has always been like that at this customer. Why
would suddenly all calls start failing with lots of timer errors in
sipXbridge?
Sven

Post by Matt White
_sip._tls.cogestim.swisscom.vpbx
cogestim.swisscom.vpbx
Those are normal as the TLS and base domain do not typically have SRV
records added.
But unless I'm just being braindead, I've never seen it fail with the
main _sip._udp and _sip._tcp
Maybe I should go check some our logs to make sure i'm not being stupid.
-m
It has always been like that. sipx, SIP trunk and all phones are on
their internal network. They explained me once why it was like that, but I
dont quite remember. It has been running like this for months and months,
so I do beleive there must be some other cause.
Do you still think it could be an issue?
Sven

Post by Matt White
Have you verified DNS is resolving properly? Proxy is getting an
error resolving the following
_sip._udp.cogestim.swisscom.vpbx
_sip._tcp.cogestim.swisscom.vpbx
-M
Hi Matt,
Here is link to the full capture and logs. sipxproxy was not in
debug unfortunately.
https://docs.google.com/open?id=0B6OtrM5F9wLsZFFrd2thYXBEVm8
Sven
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/