new patch for XX-10177

Discussion:

George Niculae

2012-06-11 10:21:48 UTC

Hey Andy,

I somehow uploaded a patch that is not complete to XX-10177, could you
please get the latest one attached to JIRA and build
(http://track.sipfoundry.org/secure/attachment/27824/0001-XX-10177-Server-stops-responding-to-registration-req.patch)?
(thanks Levend for pointing this out)

Thanks and sorry
George

andrewpitman

2012-06-11 14:48:49 UTC

Permalink

if (res == crlfCount)

I thought it looked as if there was some sort of conditional
that was missing. ;)

Fortunately, I've been so swamped with other things I
haven't had the chance to really try it out. I will apply
today and start testing.

Thanks!
--

andrewpitman

2012-06-12 22:34:42 UTC

Permalink

We've been running the patched version on our test servers
since yesterday, and so far so good. Once everything checks
out, we can apply this to customers who have experienced the
registration drops and make sure this resolves their issue.
Not sure if there's a way to fully test the fix solely in
our lab, since we haven't seen it there to begin with. Any
ideas on how we might try to trigger the same conditions in
our lab?
--

Joegen Baclor

2012-06-13 01:21:37 UTC

Permalink

Change PeerAddr => '127.0.0.1:5060' to the address of sipx and cron
this perl script every 5 seconds.

Post by andrewpitman
We've been running the patched version on our test servers
since yesterday, and so far so good. Once everything checks
out, we can apply this to customers who have experienced the
registration drops and make sure this resolves their issue.
Not sure if there's a way to fully test the fix solely in
our lab, since we haven't seen it there to begin with. Any
ideas on how we might try to trigger the same conditions in
our lab?

andrewpitman

2012-06-14 15:43:11 UTC

Permalink

Joegen, I tried downloading that Perl script and all I get
is an empty file.
--

George Niculae

2012-06-14 15:38:56 UTC

Permalink

Post by andrewpitman
Joegen, I tried downloading that Perl script and all I get
is an empty file.
--

Hm, I see the content, check this:

#!/usr/bin/perl
#udpclient.pl

use IO::Socket::INET;

# flush after every write
$| = 1;

my ($socket,$data);

# We call IO::Socket::INET->new() to create the UDP Socket
# and bind with the PeerAddr.
$socket = new IO::Socket::INET (
PeerAddr => '127.0.0.1:5060',
Proto => 'udp'
) or die "ERROR in Socket Creation : $!\n”;
#send operation
$data = “\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n”;
$socket->send($data);

sleep(2);
$socket->close();

a***@comcast.net

2012-06-14 15:52:37 UTC

Permalink

Thanks, George!

----- Original Message -----
From: "George Niculae" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Thursday, June 14, 2012 11:38:56 AM
Subject: Re: [sipx-users] new patch for XX-10177

Post by andrewpitman
Joegen, I tried downloading that Perl script and all I get
is an empty file.
--

Hm, I see the content, check this:

#!/usr/bin/perl
#udpclient.pl

use IO::Socket::INET;

# flush after every write
$| = 1;

my ($socket,$data);

# We call IO::Socket::INET->new() to create the UDP Socket
# and bind with the PeerAddr.
$socket = new IO::Socket::INET (
PeerAddr => '127.0.0.1:5060',
Proto => 'udp'
) or die "ERROR in Socket Creation : $!\nâ;
#send operation
$data = â\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nâ;
$socket->send($data);

sleep(2);
$socket->close();
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

andrewpitman

2012-06-18 15:20:30 UTC

Permalink

George, Joegen,

I've been running this Perl script against an unpatched test
server for over 3 days now, and I still haven't been able to
tickle the bug. Any ideas?

Andy
--

Joegen Baclor

2012-06-19 06:58:20 UTC

Permalink

Hi Andrew,

Did you check the logs if we are able to trigger the message corruption
in your unpatched server?

Post by andrewpitman
George, Joegen,
I've been running this Perl script against an unpatched test
server for over 3 days now, and I still haven't been able to
tickle the bug. Any ideas?
Andy

andrewpitman

2012-06-19 21:59:55 UTC

Permalink

Joegen, like this?

"2012-06-19T21:49:58.561697Z"
:14693872:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=R5DYzi', uriForm = name-addr"
"2012-06-19T21:49:58.561790Z"
:14693873:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.561817Z"
:14693874:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.561826Z"
:14693875:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"
"2012-06-19T21:49:58.592986Z"
:14693876:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=ThsH31', uriForm = name-addr"
"2012-06-19T21:49:58.593125Z"
:14693877:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.593149Z"
:14693878:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.593159Z"
:14693879:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"
"2012-06-19T21:49:58.626257Z"
:14693880:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=Ylk0nY', uriForm = name-addr"
"2012-06-19T21:49:58.626354Z"
:14693881:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.626381Z"
:14693882:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.626412Z"
:14693883:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"
"2012-06-19T21:49:58.656583Z"
:14693884:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=m4gCQM', uriForm = name-addr"
"2012-06-19T21:49:58.656678Z"
:14693885:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.656704Z"
:14693886:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.656713Z"
:14693887:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"
"2012-06-19T21:49:58.689398Z"
:14693888:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=Lf6i9l', uriForm = name-addr"
"2012-06-19T21:49:58.689532Z"
:14693889:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.689570Z"
:14693890:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.689579Z"
:14693891:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"
"2012-06-19T21:49:58.722783Z"
:14693892:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=J9JwlH', uriForm = name-addr"
"2012-06-19T21:49:58.722875Z"
:14693893:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.722902Z"
:14693894:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.722911Z"
:14693895:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"
"2012-06-19T21:49:58.753908Z"
:14693896:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "Url::parseString no valid host found at
char 0 in ';tag=1pKKTM', uriForm = name-addr"
"2012-06-19T21:49:58.753997Z"
:14693897:SIP:WARNING:pbx1.ncpbxtechs.com:SipClientUdp-12:B7
5B9B90:SipXProxy: "SipMessage::getResponseSendAddress No
VIA address, using From address"
"2012-06-19T21:49:58.754024Z"
:14693898:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipMessage::convertProtocolStringToEnum
unrecognized protocol:"
"2012-06-19T21:49:58.754046Z"
:14693899:SIP:ERR:pbx1.ncpbxtechs.com:SipClientUdp-12:B75B9B
90:SipXProxy: "SipTransaction::doFirstSend protocol not
explicitly set - using UDP"

Joegen Baclor

2012-06-20 00:57:49 UTC

Permalink

Yes that is it. From the logs that was previously posted, this was the
most evident thing that was going crazy before the server was hung. So
two things,

1. We are totally mistaken that this was the cause.
2. 1 packet every 5 seconds is not enough to cause havoc in the system.

I am hoping it is the latter. If you do not mind, try to modify the
PERL script to make it send several packets in a tight loop. Lets say
10 every second. Let that run for an hour on a busy afternoon and see
if you can get sipx bend down to its knees.

Post by andrewpitman
Joegen, like this?

Andrew Pitman

2012-06-20 01:17:17 UTC

Permalink

Hi Joegen,

Actually, after a couple of days running it with the delay, I took the 2 second sleep out of the Perl script and ran it continuously from a while loop in the shell. Still no dice.

I could make it tighter still by looping in the Perl script with no sleeps. ;)

Andy
--
Sent from my iPhone appendage

Yes that is it. From the logs that was previously posted, this was the most evident thing that was going crazy before the server was hung. So two things,
1. We are totally mistaken that this was the cause.
2. 1 packet every 5 seconds is not enough to cause havoc in the system.
I am hoping it is the latter. If you do not mind, try to modify the PERL script to make it send several packets in a tight loop. Lets say 10 every second. Let that run for an hour on a busy afternoon and see if you can get sipx bend down to its knees.

Post by andrewpitman
Joegen, like this?

Joegen Baclor

2012-06-20 01:29:50 UTC

Permalink

Stressing sipx to its limits is a good litmus test so if you could, do
it. Take a statistical snapshot of CPU and mem and see if this causes
a gradual rise. For whatever it is worth, the patch may save us a few
mb worth of logs in production or it could be a real issue altogether.
If you aren't able to produce anything from your attempts that can be
treated as evidence of a bug, let us revisit the old logs and look for
other probable causes.

Post by Andrew Pitman
Hi Joegen,
Actually, after a couple of days running it with the delay, I took the 2 second sleep out of the Perl script and ran it continuously from a while loop in the shell. Still no dice.
I could make it tighter still by looping in the Perl script with no sleeps. ;)
Andy
--
Sent from my iPhone appendage

Yes that is it. From the logs that was previously posted, this was the most evident thing that was going crazy before the server was hung. So two things,
1. We are totally mistaken that this was the cause.
2. 1 packet every 5 seconds is not enough to cause havoc in the system.
I am hoping it is the latter. If you do not mind, try to modify the PERL script to make it send several packets in a tight loop. Lets say 10 every second. Let that run for an hour on a busy afternoon and see if you can get sipx bend down to its knees.

Post by andrewpitman
Joegen, like this?

Andrew Pitman

2012-06-20 02:06:07 UTC

Permalink

Time to break out the call generator again...

--
Sent from my iPhone appendage

Stressing sipx to its limits is a good litmus test so if you could, do it. Take a statistical snapshot of CPU and mem and see if this causes a gradual rise. For whatever it is worth, the patch may save us a few mb worth of logs in production or it could be a real issue altogether. If you aren't able to produce anything from your attempts that can be treated as evidence of a bug, let us revisit the old logs and look for other probable causes.

Yes that is it. From the logs that was previously posted, this was the most evident thing that was going crazy before the server was hung. So two things,
1. We are totally mistaken that this was the cause.
2. 1 packet every 5 seconds is not enough to cause havoc in the system.
I am hoping it is the latter. If you do not mind, try to modify the PERL script to make it send several packets in a tight loop. Lets say 10 every second. Let that run for an hour on a busy afternoon and see if you can get sipx bend down to its knees.

Post by andrewpitman
Joegen, like this?

andrewpitman

2012-06-21 14:34:21 UTC

Permalink

Okay, I ran two versions of the Perl script against my test
server for a few hours last night, both of which loop and
send the crlf keepalive messages as fast as they could be
sent. One was sending UDP and the other TCP. I didn't
manage to get proxy to stop responding to registration
requests, but the CPU was pegged at about 125% on this dual
proc VM. I actually managed to fill up my /var filesystem
with all the junk going to sipXproxy.log, but didn't manage
to bring forth the issue...
--

Joegen Baclor

2012-06-21 16:50:06 UTC

Permalink

Post by andrewpitman
Okay, I ran two versions of the Perl script against my test
server for a few hours last night, both of which loop and
send the crlf keepalive messages as fast as they could be
sent. One was sending UDP and the other TCP. I didn't
manage to get proxy to stop responding to registration
requests, but the CPU was pegged at about 125% on this dual
proc VM. I actually managed to fill up my /var filesystem
with all the junk going to sipXproxy.log, but didn't manage
to bring forth the issue...

Then it is not the issue. However, that perl script just became a DOS
attack tool against sipx so we need to accept the patch just to prevent
what you have done from happening. Even if the patch did not really
solve your issue, would you mind applying it and see if you can fill up
/var/ folder with the patch active?

Regarding the freezing issue, I'm gonna have to look again. If you have
a hunch where I should be looking at, let me know.

Andrew Pitman

2012-06-21 17:32:27 UTC

Permalink

Joegen, I don't mind at all. I've got a couple of servers with the patch already applied. I'll point my new "DOS generator" at those and watch for a few days. ;)

--
Sent from my iPhone appendage

Then it is not the issue. However, that perl script just became a DOS attack tool against sipx so we need to accept the patch just to prevent what you have done from happening. Even if the patch did not really solve your issue, would you mind applying it and see if you can fill up /var/ folder with the patch active?
Regarding the freezing issue, I'm gonna have to look again. If you have a hunch where I should be looking at, let me know.

andrewpitman

2012-06-21 18:09:29 UTC

Permalink

Joegen,

Actually, the patch seems to be very specific in what it
matches. For example, if I reformat the message I'm sending
so that it consists of line feeds, then carriage returns
instead of CRLFs, I can still generate tons of logging. As
a fix for a potential DOS attack, I don't think it's
complete.
--

andrewpitman

2012-08-02 16:19:01 UTC

Permalink

Hi Joegen!

I dug around a bit in the code, and I might have a starting
point for where to look for this...

When this bug has manifested itself, we've been able to
recover by restarting just sipXproxy, and not both proxy and
registrar, so the issue doesn't seem to be with registrar.
When the server stops responding to registration requests,
sipXproxy loses its connection to sipregistrar on port
5070/tcp. sipregistrar remains listening on port 5070 and
happily accepts the connection from sipXproxy when that's
restarted.

Also, based on Mike's comments when he visited the other
day, it does not seem like this issue has shown up for
installations which do not have many remote workers. In our
configurations here, with some exceptions all of our phones
are "remote workers," and in our setup we have to deal with
both near and far end NAT traversal.

Based on all that, and assuming the problem was probably
with sipXproxy (or sipXtackLib) and probably had something
to do with code dealing with NAT traversal, I came across
the code in sipXproxy that deals with processing of
forwardingrules.xml. If I'm reading the code (and comments)
correctly, it looks like if a request does not have route
state information (or if it does have NON-mutable route
state AND the URI is in the local domain AND globally
routable), forwarding rules is followed. Looking at the
forwarding rules xml file, it looks like the catchall
default is to send all other requests to the registry
service.

So, here's a thought... Given that forwarding to
sipregistrar is the default, what kind of malformed crap
could end up getting processed by that part of the code in
sipXproxy? It seems to me that it would be more likely to
bail there based on that.

I'll continue my digging there, but I wanted to let you
know. Would you mind having a look and let me know what you
think? I'll forward along some logs from a recent hang, as
well to make sure we're still on the right track.

Thanks!

Joegen Baclor wrote on Thu, 21 June 2012 12:50

Post by Joegen Baclor
Then it is not the issue. However, that perl script just
became a DOS
attack tool against sipx so we need to accept the patch
just to prevent
what you have done from happening. Even if the patch
did not really
solve your issue, would you mind applying it and see if
you can fill up
/var/ folder with the patch active?
Regarding the freezing issue, I'm gonna have to look
again. If you have
a hunch where I should be looking at, let me know.

Tony Graziano

2012-08-02 16:28:25 UTC

Permalink

The malformed crap could easily come from a misconfigured or badly designed
UA by the way. Also realize I have never seen it even with remote user
traversal WHEN I lick down pps to port 5060 in the firewall to a sane
functional number. One assumes you inspected the logs to verify there was
no outside attempt to spam calls via the proxy (I.e. INVITE)?

Post by andrewpitman
Hi Joegen!
I dug around a bit in the code, and I might have a starting
point for where to look for this...
When this bug has manifested itself, we've been able to
recover by restarting just sipXproxy, and not both proxy and
registrar, so the issue doesn't seem to be with registrar.
When the server stops responding to registration requests,
sipXproxy loses its connection to sipregistrar on port
5070/tcp. sipregistrar remains listening on port 5070 and
happily accepts the connection from sipXproxy when that's
restarted.
Also, based on Mike's comments when he visited the other
day, it does not seem like this issue has shown up for
installations which do not have many remote workers. In our
configurations here, with some exceptions all of our phones
are "remote workers," and in our setup we have to deal with
both near and far end NAT traversal.
Based on all that, and assuming the problem was probably
with sipXproxy (or sipXtackLib) and probably had something
to do with code dealing with NAT traversal, I came across
the code in sipXproxy that deals with processing of
forwardingrules.xml. If I'm reading the code (and comments)
correctly, it looks like if a request does not have route
state information (or if it does have NON-mutable route
state AND the URI is in the local domain AND globally
routable), forwarding rules is followed. Looking at the
forwarding rules xml file, it looks like the catchall
default is to send all other requests to the registry
service.
So, here's a thought... Given that forwarding to
sipregistrar is the default, what kind of malformed crap
could end up getting processed by that part of the code in
sipXproxy? It seems to me that it would be more likely to
bail there based on that.
I'll continue my digging there, but I wanted to let you
know. Would you mind having a look and let me know what you
think? I'll forward along some logs from a recent hang, as
well to make sure we're still on the right track.
Thanks!
Joegen Baclor wrote on Thu, 21 June 2012 12:50

--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net

Michael Picher

2012-08-02 16:42:12 UTC

Permalink

do you lick down a lot of ports? :-P

i think these are all Polycom phones with 3.2.6 firmware.

the only other thing a bit odd is they are coming through a Cisco ASA which
is known to work but could be a question mark. I think they were going to
try to route around this and then in through a pfSense box to see if the
same thing happens.

mike

Post by Tony Graziano
The malformed crap could easily come from a misconfigured or badly
designed UA by the way. Also realize I have never seen it even with remote
user traversal WHEN I lick down pps to port 5060 in the firewall to a sane
functional number. One assumes you inspected the logs to verify there was
no outside attempt to spam calls via the proxy (I.e. INVITE)?

--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.**net<http://myhelp.myitdepartment.net>
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
Michael Picher, Director of Technical Services
eZuce, Inc.

300 Brickstone Square****

Suite 201****

Andover, MA. 01810
O.978-296-1005 X2015
M.207-956-0262
@mpicher <http://twitter.com/mpicher>
linkedin <http://www.linkedin.com/profile/view?id=35504760&trk=tab_pro>
www.ezuce.com

------------------------------------------------------------------------------------------------------------
There are 10 kinds of people in the world, those who understand binary and
those who don't.

a***@comcast.net

2012-08-02 17:02:14 UTC

Permalink

Yeah, we actually "licked" it down to an allow list for only our customers on the ASA. ;) I believe before it actually gets to that point we have actually already filtered with country block.

The phones are indeed all Polycom 3.2.6/4.3.1.

The big variable here is that the firewalls on the remote end vary.

----- Original Message -----
From: "Michael Picher" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Thursday, August 2, 2012 12:42:12 PM
Subject: Re: [sipx-users] new patch for XX-10177

do you lick down a lot of ports? :-P

i think these are all Polycom phones with 3.2.6 firmware.

the only other thing a bit odd is they are coming through a Cisco ASA which is known to work but could be a question mark. I think they were going to try to route around this and then in through a pfSense box to see if the same thing happens.

mike

On Thu, Aug 2, 2012 at 12:28 PM, Tony Graziano < ***@myitdepartment.net > wrote:

The malformed crap could easily come from a misconfigured or badly designed UA by the way. Also realize I have never seen it even with remote user traversal WHEN I lick down pps to port 5060 in the firewall to a sane functional number. One assumes you inspected the logs to verify there was no outside attempt to spam calls via the proxy (I.e. INVITE)?

On Aug 2, 2012 12:05 PM, "andrewpitman" < ***@comcast.net > wrote:

<blockquote>

Hi Joegen!

I dug around a bit in the code, and I might have a starting
point for where to look for this...

When this bug has manifested itself, we've been able to
recover by restarting just sipXproxy, and not both proxy and
registrar, so the issue doesn't seem to be with registrar.
When the server stops responding to registration requests,
sipXproxy loses its connection to sipregistrar on port
5070/tcp. sipregistrar remains listening on port 5070 and
happily accepts the connection from sipXproxy when that's
restarted.

Also, based on Mike's comments when he visited the other
day, it does not seem like this issue has shown up for
installations which do not have many remote workers. In our
configurations here, with some exceptions all of our phones
are "remote workers," and in our setup we have to deal with
both near and far end NAT traversal.

Based on all that, and assuming the problem was probably
with sipXproxy (or sipXtackLib) and probably had something
to do with code dealing with NAT traversal, I came across
the code in sipXproxy that deals with processing of
forwardingrules.xml. If I'm reading the code (and comments)
correctly, it looks like if a request does not have route
state information (or if it does have NON-mutable route
state AND the URI is in the local domain AND globally
routable), forwarding rules is followed. Looking at the
forwarding rules xml file, it looks like the catchall
default is to send all other requests to the registry
service.

So, here's a thought... Given that forwarding to
sipregistrar is the default, what kind of malformed crap
could end up getting processed by that part of the code in
sipXproxy? It seems to me that it would be more likely to
bail there based on that.

I'll continue my digging there, but I wanted to let you
know. Would you mind having a look and let me know what you
think? I'll forward along some logs from a recent hang, as
well to make sure we're still on the right track.

Thanks!

Joegen Baclor wrote on Thu, 21 June 2012 12:50

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment. net

Helpdesk Customers: http://myhelp.myitdepartment. net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

</blockquote>
--
Michael Picher, Director of Technical Services
eZuce, Inc.

300 Brickstone Square

Suite 201

Andover, MA. 01810
O.978-296-1005 X2015
M.207-956-0262
@mpicher < http://twitter.com/mpicher >
linkedin
www.ezuce.com

------------------------------------------------------------------------------------------------------------
There are 10 kinds of people in the world, those who understand binary and those who don't.

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

a***@comcast.net

2012-08-02 22:27:49 UTC

Permalink

Okay, I uploaded the registrar and proxy logs from the time in question to the JIRA. AFAICT, the time this happened was at 2012-07-31T18:52:10.

A couple of things from sipXproxy.log that may or may not be pertinent:

At 18:41:32, there are a couple of socket errors.

Right before the hang at 18:52, there are a bunch of messages such as: "OsMsgQShared::doSendCore message queue 'AsynchMediaRelayRequestSender-16' is over half full - count = 99, max = 100"

At 18:52:55, the destructor for an object of class OsBSemLinux (defined in sipXportLib) is called. The log entry says : "OsBSemLinux::~OsBSemLinux pt_sem_destroy returned 16 in task 2966346640"

Hope this helps,
Andy

----- Original Message -----
From: "Michael Picher" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Thursday, August 2, 2012 12:42:12 PM
Subject: Re: [sipx-users] new patch for XX-10177

do you lick down a lot of ports? :-P

i think these are all Polycom phones with 3.2.6 firmware.

the only other thing a bit odd is they are coming through a Cisco ASA which is known to work but could be a question mark. I think they were going to try to route around this and then in through a pfSense box to see if the same thing happens.

mike

On Thu, Aug 2, 2012 at 12:28 PM, Tony Graziano < ***@myitdepartment.net > wrote:

The malformed crap could easily come from a misconfigured or badly designed UA by the way. Also realize I have never seen it even with remote user traversal WHEN I lick down pps to port 5060 in the firewall to a sane functional number. One assumes you inspected the logs to verify there was no outside attempt to spam calls via the proxy (I.e. INVITE)?

On Aug 2, 2012 12:05 PM, "andrewpitman" < ***@comcast.net > wrote:

<blockquote>

Hi Joegen!

I dug around a bit in the code, and I might have a starting
point for where to look for this...

When this bug has manifested itself, we've been able to
recover by restarting just sipXproxy, and not both proxy and
registrar, so the issue doesn't seem to be with registrar.
When the server stops responding to registration requests,
sipXproxy loses its connection to sipregistrar on port
5070/tcp. sipregistrar remains listening on port 5070 and
happily accepts the connection from sipXproxy when that's
restarted.

Also, based on Mike's comments when he visited the other
day, it does not seem like this issue has shown up for
installations which do not have many remote workers. In our
configurations here, with some exceptions all of our phones
are "remote workers," and in our setup we have to deal with
both near and far end NAT traversal.

Based on all that, and assuming the problem was probably
with sipXproxy (or sipXtackLib) and probably had something
to do with code dealing with NAT traversal, I came across
the code in sipXproxy that deals with processing of
forwardingrules.xml. If I'm reading the code (and comments)
correctly, it looks like if a request does not have route
state information (or if it does have NON-mutable route
state AND the URI is in the local domain AND globally
routable), forwarding rules is followed. Looking at the
forwarding rules xml file, it looks like the catchall
default is to send all other requests to the registry
service.

So, here's a thought... Given that forwarding to
sipregistrar is the default, what kind of malformed crap
could end up getting processed by that part of the code in
sipXproxy? It seems to me that it would be more likely to
bail there based on that.

I'll continue my digging there, but I wanted to let you
know. Would you mind having a look and let me know what you
think? I'll forward along some logs from a recent hang, as
well to make sure we're still on the right track.

Thanks!

Joegen Baclor wrote on Thu, 21 June 2012 12:50

Joegen Baclor

2012-08-03 02:28:09 UTC

Permalink

This is probably the most sensible cuase. a deadlock on transport
queues. Thanks for the info. Can you update the jira with this?
including the pertinent logs if possible.

Post by a***@comcast.net
Okay, I uploaded the registrar and proxy logs from the time in
question to the JIRA. AFAICT, the time this happened was at
2012-07-31T18:52:10.
At 18:41:32, there are a couple of socket errors.
"OsMsgQShared::doSendCore message queue
'AsynchMediaRelayRequestSender-16' is over half full - count = 99, max
= 100"
At 18:52:55, the destructor for an object of class OsBSemLinux
"OsBSemLinux::~OsBSemLinux pt_sem_destroy returned 16 in task 2966346640"
Hope this helps,
Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Thursday, August 2, 2012 12:42:12 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
do you lick down a lot of ports? :-P
i think these are all Polycom phones with 3.2.6 firmware.
the only other thing a bit odd is they are coming through a Cisco ASA
which is known to work but could be a question mark. I think they
were going to try to route around this and then in through a pfSense
box to see if the same thing happens.
mike
On Thu, Aug 2, 2012 at 12:28 PM, Tony Graziano
The malformed crap could easily come from a misconfigured or badly
designed UA by the way. Also realize I have never seen it even
with remote user traversal WHEN I lick down pps to port 5060 in
the firewall to a sane functional number. One assumes you
inspected the logs to verify there was no outside attempt to spam
calls via the proxy (I.e. INVITE)?
Hi Joegen!
I dug around a bit in the code, and I might have a starting
point for where to look for this...
When this bug has manifested itself, we've been able to
recover by restarting just sipXproxy, and not both proxy and
registrar, so the issue doesn't seem to be with registrar.
When the server stops responding to registration requests,
sipXproxy loses its connection to sipregistrar on port
5070/tcp. sipregistrar remains listening on port 5070 and
happily accepts the connection from sipXproxy when that's
restarted.
Also, based on Mike's comments when he visited the other
day, it does not seem like this issue has shown up for
installations which do not have many remote workers. In our
configurations here, with some exceptions all of our phones
are "remote workers," and in our setup we have to deal with
both near and far end NAT traversal.
Based on all that, and assuming the problem was probably
with sipXproxy (or sipXtackLib) and probably had something
to do with code dealing with NAT traversal, I came across
the code in sipXproxy that deals with processing of
forwardingrules.xml. If I'm reading the code (and comments)
correctly, it looks like if a request does not have route
state information (or if it does have NON-mutable route
state AND the URI is in the local domain AND globally
routable), forwarding rules is followed. Looking at the
forwarding rules xml file, it looks like the catchall
default is to send all other requests to the registry
service.
So, here's a thought... Given that forwarding to
sipregistrar is the default, what kind of malformed crap
could end up getting processed by that part of the code in
sipXproxy? It seems to me that it would be more likely to
bail there based on that.
I'll continue my digging there, but I wanted to let you
know. Would you mind having a look and let me know what you
think? I'll forward along some logs from a recent hang, as
well to make sure we're still on the right track.
Thanks!
Joegen Baclor wrote on Thu, 21 June 2012 12:50

--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
Telephone: 434.984.8426 <tel:434.984.8426>
Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
Michael Picher, Director of Technical Services
eZuce, Inc.
300 Brickstone Square
Suite 201
Andover, MA. 01810
O.978-296-1005 X2015
M.207-956-0262
@mpicher <http://twitter.com/mpicher>
linkedin <http://www.linkedin.com/profile/view?id=35504760&trk=tab_pro>
www.ezuce.com <http://www.ezuce.com>
------------------------------------------------------------------------------------------------------------
There are 10 kinds of people in the world, those who understand binary and those who don't.
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

a***@comcast.net

2012-08-08 20:10:37 UTC

Permalink

Hi Joegen,

Have you had any luck tracking down the root cause of this issue?

The problem just occurred again on another one of our systems. It looks like in addition to sipregistrar on tcp port 5070, sipXproxy also loses some other persistent connections to various services. Comparing the output of netstat from before and after the restart of sipXproxy, I noticed the following connections missing: sipregistrar 5075/tcp, sipstatus 5110/tcp, sipxsaa 5170/tcp and the media relay 9090/tcp.

We did see in the logs that once again, that destructor ran right before the problem was noticed.

I've updated the JIRA to that effect.

Thanks!
Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Thursday, August 2, 2012 10:28:09 PM
Subject: Re: [sipx-users] new patch for XX-10177

This is probably the most sensible cuase. a deadlock on transport queues. Thanks for the info. Can you update the jira with this? including the pertinent logs if possible.

On 08/03/2012 06:27 AM, ***@comcast.net wrote:

Okay, I uploaded the registrar and proxy logs from the time in question to the JIRA. AFAICT, the time this happened was at 2012-07-31T18:52:10.

A couple of things from sipXproxy.log that may or may not be pertinent:

At 18:41:32, there are a couple of socket errors.

Right before the hang at 18:52, there are a bunch of messages such as: "OsMsgQShared::doSendCore message queue 'AsynchMediaRelayRequestSender-16' is over half full - count = 99, max = 100"

At 18:52:55, the destructor for an object of class OsBSemLinux (defined in sipXportLib) is called. The log entry says: "OsBSemLinux::~OsBSemLinux pt_sem_destroy returned 16 in task 2966346640"

Hope this helps,
Andy

----- Original Message -----
From: "Michael Picher" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Thursday, August 2, 2012 12:42:12 PM
Subject: Re: [sipx-users] new patch for XX-10177

do you lick down a lot of ports? :-P

i think these are all Polycom phones with 3.2.6 firmware.

the only other thing a bit odd is they are coming through a Cisco ASA which is known to work but could be a question mark. I think they were going to try to route around this and then in through a pfSense box to see if the same thing happens.

mike

On Thu, Aug 2, 2012 at 12:28 PM, Tony Graziano < ***@myitdepartment.net > wrote:

<blockquote>

The malformed crap could easily come from a misconfigured or badly designed UA by the way. Also realize I have never seen it even with remote user traversal WHEN I lick down pps to port 5060 in the firewall to a sane functional number. One assumes you inspected the logs to verify there was no outside attempt to spam calls via the proxy (I.e. INVITE)?

On Aug 2, 2012 12:05 PM, "andrewpitman" < ***@comcast.net > wrote:

<blockquote>

Hi Joegen!

I dug around a bit in the code, and I might have a starting
point for where to look for this...

When this bug has manifested itself, we've been able to
recover by restarting just sipXproxy, and not both proxy and
registrar, so the issue doesn't seem to be with registrar.
When the server stops responding to registration requests,
sipXproxy loses its connection to sipregistrar on port
5070/tcp. sipregistrar remains listening on port 5070 and
happily accepts the connection from sipXproxy when that's
restarted.

Also, based on Mike's comments when he visited the other
day, it does not seem like this issue has shown up for
installations which do not have many remote workers. In our
configurations here, with some exceptions all of our phones
are "remote workers," and in our setup we have to deal with
both near and far end NAT traversal.

Based on all that, and assuming the problem was probably
with sipXproxy (or sipXtackLib) and probably had something
to do with code dealing with NAT traversal, I came across
the code in sipXproxy that deals with processing of
forwardingrules.xml. If I'm reading the code (and comments)
correctly, it looks like if a request does not have route
state information (or if it does have NON-mutable route
state AND the URI is in the local domain AND globally
routable), forwarding rules is followed. Looking at the
forwarding rules xml file, it looks like the catchall
default is to send all other requests to the registry
service.

So, here's a thought... Given that forwarding to
sipregistrar is the default, what kind of malformed crap
could end up getting processed by that part of the code in
sipXproxy? It seems to me that it would be more likely to
bail there based on that.

I'll continue my digging there, but I wanted to let you
know. Would you mind having a look and let me know what you
think? I'll forward along some logs from a recent hang, as
well to make sure we're still on the right track.

Thanks!

Joegen Baclor wrote on Thu, 21 June 2012 12:50

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

</blockquote>
--
Michael Picher, Director of Technical Services
eZuce, Inc.

300 Brickstone Square

Suite 201

Andover, MA. 01810
O.978-296-1005 X2015
M.207-956-0262
@mpicher < http://twitter.com/mpicher >
linkedin
www.ezuce.com

------------------------------------------------------------------------------------------------------------
There are 10 kinds of people in the world, those who understand binary and those who don't.

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

andrewpitman

2012-08-20 13:33:48 UTC

Permalink

Joegen, it looks like the keepalive processing in
SipClient.cpp in the release-4.4 sipXtackLib is still very
specific in what it matches. I thought it was supposed to
consider any combination of CR's and LF's (as long as the
buffer contains only these) as a keepalive?

Joegen Baclor wrote on Thu, 02 August 2012 22:28

Post by Joegen Baclor
Then it is not the issue. However, that perl script

just

Post by Joegen Baclor
became a DOS
attack tool against sipx so we need to accept the

patch

Post by Joegen Baclor
just to prevent
what you have done from happening. Even if the

patch

Post by Joegen Baclor
did not really
solve your issue, would you mind applying it and see

Post by Joegen Baclor
you can fill up
/var/ folder with the patch active?
Regarding the freezing issue, I'm gonna have to

look

Post by Joegen Baclor
again. If you have
a hunch where I should be looking at, let me know.

Joegen Baclor

2012-08-21 00:41:01 UTC

Permalink

I'll discuss this with GEroge later. Thanks for pointing it out.

Post by andrewpitman
Joegen, it looks like the keepalive processing in
SipClient.cpp in the release-4.4 sipXtackLib is still very
specific in what it matches. I thought it was supposed to
consider any combination of CR's and LF's (as long as the
buffer contains only these) as a keepalive?
Joegen Baclor wrote on Thu, 02 August 2012 22:28

Post by Joegen Baclor
Then it is not the issue. However, that perl script

just

Post by Joegen Baclor
became a DOS
attack tool against sipx so we need to accept the

patch

Post by Joegen Baclor
just to prevent
what you have done from happening. Even if the

patch

Post by Joegen Baclor
did not really
solve your issue, would you mind applying it and see

Post by Joegen Baclor
you can fill up
/var/ folder with the patch active?
Regarding the freezing issue, I'm gonna have to

look

Post by Joegen Baclor
again. If you have
a hunch where I should be looking at, let me know.

andrewpitman

2012-08-21 09:59:23 UTC

Permalink

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

Tony Graziano

2012-08-21 17:01:03 UTC

Permalink

can you elaborate what the UA's are that are involved in that transaction?

Post by andrewpitman
Joegen, George,
I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"
Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.
Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>

a***@comcast.net

2012-08-21 17:06:24 UTC

Permalink

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Joegen Baclor

2012-08-22 02:16:20 UTC

Permalink

I've taken a quick look at the NAT traversal plugin and it seems to be
not accounting PUBLISH as a potential stream modifier. But it is not
clear to me how it would cause a hang. Can you disable session-timers
in the phones and see if that somehow lessens the occurence of this
incident?

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670
and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 1:01:03 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
can you elaborate what the UA's are that are involved in that transaction?
On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman
Joegen, George,
I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"
Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.
Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about
sipX-CoLab 2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>
Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Dave Deutschman

2012-08-22 17:58:17 UTC

Permalink

I have also seen this on a 4.4 system.

Calls were not being processed and the caller heard dead air. When a call
was researched, I found that sipXproxy was loading the NAT plugin for the
call even though the call was outbound to the PSTN via an AudioCodes
gateway. On this system, NAT traversal was enabled and the Address Type
was set to STUN. Since the system was not supporting remote users, we
disabled NAT Traversal and changed the Address Type to a public IP address.
The problem has not reoccurred since the changes to the configuration of the
system.

From: sipx-users-***@list.sipfoundry.org
[mailto:sipx-users-***@list.sipfoundry.org] On Behalf Of Joegen Baclor
Sent: Tuesday, August 21, 2012 7:16 PM
To: Discussion list for users of sipXecs software
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not
accounting PUBLISH as a potential stream modifier. But it is not clear to
me how it would cause a hang. Can you disable session-timers in the phones
and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a
SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

_____

From: "Tony Graziano" <mailto:***@myitdepartment.net>
<***@myitdepartment.net>
To: "Discussion list for users of sipXecs software"
<mailto:sipx-***@list.sipfoundry.org> <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman <***@comcast.net>
wrote:

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone:
<file:///C:\Program%20Files%20(x86)\eZuce\openUC%20Outlook%20Add-in\StatusIm
ages\phone.png> 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013! <http://sipxcolab2013.eventbrite.com/?discount=tony2013>

LAN/Telephony/Security and Control Systems Helpdesk:

Telephone:
<file:///C:\Program%20Files%20(x86)\eZuce\openUC%20Outlook%20Add-in\StatusIm
ages\phone.png> 434.984.8426

sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net

Blog: http://blog.myitdepartment.net

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_____

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2197 / Virus Database: 2437/5215 - Release Date: 08/21/12

a***@comcast.net

2012-08-22 17:58:57 UTC

Permalink

Joegen,

Actually, it looks like we have the keepalive.sessionTimers option turned off for all our phones anyway. I looked in the pcap we have from a hang just this morning and I didn't see any messages matching the PUBLISH method. I can send you these off list if you'd like.

-Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Tuesday, August 21, 2012 10:16:20 PM
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not accounting PUBLISH as a potential stream modifier. But it is not clear to me how it would cause a hang. Can you disable session-timers in the phones and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

Joegen Baclor

2012-08-22 21:32:34 UTC

Permalink

Ooops, typo. Its not PUBLISH but UPDATE.

Post by andrewpitman
Joegen,
Actually, it looks like we have the keepalive.sessionTimers option
turned off for all our phones anyway. I looked in the pcap we have
from a hang just this morning and I didn't see any messages matching
the PUBLISH method. I can send you these off list if you'd like.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 10:16:20 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
I've taken a quick look at the NAT traversal plugin and it seems to be
not accounting PUBLISH as a potential stream modifier. But it is not
clear to me how it would cause a hang. Can you disable session-timers
in the phones and see if that somehow lessens the occurence of this
incident?
They're all Polycom phones. Mostly 450s, with a couple of 550s, a
670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with
bootrom 4.3.1.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 1:01:03 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
can you elaborate what the UA's are that are involved in that transaction?
On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman
Joegen, George,
I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"
Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.
Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>
Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
_______________________________________________
sipx-users mailing list
List Archive:http://list.sipfoundry.org/archive/sipx-users/

a***@comcast.net

2012-08-23 16:13:15 UTC

Permalink

Okay, I don't see any UPDATE messages in the half hour preceding the hang.

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, August 22, 2012 5:32:34 PM
Subject: Re: [sipx-users] new patch for XX-10177

Ooops, typo. Its not PUBLISH but UPDATE.

On 08/23/2012 01:58 AM, ***@comcast.net wrote:

Joegen,

Actually, it looks like we have the keepalive.sessionTimers option turned off for all our phones anyway. I looked in the pcap we have from a hang just this morning and I didn't see any messages matching the PUBLISH method. I can send you these off list if you'd like.

-Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Tuesday, August 21, 2012 10:16:20 PM
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not accounting PUBLISH as a potential stream modifier. But it is not clear to me how it would cause a hang. Can you disable session-timers in the phones and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

<blockquote>

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

</blockquote>

a***@comcast.net

2012-08-23 16:13:57 UTC

Permalink

The hang happened right at the end of this one.

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, August 22, 2012 5:32:34 PM
Subject: Re: [sipx-users] new patch for XX-10177

Ooops, typo. Its not PUBLISH but UPDATE.

On 08/23/2012 01:58 AM, ***@comcast.net wrote:

Joegen,

Actually, it looks like we have the keepalive.sessionTimers option turned off for all our phones anyway. I looked in the pcap we have from a hang just this morning and I didn't see any messages matching the PUBLISH method. I can send you these off list if you'd like.

-Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Tuesday, August 21, 2012 10:16:20 PM
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not accounting PUBLISH as a potential stream modifier. But it is not clear to me how it would cause a hang. Can you disable session-timers in the phones and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

<blockquote>

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

</blockquote>

a***@comcast.net

2012-09-12 19:19:55 UTC

Permalink

Joegen,

We recently upgraded a bunch of our servers to 4.4.0 update #18, and while this issue doesn't occur as frequently, we are still experiencing it.

I was going to try running a version that has the original patch (any combination of CR-LFs in a keepalive message) and see if that gets us anywhere, but not being able to produce a hang on demand kind of limits our testing options for this.

Not being totally convinced it is the keepalive processing, are there any other places we could look for the root cause of this problem?

Thanks,
Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, August 22, 2012 5:32:34 PM
Subject: Re: [sipx-users] new patch for XX-10177

Ooops, typo. Its not PUBLISH but UPDATE.

On 08/23/2012 01:58 AM, ***@comcast.net wrote:

Joegen,

Actually, it looks like we have the keepalive.sessionTimers option turned off for all our phones anyway. I looked in the pcap we have from a hang just this morning and I didn't see any messages matching the PUBLISH method. I can send you these off list if you'd like.

-Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Tuesday, August 21, 2012 10:16:20 PM
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not accounting PUBLISH as a potential stream modifier. But it is not clear to me how it would cause a hang. Can you disable session-timers in the phones and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

<blockquote>

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

</blockquote>

Joegen Baclor

2012-09-19 11:42:48 UTC

Permalink

Andrewe any news on the patch i sent you offlist?

Post by andrewpitman
Joegen,
We recently upgraded a bunch of our servers to 4.4.0 update #18, and
while this issue doesn't occur as frequently, we are still
experiencing it.
I was going to try running a version that has the original patch (any
combination of CR-LFs in a keepalive message) and see if that gets us
anywhere, but not being able to produce a hang on demand kind of
limits our testing options for this.
Not being totally convinced it is the keepalive processing, are there
any other places we could look for the root cause of this problem?
Thanks,
Andy
------------------------------------------------------------------------
*Cc: *"Discussion list for users of sipXecs software"
*Sent: *Wednesday, August 22, 2012 5:32:34 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
Ooops, typo. Its not PUBLISH but UPDATE.
Joegen,
Actually, it looks like we have the keepalive.sessionTimers option
turned off for all our phones anyway. I looked in the pcap we
have from a hang just this morning and I didn't see any messages
matching the PUBLISH method. I can send you these off list if
you'd like.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 10:16:20 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
I've taken a quick look at the NAT traversal plugin and it seems
to be not accounting PUBLISH as a potential stream modifier. But
it is not clear to me how it would cause a hang. Can you disable
session-timers in the phones and see if that somehow lessens the
occurence of this incident?
They're all Polycom phones. Mostly 450s, with a couple of
550s, a 670 and a SoundStation 6000 in the mix. Firmware
3.2.6 with bootrom 4.3.1.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 1:01:03 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
can you elaborate what the UA's are that are involved in that transaction?
On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman
Joegen, George,
I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"
Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.
Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

a***@comcast.net

2012-09-19 20:58:24 UTC

Permalink

Hi Joegen,

I just put it on my test system. We'll see. ..

Thanks!
Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, September 19, 2012 7:42:48 AM
Subject: Re: [sipx-users] new patch for XX-10177

Andrewe any news on the patch i sent you offlist?

On 09/13/2012 03:19 AM, ***@comcast.net wrote:

Joegen,

We recently upgraded a bunch of our servers to 4.4.0 update #18, and while this issue doesn't occur as frequently, we are still experiencing it.

I was going to try running a version that has the original patch (any combination of CR-LFs in a keepalive message) and see if that gets us anywhere, but not being able to produce a hang on demand kind of limits our testing options for this.

Not being totally convinced it is the keepalive processing, are there any other places we could look for the root cause of this problem?

Thanks,
Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, August 22, 2012 5:32:34 PM
Subject: Re: [sipx-users] new patch for XX-10177

Ooops, typo. Its not PUBLISH but UPDATE.

On 08/23/2012 01:58 AM, ***@comcast.net wrote:

<blockquote>

Joegen,

Actually, it looks like we have the keepalive.sessionTimers option turned off for all our phones anyway. I looked in the pcap we have from a hang just this morning and I didn't see any messages matching the PUBLISH method. I can send you these off list if you'd like.

-Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Tuesday, August 21, 2012 10:16:20 PM
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not accounting PUBLISH as a potential stream modifier. But it is not clear to me how it would cause a hang. Can you disable session-timers in the phones and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

<blockquote>

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

</blockquote>

</blockquote>

Joegen Baclor

2012-09-25 05:44:22 UTC

Permalink

Andrew,

any update on this?

Post by Andrew Pitman
Hi Joegen,
I just put it on my test system. We'll see...
Thanks!
Andy
------------------------------------------------------------------------
*Cc: *"Discussion list for users of sipXecs software"
*Sent: *Wednesday, September 19, 2012 7:42:48 AM
*Subject: *Re: [sipx-users] new patch for XX-10177
Andrewe any news on the patch i sent you offlist?
Joegen,
We recently upgraded a bunch of our servers to 4.4.0 update #18,
and while this issue doesn't occur as frequently, we are still
experiencing it.
I was going to try running a version that has the original patch
(any combination of CR-LFs in a keepalive message) and see if that
gets us anywhere, but not being able to produce a hang on demand
kind of limits our testing options for this.
Not being totally convinced it is the keepalive processing, are
there any other places we could look for the root cause of this
problem?
Thanks,
Andy
------------------------------------------------------------------------
*Cc: *"Discussion list for users of sipXecs software"
*Sent: *Wednesday, August 22, 2012 5:32:34 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
Ooops, typo. Its not PUBLISH but UPDATE.
Joegen,
Actually, it looks like we have the keepalive.sessionTimers
option turned off for all our phones anyway. I looked in the
pcap we have from a hang just this morning and I didn't see
any messages matching the PUBLISH method. I can send you these
off list if you'd like.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 10:16:20 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
I've taken a quick look at the NAT traversal plugin and it
seems to be not accounting PUBLISH as a potential stream
modifier. But it is not clear to me how it would cause a hang.
Can you disable session-timers in the phones and see if that
somehow lessens the occurence of this incident?
They're all Polycom phones. Mostly 450s, with a couple of
550s, a 670 and a SoundStation 6000 in the mix. Firmware
3.2.6 with bootrom 4.3.1.
-Andy
------------------------------------------------------------------------
*To: *"Discussion list for users of sipXecs software"
*Sent: *Tuesday, August 21, 2012 1:01:03 PM
*Subject: *Re: [sipx-users] new patch for XX-10177
can you elaborate what the UA's are that are involved in
that transaction?
On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman
Joegen, George,
I noticed some messages in my sipXproxy log from a
hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see
messages
"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected
event
SuccessfulResponse while in state 'WaitingForInvite'"
Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption
every time
this happens.
Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me
about sipX-CoLab 2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>
Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
_______________________________________________
sipx-users mailing list
List Archive:http://list.sipfoundry.org/archive/sipx-users/

Gerald Drouillard

2012-09-25 12:44:16 UTC

Permalink

Post by Joegen Baclor
Andrew,
any update on this?

--
Regards
--------------------------------------
Gerald Drouillard
Technology Architect
Drouillard & Associates, Inc.
http://www.Drouillard.biz

Joegen Baclor

2012-09-25 13:04:35 UTC

Permalink

Thanks Gerald,

Good to hear you are getting better mileage with patch #19. I sent a
patch to Andrew that is not included yet with patch #19. It has
something to do with TCP packets not getting reported to NAT traversal
plugin.

Joegen

Post by Gerald Drouillard

Post by Joegen Baclor
Andrew,
any update on this?

Update #19 fixed many issues we were having with offsite registration.
With #18 we had turned off 5061 and that seemed to fix most clients.
We have not turned on 5061 with #19 yet. I would like a day or two of
stability before working it back in.
--
Regards
--------------------------------------
Gerald Drouillard
Technology Architect
Drouillard & Associates, Inc.
http://www.Drouillard.biz
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

a***@comcast.net

2012-09-26 18:16:25 UTC

Permalink

Hi Joegen!

So far everything seems to work with the 4.4.0 update #19 build I made yesterday with your sipXtackLib patch. At least, nothing new is broken but since we are unable to reproduce the hang in sipXproxy, I'm not sure how useful our testing has been as far as a resolution to this JIRA goes.

Any ideas on how we might tickle this?

Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, September 25, 2012 1:44:22 AM
Subject: Re: [sipx-users] new patch for XX-10177

Andrew,

any update on this?

On 09/20/2012 04:58 AM, ***@comcast.net wrote:

Hi Joegen,

I just put it on my test system. We'll see...

Thanks!
Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, September 19, 2012 7:42:48 AM
Subject: Re: [sipx-users] new patch for XX-10177

Andrewe any news on the patch i sent you offlist?

On 09/13/2012 03:19 AM, ***@comcast.net wrote:

<blockquote>

Joegen,

We recently upgraded a bunch of our servers to 4.4.0 update #18, and while this issue doesn't occur as frequently, we are still experiencing it.

I was going to try running a version that has the original patch (any combination of CR-LFs in a keepalive message) and see if that gets us anywhere, but not being able to produce a hang on demand kind of limits our testing options for this.

Not being totally convinced it is the keepalive processing, are there any other places we could look for the root cause of this problem?

Thanks,
Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: ***@comcast.net
Cc: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Wednesday, August 22, 2012 5:32:34 PM
Subject: Re: [sipx-users] new patch for XX-10177

Ooops, typo. Its not PUBLISH but UPDATE.

On 08/23/2012 01:58 AM, ***@comcast.net wrote:

<blockquote>

Joegen,

Actually, it looks like we have the keepalive.sessionTimers option turned off for all our phones anyway. I looked in the pcap we have from a hang just this morning and I didn't see any messages matching the PUBLISH method. I can send you these off list if you'd like.

-Andy

----- Original Message -----
From: "Joegen Baclor" <***@ezuce.com>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Cc: ***@comcast.net
Sent: Tuesday, August 21, 2012 10:16:20 PM
Subject: Re: [sipx-users] new patch for XX-10177

I've taken a quick look at the NAT traversal plugin and it seems to be not accounting PUBLISH as a potential stream modifier. But it is not clear to me how it would cause a hang. Can you disable session-timers in the phones and see if that somehow lessens the occurence of this incident?

On 08/22/2012 01:06 AM, ***@comcast.net wrote:

<blockquote>

They're all Polycom phones. Mostly 450s, with a couple of 550s, a 670 and a SoundStation 6000 in the mix. Firmware 3.2.6 with bootrom 4.3.1.

-Andy

----- Original Message -----
From: "Tony Graziano" <***@myitdepartment.net>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Tuesday, August 21, 2012 1:01:03 PM
Subject: Re: [sipx-users] new patch for XX-10177

can you elaborate what the UA's are that are involved in that transaction?

On Tue, Aug 21, 2012 at 5:59 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Joegen, George,

I noticed some messages in my sipXproxy log from a hang on a
customer system just today, which may or may not be
pertinent. Besides the parsing errors, I also see messages
such as this immediately before the hang:

"2012-08-21T15:40:56.031862Z"
:269589:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'83226727-F4CE51E8': Received unexpected
event InviteRequest while in state 'WaitingForMediaOffer'"
"2012-08-21T15:40:56.233883Z"
:269590:SIP:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "SipUserAgent::send INVITE request matches
existing transaction"
"2012-08-21T15:40:56.323512Z"
:269591:SIP:ERR:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB90:S
ipXProxy: "SipUserAgent::handleMessage SIP message timeout
expired with no matching transaction"
"2012-08-21T15:40:56.414306Z"
:269592:NAT:WARNING:pbx1.sipdomain.com:SipRouter-15:B5DB9B90
:SipXProxy: "'5FHSHt9cQ2XBS':Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.516706Z"
:269593:NAT:WARNING:pbx1.sipdomain.com:SipUserAgent-2:B5EBAB
90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
UpdateRequest while in state 'WaitingForInvite'"
"2012-08-21T15:40:56.523583Z"
:269594:NAT:WARNING:pbx1.sipdomain.com:SipClientTcp-4212:B19
FAB90:SipXProxy: "'5FHSHt9cQ2XBS': Received unexpected event
SuccessfulResponse while in state 'WaitingForInvite'"

Since this has been happening to some of our customers more
than once within a 24 hour period, in order to keep them
from cancelling with us we've had to resort to a watchdog
script which sends OPTIONS messages to the servers
periodically and restarts sipXproxy if it fails to respond.
Obviously this is just a band-aid workaround until we can
resolve this issue, since this means they incur
approximately 3-4 minutes of service interruption every time
this happens.

Joegen Baclor wrote on Mon, 20 August 2012 13:45

Post by Joegen Baclor
I'll discuss this with GEroge later. Thanks for pointing
it out.

--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab 2013!

LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

_______________________________________________
sipx-users mailing list sipx-***@list.sipfoundry.org List Archive: http://list.sipfoundry.org/archive/sipx-users/
</blockquote>

</blockquote>

</blockquote>

</blockquote>

George Niculae

2012-08-22 09:18:26 UTC

Permalink

Andy, the latest 4.4 RPMs does not include patch for XX-10177 so most
likely you'll still see the same issue (or are you still running RPMs
you built with the patch)

George

andrewpitman

2012-08-02 16:57:30 UTC

Permalink

Yeah, we've ruled that out. If it is a UA sending malformed
SIP traffic, sipXproxy still should drop that or respond
with an error (and maybe write a log entry to that effect)
but it should not cause it to lose its socket connection to
sipregistrar and have to be restarted.

Tony Graziano wrote on Thu, 02 August 2012 12:28

Post by Tony Graziano
The malformed crap could easily come from a
misconfigured or badly designed
UA by the way. Also realize I have never seen it even
with remote user
traversal WHEN I lick down pps to port 5060 in the
firewall to a sane
functional number. One assumes you inspected the logs to
verify there was
no outside attempt to spam calls via the proxy (I.e.
INVITE)?

Joegen Baclor

2012-06-22 04:28:04 UTC

Permalink

Hi Andrew,

Yes you have a point. Modifying the for loop to something like below
should cover the situation you have pointed out

int crlfCount = 0;
for (int i = 0; i< res; i++)
{
if (readBuffer(i) == '\r' || readBuffer(i) == '\n')
{
crlfCount+++;
} else
{
break;
}
}

If you can generate your own rpms, feel free to modify and attach the patch to the tracker. If you are not setup to build rpms, let me know, I'll compile a new RPM withi this change. However, you will ahve to point your repo to stage because there has been changes to some sipXtackLib header components recently that would mean you need to update all c++ components that links to sipxtacklib.

andrewpitman

2012-06-22 15:28:42 UTC

Permalink

Nah, I can just pull the source down with git, patch,
compile and build my own rpms. Thanks!

I'll also post the patch to the JIRA.
--

Hay, Nathan

2012-06-22 18:43:36 UTC

Permalink

I believe we are experiencing this problem.

When I turn up the logging, I get these lines over and over again in the
sipXproxy log:

"2012-06-22T17:33:13.804499Z":26:SIP:WARNING:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"SipMessage::getResponseSendAddress
No VIA address, using

From address"

Nah, I can just pull the source down with git, patch,
compile and build my own rpms. Thanks!
I'll also post the patch to the JIRA.
--
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

--
Nathan P. Hay
Network Engineer | Information Technology
Cedarville University | www.cedarville.edu
937-766-7905
twitter: @nathanphay

Domenico Chierico

2012-06-25 09:30:52 UTC

Permalink

see something strange in the code, can this be a leftover:

if (readBuffer(i) == '\r' && readBuffer(i) == '\n')

shouldn't be something like:

if (readBuffer(i) == '\r' && readBuffer(i+1) == '\n')

Post by Hay, Nathan
I believe we are experiencing this problem.
When I turn up the logging, I get these lines over and over again in the
"2012-06-22T17:33:13.804499Z":26:SIP:WARNING:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"SipMessage::getResponseSendAddress
No VIA address, using

From address"

--
Nathan P. Hay
Network Engineer | Information Technology
Cedarville University | www.cedarville.edu
937-766-7905
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Domenico Chierico

2012-06-25 10:56:19 UTC

Permalink

or maybe

if (readBuffer(i) == '\r' || readBuffer(i) == '\n')

On Mon, Jun 25, 2012 at 11:30 AM, Domenico Chierico <

Post by Domenico Chierico
if (readBuffer(i) == '\r' && readBuffer(i) == '\n')
if (readBuffer(i) == '\r' && readBuffer(i+1) == '\n')

From address"

George Niculae

2012-06-25 10:57:47 UTC

Permalink

On Mon, Jun 25, 2012 at 1:56 PM, Domenico Chierico

Post by Domenico Chierico
or maybe
if (readBuffer(i) == '\r' || readBuffer(i) == '\n')
On Mon, Jun 25, 2012 at 11:30 AM, Domenico Chierico

Post by Domenico Chierico
if (readBuffer(i) == '\r' && readBuffer(i) == '\n')
if (readBuffer(i) == '\r' && readBuffer(i+1) == '\n')

I guess that's an leftover, it should be if (readBuffer(i) == '\r' &&
readBuffer(i+1) == '\n'), Joegen please confirm this,

Thanks
George

a***@comcast.net

2012-06-25 14:26:47 UTC

Permalink

Yeah, I had corrected that in the patch I posted to the JIRA.

----- Original Message -----
From: "Domenico Chierico" <***@sip2ser.it>
To: "Discussion list for users of sipXecs software" <sipx-***@list.sipfoundry.org>
Sent: Monday, June 25, 2012 6:56:19 AM
Subject: Re: [sipx-users] new patch for XX-10177

or maybe

if (readBuffer(i) == '\r' || readBuffer(i) == '\n')

On Mon, Jun 25, 2012 at 11:30 AM, Domenico Chierico < ***@sip2ser.it > wrote:

see something strange in the code, can this be a leftover:

if (readBuffer(i) == '\r' && readBuffer(i) == '\n')

shouldn't be something like:

if (readBuffer(i) == '\r' && readBuffer(i+1) == '\n')

On Fri, Jun 22, 2012 at 8:43 PM, Hay, Nathan < ***@cedarville.edu > wrote:

<blockquote>
I believe we are experiencing this problem.

When I turn up the logging, I get these lines over and over again in the sipXproxy log:

"2012-06-22T17:33:13.804499Z":26:SIP:WARNING:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"SipMessage::getResponseSendAddress No VIA address, using

From address"

"2012-06-22T17:33:13.804534Z":27:SIP:ERR:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"SipMessage::convertProtocolStringToEnum unrecognized protocol:"
"2012-06-22T17:33:13.804561Z":28:SIP:ERR:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"SipTransaction::doFirstSend protocol not explicitly set - using
UDP"
"2012-06-22T17:33:13.804765Z":29:SIP:ERR:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"Url::parseString no valid host found at char 0 in ';tag=jSD67t',
uriForm = name-addr"

This is causing high CPU usage and crashing our system every week or so. We have about 700 users and 550 phones defined on our system.

Any suggestions for a resolution? Can I provide anything from our system to help you resolve this bug?

Thanks,

Nathan

On Fri, Jun 22, 2012 at 11:28 AM, andrewpitman < ***@comcast.net > wrote:

<blockquote>

Nah, I can just pull the source down with git, patch,
compile and build my own rpms. Thanks!

I'll also post the patch to the JIRA.
--
_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/
--
Nathan P. Hay
Network Engineer | Information Technology
Cedarville University | www.cedarville.edu
937-766-7905
twitter: @nathanphay

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

</blockquote>

</blockquote>

_______________________________________________
sipx-users mailing list
sipx-***@list.sipfoundry.org
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Domenico Chierico

2012-06-25 14:41:56 UTC

Permalink

so this doesn't fix the issue anyway?

Post by a***@comcast.net
Yeah, I had corrected that in the patch I posted to the JIRA.
------------------------------
*To: *"Discussion list for users of sipXecs software" <
*Sent: *Monday, June 25, 2012 6:56:19 AM
*Subject: *Re: [sipx-users] new patch for XX-10177
or maybe
if (readBuffer(i) == '\r' || readBuffer(i) == '\n')
On Mon, Jun 25, 2012 at 11:30 AM, Domenico Chierico <

Post by Domenico Chierico
if (readBuffer(i) == '\r' && readBuffer(i) == '\n')
if (readBuffer(i) == '\r' && readBuffer(i+1) == '\n')

Post by Hay, Nathan
I believe we are experiencing this problem.
"2012-06-22T17:33:13.804499Z":26:SIP:WARNING:sip1.cedarville.edu:SipClientTcp-18:4159E940:SipXProxy:"SipMessage::getResponseSendAddress
No VIA address, using

From address"

_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

andrewpitman

2012-06-25 15:24:34 UTC

Permalink

No, but it does keep sipXproxy from logging when it receives
these keepalives (or similar), heading off a potential
avenue for a DOS attack. If you get enough of them, it
could fill the filesystem sipxecs logs to.

Andy

Domenico Chierico wrote on Mon, 25 June 2012 10:41

Post by Domenico Chierico
so this doesn't fix the issue anyway?

Joegen Baclor

2012-06-20 02:00:37 UTC

Permalink

Andrew,

Another possibility is TCP. We are sending it UDP currently. Can you
also try sending through TCP?

Yes that is it. From the logs that was previously posted, this was the most evident thing that was going crazy before the server was hung. So two things,
1. We are totally mistaken that this was the cause.
2. 1 packet every 5 seconds is not enough to cause havoc in the system.
I am hoping it is the latter. If you do not mind, try to modify the PERL script to make it send several packets in a tight loop. Lets say 10 every second. Let that run for an hour on a busy afternoon and see if you can get sipx bend down to its knees.

Post by andrewpitman
Joegen, like this?

Andrew Pitman

2012-06-20 02:03:23 UTC

Permalink

Sure, I'll give that a try.

--
Sent from my iPhone appendage

Post by Joegen Baclor
Andrew,
Another possibility is TCP. We are sending it UDP currently. Can you also try sending through TCP?

Yes that is it. From the logs that was previously posted, this was the most evident thing that was going crazy before the server was hung. So two things,
1. We are totally mistaken that this was the cause.
2. 1 packet every 5 seconds is not enough to cause havoc in the system.
I am hoping it is the latter. If you do not mind, try to modify the PERL script to make it send several packets in a tight loop. Lets say 10 every second. Let that run for an hour on a busy afternoon and see if you can get sipx bend down to its knees.

Post by andrewpitman
Joegen, like this?