High CPU sipXproxy (update #22)

Discussion:

Tony Graziano

2012-11-05 19:16:39 UTC

I am looking at a strange issue with a system which had a drive failure. We
replaced the drive and reloaded (did not restore) the system, then updated
it to the latest update. We see the proxy staying steady at 10% CPU, with
not active calls or transactions. It is a basic system with trunking and 12
phones, there should not be such a load.

I have sent the server its profiles. I have restarted the system. There is
no memory or swap memory issue. I have reviewed the configuration and all
of the speeddials and registrations.

The first thing I noticed is that noone was able to place outbound calls
easily, then when I started looking into it I checked user speeddials,
presense and overall configuration and hardware functionality. I still see
no issues except that the sipXproxy is taking up "enormous" CPU time. There
are 12 phones and a total of 24 subscriptions. Does anyone else have an
install similar and can verify whether they are seeing this or not?
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>

--
LAN/Telephony/Security and Control Systems Helpdesk:
Telephone: 434.984.8426
sip: ***@voice.myitdepartment.net

Helpdesk Customers: http://myhelp.myitdepartment.net
Blog: http://blog.myitdepartment.net

George Niculae

2012-11-05 20:16:44 UTC

Permalink

On Mon, Nov 5, 2012 at 9:16 PM, Tony Graziano

Post by Tony Graziano
I am looking at a strange issue with a system which had a drive failure.
We replaced the drive and reloaded (did not restore) the system, then
updated it to the latest update. We see the proxy staying steady at 10%
CPU, with not active calls or transactions. It is a basic system with
trunking and 12 phones, there should not be such a load.
I have sent the server its profiles. I have restarted the system. There is
no memory or swap memory issue. I have reviewed the configuration and all
of the speeddials and registrations.
The first thing I noticed is that noone was able to place outbound calls
easily, then when I started looking into it I checked user speeddials,
presense and overall configuration and hardware functionality. I still see
no issues except that the sipXproxy is taking up "enormous" CPU time. There
are 12 phones and a total of 24 subscriptions. Does anyone else have an
install similar and can verify whether they are seeing this or not?

Tony,

we didn't see this during regression testing patch 22 or on sites that we
upgraded. Is there any weird activity in proxy / registrar logs?

George

Gerald Drouillard

2012-11-05 21:11:26 UTC

Permalink

Post by Tony Graziano
I am looking at a strange issue with a system which had a drive
failure. We replaced the drive and reloaded (did not restore) the
system, then updated it to the latest update. We see the proxy staying
steady at 10% CPU, with not active calls or transactions. It is a
basic system with trunking and 12 phones, there should not be such a load.
I have sent the server its profiles. I have restarted the system.
There is no memory or swap memory issue. I have reviewed the
configuration and all of the speeddials and registrations.
The first thing I noticed is that noone was able to place outbound
calls easily, then when I started looking into it I checked user
speeddials, presense and overall configuration and hardware
functionality. I still see no issues except that the sipXproxy is
taking up "enormous" CPU time. There are 12 phones and a total of 24
subscriptions. Does anyone else have an install similar and can verify
whether they are seeing this or not?
--

Actually we see a little bit of a decrease since Oct 1ish.

In this one we have 83 active registrations about 20 offsite and 8G
memory with and SSD drive.

Did you look at the logging levels?
You can always grab a snapshot and look at what log file is the biggest
for an indication of where the activity is or even a packet capture on
the server.

Tony Graziano

2012-11-05 21:50:02 UTC

Permalink

I am seeing the following message within the rls logs:

sipxrls:"OsSSLServerSocket SSL_accept SSL handshake error:\n SSL error: 1
'error:00000001:lib(0):func(0):reason(1)'"
sipxrls:"OsSSLServerSocket SSL_accept SSL handshake error:\n SSL error:
336027900 'error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
protocol'"
sipxrls:"SipPublishContentMgr::getContent no container found for key
'sip:~~rl~C~~~id~xmpprlsclient...

(as is relates to the RLS component)

So I am wondering if someone can explain what the "unknown protocol" means
in this instance. The certificate was created in the exact way it should
have, by the system, one time at startup. I see presenceserver says
disabled but shows "running" in sipxconfig and if I start manually via
sipxproc it stays "running" (no change in sipxconfig).

I then tried to disable TLS and that broke nat traversal rules and failed
to start proxy, so that did not help.

I tried deleting the tmp imdb.* files and restarting presence from
sipxconfig but that did not help. The ownership of the files and sizes look
accurate (they were recreated when I restarted presence manually).

So this is SOLVED as far as the CPU level is concerned. I found a device
that has not been reconfigured (a valcom paging gateway) that is
essentially trying to register without an account, and the registrar logs
show 50-100 per minute (attempts).

I still think there is an SSL issue. Does anyone have any ideas on how to
figure this out?

On Mon, Nov 5, 2012 at 2:16 PM, Tony Graziano

--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
sip: ***@voice.myitdepartment.net
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
Linked-In Profile:
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~

Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>

Gerald Drouillard

2012-11-05 23:59:17 UTC

Permalink

The rate limiting iptables rule may have help you here. But not fixed
the problem ;-)

Post by Tony Graziano
I still think there is an SSL issue. Does anyone have any ideas on how
to figure this out?

That is what I think also. I have disable 5061 forwarding on the
firewall for remote clients way back around #18 (I believe) and have
enjoyed a few weeks of quite with #22 now. All local clients are not
using ssl.

--
Regards
--------------------------------------
Gerald Drouillard
Technology Architect
Drouillard & Associates, Inc.
http://www.Drouillard.biz

George Niculae

2012-11-06 00:16:19 UTC

Permalink

Is there any exception related to this startup issue in sipxsupervisor.log
file?

Post by Tony Graziano
I then tried to disable TLS and that broke nat traversal rules and failed
to start proxy, so that did not help.
I tried deleting the tmp imdb.* files and restarting presence from
sipxconfig but that did not help. The ownership of the files and sizes look
accurate (they were recreated when I restarted presence manually).
So this is SOLVED as far as the CPU level is concerned. I found a device
that has not been reconfigured (a valcom paging gateway) that is
essentially trying to register without an account, and the registrar logs
show 50-100 per minute (attempts).
I still think there is an SSL issue. Does anyone have any ideas on how to
figure this out?
On Mon, Nov 5, 2012 at 2:16 PM, Tony Graziano <

Post by Tony Graziano
I am looking at a strange issue with a system which had a drive failure.
We replaced the drive and reloaded (did not restore) the system, then
updated it to the latest update. We see the proxy staying steady at 10%
CPU, with not active calls or transactions. It is a basic system with
trunking and 12 phones, there should not be such a load.
I have sent the server its profiles. I have restarted the system. There
is no memory or swap memory issue. I have reviewed the configuration and
all of the speeddials and registrations.
The first thing I noticed is that noone was able to place outbound calls
easily, then when I started looking into it I checked user speeddials,
presense and overall configuration and hardware functionality. I still see
no issues except that the sipXproxy is taking up "enormous" CPU time. There
are 12 phones and a total of 24 subscriptions. Does anyone else have an
install similar and can verify whether they are seeing this or not?
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>

--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>
Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.**net<http://myhelp.myitdepartment.net>
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Joegen Baclor

2012-11-06 01:04:05 UTC

Permalink

Two things can cause this.

1. You have a port scanner scanning your TCP ports.
2. You have a remote connection attempting to connect as using an
unsupported version of SSL (TLSv1, SSLV2, SSLV3). I have checked
sipXportLib ssl implementation and we are configured to support all
three so this makes it unlikely.

For further reading: http://www.openssl.org/docs/ssl/SSL_CTX_new.html

Post by Tony Graziano
sipxrls:"OsSSLServerSocket SSL_accept SSL handshake error:\n SSL
error: 1 'error:00000001:lib(0):func(0):reason(1)'"
sipxrls:"OsSSLServerSocket SSL_accept SSL handshake error:\n SSL
error: 336027900 'error:140760FC:SSL
routines:SSL23_GET_CLIENT_HELLO:unknown protocol'"
sipxrls:"SipPublishContentMgr::getContent no container found for key
'sip:~~rl~C~~~id~xmpprlsclient...
(as is relates to the RLS component)
So I am wondering if someone can explain what the "unknown protocol"
means in this instance. The certificate was created in the exact way
it should have, by the system, one time at startup. I see
presenceserver says disabled but shows "running" in sipxconfig and if
I start manually via sipxproc it stays "running" (no change in
sipxconfig).
I then tried to disable TLS and that broke nat traversal rules and
failed to start proxy, so that did not help.
I tried deleting the tmp imdb.* files and restarting presence from
sipxconfig but that did not help. The ownership of the files and sizes
look accurate (they were recreated when I restarted presence manually).
So this is SOLVED as far as the CPU level is concerned. I found a
device that has not been reconfigured (a valcom paging gateway) that
is essentially trying to register without an account, and the
registrar logs show 50-100 per minute (attempts).
I still think there is an SSL issue. Does anyone have any ideas on how
to figure this out?
On Mon, Nov 5, 2012 at 2:16 PM, Tony Graziano
I am looking at a strange issue with a system which had a drive
failure. We replaced the drive and reloaded (did not restore) the
system, then updated it to the latest update. We see the proxy
staying steady at 10% CPU, with not active calls or transactions.
It is a basic system with trunking and 12 phones, there should not
be such a load.
I have sent the server its profiles. I have restarted the system.
There is no memory or swap memory issue. I have reviewed the
configuration and all of the speeddials and registrations.
The first thing I noticed is that noone was able to place outbound
calls easily, then when I started looking into it I checked user
speeddials, presense and overall configuration and hardware
functionality. I still see no issues except that the sipXproxy is
taking up "enormous" CPU time. There are 12 phones and a total of
24 subscriptions. Does anyone else have an install similar and can
verify whether they are seeing this or not?
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about
sipX-CoLab 2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about
sipX-CoLab 2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>
Telephone: 434.984.8426
Helpdesk Customers: http://myhelp.myitdepartment.net
<http://myhelp.myitdepartment.net>
Blog: http://blog.myitdepartment.net
_______________________________________________
sipx-users mailing list
List Archive: http://list.sipfoundry.org/archive/sipx-users/

Josh Patten

2012-11-06 01:25:14 UTC

Permalink

The SSL error in the RLS logs is actually benign, I'm not so sure about the
no container found error.

On Mon, Nov 5, 2012 at 3:50 PM, Tony Graziano

Post by Tony Graziano
1 'error:00000001:lib(0):func(0):reason(1)'"
336027900 'error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
protocol'"
sipxrls:"SipPublishContentMgr::getContent no container found for key
'sip:~~rl~C~~~id~xmpprlsclient...
(as is relates to the RLS component)
So I am wondering if someone can explain what the "unknown protocol" means
in this instance. The certificate was created in the exact way it should
have, by the system, one time at startup. I see presenceserver says
disabled but shows "running" in sipxconfig and if I start manually via
sipxproc it stays "running" (no change in sipxconfig).
I then tried to disable TLS and that broke nat traversal rules and failed
to start proxy, so that did not help.
I tried deleting the tmp imdb.* files and restarting presence from
sipxconfig but that did not help. The ownership of the files and sizes look
accurate (they were recreated when I restarted presence manually).
So this is SOLVED as far as the CPU level is concerned. I found a device
that has not been reconfigured (a valcom paging gateway) that is
essentially trying to register without an account, and the registrar logs
show 50-100 per minute (attempts).
I still think there is an SSL issue. Does anyone have any ideas on how to
figure this out?
On Mon, Nov 5, 2012 at 2:16 PM, Tony Graziano <

Post by Tony Graziano
I am looking at a strange issue with a system which had a drive failure.
We replaced the drive and reloaded (did not restore) the system, then
updated it to the latest update. We see the proxy staying steady at 10%
CPU, with not active calls or transactions. It is a basic system with
trunking and 12 phones, there should not be such a load.
I have sent the server its profiles. I have restarted the system. There
is no memory or swap memory issue. I have reviewed the configuration and
all of the speeddials and registrations.
The first thing I noticed is that noone was able to place outbound calls
easily, then when I started looking into it I checked user speeddials,
presense and overall configuration and hardware functionality. I still see
no issues except that the sipXproxy is taking up "enormous" CPU time. There
are 12 phones and a total of 24 subscriptions. Does anyone else have an
install similar and can verify whether they are seeing this or not?
--
~~~~~~~~~~~~~~~~~~
Tony Graziano, Manager
Telephone: 434.984.8430
Fax: 434.465.6833
~~~~~~~~~~~~~~~~~~
http://www.linkedin.com/pub/tony-graziano/14/4a6/7a4
Ask about our Internet Fax services!
~~~~~~~~~~~~~~~~~~
Using or developing for sipXecs from SIPFoundry? Ask me about sipX-CoLab
2013!
<http://sipxcolab2013.eventbrite.com/?discount=tony2013>

--
Josh Patten
eZuce
Solutions Architect
O.978-296-1005 X2050
M.979-574-5699
http://www.ezuce.com