0

I am facing a very strange issue with Kamailio, our Azure VM is D2 configuration with 2 cores and 7 GB of RAM, Ubuntu 14.04 LTS, through-out the day we see high CPU usage from Kamailio, and intermittently we face issue that Kamailio stops accepting new registrations. When I check the location table, or via kamctl ul show, it shows only around 900+ users. But still Kamailio could not take new registrations with that small number of user.I believe Kamailio can handle 350000 users per core. Also tcp_connections are at tcp_max_connection = 60000, I receive following error:

Feb 4 01:02:21 urcmcs /usr/local/sbin/kamailio[1956]: ERROR: [tcp_read.c:299]: tcp_read_data(): error reading: Connection timed out (110) Feb 4 01:02:21 urcmcs /usr/local/sbin/kamailio[1956]: ERROR: [tcp_read.c:1326]: tcp_read_req(): ERROR: tcp_read_req: error reading Feb 4 01:02:42 urcmcs /usr/local/sbin/kamailio[1969]: ERROR: [tcp_read.c:299]: tcp_read_data(): error reading: Connection timed out (110) Feb 4 01:02:42 urcmcs /usr/local/sbin/kamailio[1969]: ERROR: [tcp_read.c:1326]: tcp_read_req(): ERROR: tcp_read_req: error reading Feb 4 01:02:54 urcmcs /usr/local/sbin/kamailio[1959]: ERROR: [tcp_read.c:299]: tcp_read_data(): error reading: Connection timed out (110) Feb 4 01:02:54 urcmcs /usr/local/sbin/kamailio[1959]: ERROR: [tcp_read.c:1326]: tcp_read_req(): ERROR: tcp_read_req: error reading

Searched everywhere but cannot find a solution...

Sharon Nathaniel
  • 1,467
  • 20
  • 28

2 Answers2

0

The errors suggest that the tcp connections got broken and reading timed out.

The way to troubleshoot when Kamailio seems not to be handling the traffic:

  • be sure you don't have a stateful firewall that does packet limiting (e.g. selinux)
  • send a packet from the same server using sipsak or sipp and see if there is a response -- this should rule out the network problems
  • use gdb to attach to kamailio processes and see what they are doing: gdb /path/to/kamailio PID. You can use 'kamctl ps' to see the PIDs for kamailio processes and their roles.
  • if you do 'top' and some kamailio processes are using a lot of cpu, investigate them first with the gdb as above
miconda
  • 1,754
  • 11
  • 14
  • Hi, thanks for the help; though I had already tried these before. I think there are more than 60000 connections in parallel that might be causing issues .. not sure though. – Sharon Nathaniel Feb 11 '16 at 06:42
0

I finally found the issue, Ubuntu kernel 3.19.0-28-generic, is known to have some problems with network performance. This was causing very high usage and connection problems. Updating the kernel to the newest solved the problem.

Sharon Nathaniel
  • 1,467
  • 20
  • 28