Project

General

Profile

Issue #1484

Running swanctl or ipsec commands hang with no output

Added by Danny Kulchinsky about 6 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
configuration
Affected version:
5.3.5
Resolution:
No change required

Description

We are trying to run basic commands such as 'swanctl -S' or 'ipsec statusall' and the command just hangs with no output. We waited up to 10 minutes and still no output.

Using strace, we see that it hangs on connecting to charon.vici:
connect(8, {sa_family=AF_LOCAL, sun_path="/var/run/charon.vici"}, 22

Full strace log attached.

It seems that charon is working, but we are unable to run any command against it to see what's going on.

Just before this happened we did send a SIGHUP signal to charon since we wanted to increase the number of sockets to the Radius servers.

strace_swanctl_-S.log (70 KB) strace_swanctl_-S.log "strace swanctl -S" log Danny Kulchinsky, 26.05.2016 17:42

History

#1 Updated by Danny Kulchinsky about 6 years ago

Danny Kulchinsky wrote:

We are trying to run basic commands such as 'swanctl -S' or 'ipsec statusall' and the command just hangs with no output. We waited up to 10 minutes and still no output.

Using strace, we see that it hangs on connecting to charon.vici:
connect(8, {sa_family=AF_LOCAL, sun_path="/var/run/charon.vici"}, 22

Full strace log attached.

It seems that charon is working, but we are unable to run any command against it to see what's going on.

Just before this happened we did send a SIGHUP signal to charon since we wanted to increase the number of sockets to the Radius servers.

Could this be related to Bug #1185 ?

We removed this server from DNS record so there are no new tunnel setup requests, however CPU usage of charon seems quite high (~190% on an 8-core machine). there's almost no traffic on the external interface but I'm able to ping some tunnel Virtual IPs.

# netstat -na | grep charon
unix  2      [ ACC ]     STREAM     LISTENING     19277  /var/run/charon.ctl
unix  2      [ ACC ]     STREAM     LISTENING     19279  /var/run/charon.vici
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  3      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.ctl
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.vici
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.vici
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.vici
unix  2      [ ]         STREAM     CONNECTING    0      /var/run/charon.vici

#2 Updated by Danny Kulchinsky about 6 years ago

Problem fixed !

It was all due to insufficient "Max open files" limit for the user used to run charon.

I stumbled upon the following log which gave me a direction:
2016-05-26 18:37:51.879 28[LIB] <XXXXXXXXX|2743058> creating stdout pipe failed: Too many open files

I updated the settings in /etc/security/limits.conf and set the runtime limits in /proc/<charon pid>/limits file and it works :)

# swanctl -S
uptime: 3 days, since May 23 12:34:29 2016
worker threads: 64 total, 59 idle, working: 4/0/1/0
job queues: 0/0/0/0
jobs scheduled: 56198
IKE_SAs: 522 total, 0 half-open
mallinfo: sbrk 10547200, mmap 2629632, used 1565728, free 8981472

#3 Updated by Tobias Brunner about 6 years ago

  • Category set to configuration
  • Status changed from New to Closed
  • Priority changed from High to Normal
  • Resolution set to No change required

Also available in: Atom PDF