strongSwan

It is pfSense, it seems at least a couple of pfsense users are running into this same issue. Thanks for the tips, ill probably install a fresh FreeBSD/strongswan and see if i can get it to act like the pfSense version.

Didn't realize someone had opened a ticket here. I've confirmed the same memory leak occurs on stock FreeBSD using the strongswan available via 'pkg install'.

The port we build strongswan from, along with changes, is here:
https://github.com/pfsense/FreeBSD-ports/tree/devel/security/strongswan
but not only specific to anything we're doing there. There might be something there that makes some situations worse, but it is replicable on stock FreeBSD and stock strongSwan.

Bringing up 400 connections like the following, changing only left/rightsubnet on subsequent ones:

conn con1
        fragmentation = yes
        keyexchange = ikev1
        reauth = yes
        forceencaps = no
        mobike = no
        rekey = yes
        installpolicy = yes
        type = tunnel
        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = route
        left = 10.2.44.114
        right = 10.2.44.168
        leftid = 10.2.44.114
        ikelifetime = 7200s
        lifetime = 3600s
        ike = aes256-sha1-modp1024!
        leftauth = psk
        rightauth = psk
        rightid = 10.2.44.168
        aggressive = no
        rightsubnet = 100.66.1.1
        leftsubnet = 100.65.1.1

and letting them rekey, it leaks around 4 MB RAM per hour.

Any tips on running charon in valgrind? I'm failing with attempts along the lines of the following.

 # valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all --track-origins=yes ipsec start
==11175== Memcheck, a memory error detector
==11175== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==11175== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==11175== Command: /usr/local/sbin/ipsec start
==11175==

valgrind: m_syswrap/syswrap-freebsd.c:3302 (vgSysWrap_freebsd_sys_fcntl_before): Assertion 'Unimplemented functionality' failed.
valgrind: valgrind

host stacktrace:
==11175==    at 0x38052372: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x38052494: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x38052616: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x380B6B00: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x380A2EE5: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x3809F81C: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x380A0C66: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)
==11175==    by 0x380ADEDC: ??? (in /usr/local/lib/valgrind/memcheck-amd64-freebsd)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
==11175==    at 0x518D63A: _fcntl (in /lib/libc.so.7)
==11175==    by 0x50A9ED1: fcntl (in /lib/libc.so.7)
==11175==    by 0x40D0B0: ??? (in /bin/sh)
==11175==    by 0x41204C: ??? (in /bin/sh)
==11175==    by 0x4105A1: ??? (in /bin/sh)
==11175==    by 0x402EDE: ??? (in /bin/sh)
==11175==    by 0x4020FFF: ???

or similar to this post: https://lists.strongswan.org/pipermail/dev/2015-July/001415.html

I've confirmed the same memory leak occurs on stock FreeBSD using the strongswan available via 'pkg install'.

OK, thanks for the confirmation.

The port we build strongswan from, along with changes, is here:
https://github.com/pfsense/FreeBSD-ports/tree/devel/security/strongswan

Thanks, good to know.

Bringing up 400 connections like the following, changing only left/rightsubnet on subsequent ones:
[...]
and letting them rekey, it leaks around 4 MB RAM per hour.

Is it specific to rekeying (of the IKE_SA or the CHILD_SA)? Does the IKE version make a difference (the OP uses IKEv2, so I guess not really).

Any tips on running charon in valgrind? I'm failing with attempts along the lines of the following.

No idea what this particular problem is about, but even when running charon in valgrind directly it apparently doesn't work right away. We use sigwait(), which FreeBSD's valgrind does not support (unhandled syscall: 429, followed by [DMN] error -1 while waiting for a signal), so this requires patching either valgrind or charon.

Hi,

I'm working with Chris on this issue. All testing is occurring on freebsd. Once we find the issue, we'll retest on pfsense.

The leak shows up (but not as badly) with --with-printf-hooks=vstr as well.

I need to go back and re-test with --with-printf-hooks=builtin

I strongly suspect a leak in at least the TS printf-hook ("%#R").

p.s. I think this "(which I think are still not publicly available, so I can't comment on that)" was a bit unfair.

By default, strongswan built on FreeBSD ports do not force --with-printf-hooks, leaving it as 'auto' and then end up using glibc version.

A test to compare vstr versus builtin would be a good experiment

This was fixed in the latest release of pfSense 2.2.5.. https://redmine.pfsense.org/issues/5149

Adam Piasecki wrote:

This was fixed in the latest release of pfSense 2.2.5.. https://redmine.pfsense.org/issues/5149

More worked around than fixed, but the worst of the leaks are no longer a problem at least. with-printf-hooks=vstr works around the bulk of the memory leaking, then we also switched from SMP to Vici for status output because vstr broke SMP output (and we were planning on switching to vici anyway in the future).

We also updated the FreeBSD port to default to with-printf-hooks=builtin. builtin was preferred as default there because it doesn't have the vstr dependency. builtin appears to leak significantly less than glibc, the former default, but still leaks noticeably in larger scale setups (multi-hundred connections) over time.
https://www.freshports.org/security/strongswan

Was the leak fixed? Is there a specific commit to point out?

There is no leak.