Project

General

Profile

Bug #2579

5.6.2 crashes on FreeBSD

Added by Renato Botelho over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Category:
freebsd
Target version:
Start date:
Due date:
Estimated time:
Affected version:
5.6.2
Resolution:
Fixed

Description

After upgrade strongswan to 5.6.2 on FreeBSD ports tree and on pfSense some users started to report a crash. Fortunately I got access to a machine of a colleague where crash is reproducible.

Before this upgrade we were using 5.6.0, so looks like something broke between these 2 releases.

Here is ipsec.conf

config setup
    uniqueids = yes

conn con4
    fragmentation = yes
    keyexchange = ikev2
    reauth = yes
    forceencaps = no
    mobike = no

    rekey = yes
    installpolicy = yes
    type = tunnel
    dpdaction = restart
    dpddelay = 10s
    dpdtimeout = 60s
    auto = route
    left = $LEFT_IP
    right = $RIGHT_ADDRESS
    leftid = dns:$DYNDNS_CLIENT
    ikelifetime = 28800s
    lifetime = 3600s
    esp = aes256gcm128-sha1-modp2048,aes256gcm96-sha1-modp2048,aes256gcm64-sha1-modp2048!
    leftauth = pubkey
    rightauth = pubkey
    leftcert=/var/etc/ipsec/ipsec.d/certs/cert-4.crt
    leftsendcert=always
    rightca="/C=US/ST=Texas/L=Austin/O=pfmechanics/emailAddress=ca@pfmechanics.com/CN=IPsecCA/" 
    rightid = $RIGHT_IP_ID
    rightsubnet = 172.27.0.0/16
    leftsubnet = 172.21.25.0/24

Here is strongswan.conf:

starter {
    load_warning = no
    config_file = /var/etc/ipsec/ipsec.conf
}

charon {
# number of worker threads in charon
    threads = 16
    ikesa_table_size = 32
    ikesa_table_segments = 4
    init_limit_half_open = 1000
    install_routes = no
    load_modular = yes
    ignore_acquire_ts = yes

    accept_unencrypted_mainmode_messages = yes
    cisco_unity = no

    syslog {
        identifier = charon
        # log everything under daemon since it ends up in the same place regardless with our syslog.conf
        daemon {
            ike_name = yes
            dmn = 1
            mgr = 1
            ike = 1
            chd = 4
            job = 1
            cfg = 1
            knl = 1
            net = 1
            asn = 1
            enc = 1
            imc = 1
            imv = 1
            pts = 1
            tls = 1
            esp = 1
            lib = 1

        }
        # disable logging under auth so logs aren't duplicated
        auth {
            default = -1
        }
    }

    plugins {
        # Load defaults
        include /var/etc/ipsec/strongswan.d/charon/*.conf

        stroke {
            secrets_file = /var/etc/ipsec/ipsec.secrets
        }

        unity {
            load = no
        }
        attr {
            subnet = 0.0.0.0/0
            split-include = 0.0.0.0/0
            28673 = 1
        }
        xauth-generic {
            script = /etc/inc/ipsec.auth-user.php
            authcfg = Local Database
        }

    }
}

And here is part of the logfile:

Mar  5 14:04:48 pfSense charon: 05[IKE] <con4|1> received cert request for "C=US, ST=Texas, L=Austin, O=pfmechanics, E=ca@pfmechanics.com, CN=IPsecCA" 
Mar  5 14:04:48 pfSense charon: 05[IKE] <con4|1> sending cert request for "C=US, ST=Texas, L=Austin, O=pfmechanics, E=ca@pfmechanics.com, CN=IPsecCA" 
Mar  5 14:04:48 pfSense charon: 05[IKE] <con4|1> authentication of '$DYNDNS_CLIENT' (myself) with RSA_EMSA_PKCS1_SHA2_256 successful
Mar  5 14:04:48 pfSense charon: 05[DMN] <con4|1> thread 5 received 11
Mar  5 14:04:48 pfSense charon: 05[LIB] <con4|1>  dumping 2 stack frame addresses:
Mar  5 14:04:48 pfSense charon: 05[LIB] <con4|1>   /lib/libthr.so.3 @ 0x800f52000 (pthread_sigmask+0x536) [0x800f608f6]
Mar  5 14:04:48 pfSense charon: 05[LIB] <con4|1>     ->
Mar  5 14:04:48 pfSense charon: 05[LIB] <con4|1>   /lib/libthr.so.3 @ 0x800f52000 (pthread_getspecific+0xe2f) [0x800f5fe9f]
Mar  5 14:04:48 pfSense charon: 05[LIB] <con4|1>     ->
Mar  5 14:04:48 pfSense charon: 05[DMN] <con4|1> killing ourself, received critical signal
Mar  5 14:04:59 pfSense ipsec_starter[1116]: charon has died -- restart scheduled (5sec)
Mar  5 14:05:04 pfSense charon: 00[DMN] Starting IKE charon daemon (strongSwan 5.6.2, FreeBSD 11.1-RELEASE-p6, amd64)

And part of the backtrace, but without debug symbols yet. I'll work to build a package containing it and then I can update info here:

(lldb) bt
* thread #1, name = 'charon', stop reason = signal SIGABRT
  * frame #0: 0x000000080123f76a libc.so.7`__vfwprintf(fp=<unavailable>, locale=0x0000000800670c00, fmt0=0x000000080066f400, ap=0x000000080215ef00) at vfwprintf.c:712
    frame #1: 0x000000080123f6a9 libc.so.7`__vfwprintf [inlined] io_pad(howmany=19134260) at printfcommon.h:122
    frame #2: 0x000000080123f6a6 libc.so.7`__vfwprintf(fp=<unavailable>, locale=0x0000000800675c00, fmt0=0x0000000800672400, ap=<unavailable>) at vfwprintf.c:1090
    frame #3: 0x00000000004022e1 charon`___lldb_unnamed_symbol5$$charon + 145
    frame #4: 0x0000000800f608f6 libthr.so.3`_thr_suspend_all_lock [inlined] atomic_cmpset_int(expect=2147483648, src=<unavailable>) at atomic.h:220
    frame #5: 0x0000000800f608f5 libthr.so.3`_thr_suspend_all_lock [inlined] _thr_umutex_trylock2(id=<unavailable>) at thr_umtx.h:107
    frame #6: 0x0000000800f608c6 libthr.so.3`_thr_suspend_all_lock [inlined] _thr_umutex_lock at thr_umtx.h:123
    frame #7: 0x0000000800f608c6 libthr.so.3`_thr_suspend_all_lock(curthread=0xffffffdfff68a9a1) at thr_suspend_np.c:86
    frame #8: 0x0000000800f5fe9f libthr.so.3`_pthread_getaffinity_np [inlined] _thr_umutex_unlock2 at thr_umtx.h:157
    frame #9: 0x0000000800f5fe80 libthr.so.3`_pthread_getaffinity_np [inlined] _thr_umutex_unlock at thr_umtx.h:183
    frame #10: 0x0000000800f5fe80 libthr.so.3`_pthread_getaffinity_np(td=<unavailable>, cpusetsize=18446744073709551615, cpusetp=0x000000080215ef00) at thr_affinity.c:84
    frame #11: 0x00007ffffffff003

Let me know if you need more information.

Associated revisions

Revision a48f3d89 (diff)
Added by Tobias Brunner over 2 years ago

ikev2: Use correct type to check for selected signature scheme

The previous code was obviously incorrect and caused strange side effects
depending on the compiler and its optimization flags (infinite looping seen
with GCC 4.8.4, segfault when destroying the private key in build() seen
with clang 4.0.0 on FreeBSD).

Fixes #2579.

History

#1 Updated by Tobias Brunner over 2 years ago

  • Category set to freebsd
  • Status changed from New to Feedback

These backtraces are not helpful as none of the functions are located in any of strongSwan's libraries. Try attaching GDB to charon (e.g. with ipsec start --attach-gdb) and get proper backtraces (e.g. with thread apply all bt full) after the crash (or analyze any core dumps created by the crash).

#2 Updated by Renato Botelho over 2 years ago

I've started it with

--attach-gdb
and I got this with strongswan without debug symbols on:

Thread 6 (LWP 102124 of process 62221):
#0  0x0000000800ae0985 in ?? () from /usr/local/lib/ipsec/libcharon.so.0
No symbol table info available.
#1  0x0000000800ae76a8 in ?? () from /usr/local/lib/ipsec/libcharon.so.0
No symbol table info available.
#2  0x0000000800adc189 in ?? () from /usr/local/lib/ipsec/libcharon.so.0
No symbol table info available.
#3  0x0000000800adad70 in ?? () from /usr/local/lib/ipsec/libcharon.so.0
No symbol table info available.
#4  0x0000000800acf139 in ?? () from /usr/local/lib/ipsec/libcharon.so.0
No symbol table info available.
#5  0x0000000800ac66b1 in ?? () from /usr/local/lib/ipsec/libcharon.so.0
No symbol table info available.
#6  0x0000000800863b87 in ?? () from /usr/local/lib/ipsec/libstrongswan.so.0
No symbol table info available.
#7  0x0000000800877060 in ?? () from /usr/local/lib/ipsec/libstrongswan.so.0
No symbol table info available.
#8  0x0000000800f5ac05 in ?? () from /lib/libthr.so.3
No symbol table info available.
#9  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdf7fa000

After build strongswan with debug symbols it stopped crashing.

#3 Updated by Renato Botelho over 2 years ago

FYI, I've built 5.6.1 package to attempt to identify first broken release and it crashed the same way. So the problem was introduced in 5.6.1

#4 Updated by Tobias Brunner over 2 years ago

I've started it with [...] and I got this with strongswan without debug symbols on:

[...]

After build strongswan with debug symbols it stopped crashing.

Thanks, I noticed that too (got access to a FreeBSD machine). It's not related to the symbols, but -O2. If you set DEBUG_FLAGS to -g -O2 in the port's Makefile it crashes too and the location seems to be here: source:src/libcharon/sa/ikev2/authenticators/pubkey_authenticator.c#L404 No idea why that would crash (and only with -O2). Perhaps a compiler bug.

#5 Updated by Renato Botelho over 2 years ago

Tobias Brunner wrote:

I've started it with [...] and I got this with strongswan without debug symbols on:

[...]

After build strongswan with debug symbols it stopped crashing.

Thanks, I noticed that too (got access to a FreeBSD machine). It's not related to the symbols, but -O2. If you set DEBUG_FLAGS to -g -O2 in the port's Makefile it crashes too and the location seems to be here: source:src/libcharon/sa/ikev2/authenticators/pubkey_authenticator.c#L404 No idea why that would crash (and only with -O2). Perhaps a compiler bug.

I'm building it with gcc right now and I will let you know the results ASAP

#6 Updated by Tobias Brunner over 2 years ago

I think I found the bug. The patch in the 2579-pubkey-scheme branch should fix it (or put the attached patch in files/ of the FreeBSD port).

#7 Updated by Francois ten Krooden over 2 years ago

I am busy running the patch with through the strongSwan tests that we have working on FreeBSD.
It is looking good thus far.

#8 Updated by Renato Botelho over 2 years ago

This seems to fix at least the use case I got that was crashing. I'll ask for pfSense users to test it as well but so far so good.

Thanks

#9 Updated by Renato Botelho over 2 years ago

More pfSense users confirmed the fix

#10 Updated by Tobias Brunner over 2 years ago

  • Tracker changed from Issue to Bug
  • Status changed from Feedback to Closed
  • Assignee set to Tobias Brunner
  • Target version set to 5.6.3
  • Resolution set to Fixed

Merged to master.

Also available in: Atom PDF