Project

General

Profile

Issue #1518

Why does the performance of strongswan decreases dramatically when the number of connection is huge. pls help me!

Added by Bin Liu over 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Affected version:
5.4.0
Resolution:
No feedback

Description

I test the performance of the ike negotiation by strongswan when the number of connection is large. And find the result is strange. Strongswan are both used by client and server. The client start many connections while the server (security gateway) has one with auto = add.
My OS is linux and board has 24 logical cores.
1) if client starts 1 conns, it takes 160ms more or less to establish all tunnels.
2) if client starts 16 conns, it takes only 370ms to establish all tunnels. because of the multi-threads it is very fast i think.
3) if client starts 1000 conns, it takes 25s more or less to establish all tunnels. nearly 400 tunnels up every 10 sec.
4) Howerver, if the client starts 5000 conns, it takes 50 minutes or more to establish all tunnels. It seems the performance decreases dramatically unbelievable. I see that from the beginning 40 tunnels up every 10 sec and soon 20 tunnels up every 10 sec. it becomes more and more slow as time passed by.

i can not understand why.
1) does the client send too slowly or the server process too slowly when the number of conn is large? what may happen?(i see the server does not take much cpu usage and is in "S" state while client always takes 100%)
2) how can i improve the performance of ike nego when the number of conn is large? i have 24 cores and only one is used by charon. it is a waste.

please help me. my hardware (24cores and NUMA) on machine is very strong and not a bottleneck i think.

the config file and packets captured by wireshark in 1000 and 5000 connections is provided in attachment.

ipsec_client.conf (870 KB) ipsec_client.conf Bin Liu, 16.06.2016 08:37
ipsec_server.conf (460 Bytes) ipsec_server.conf Bin Liu, 16.06.2016 08:37
test_5000 conns .rar (4.64 MB) test_5000 conns .rar Bin Liu, 16.06.2016 08:39
test_1000 conns.rar (1.25 MB) test_1000 conns.rar Bin Liu, 16.06.2016 08:39

Related issues

Related to Issue #1519: SA establish failed using load-tester as client in PSK scenario, always reporting MAC mismatched on server.Closed

History

#1 Updated by Tobias Brunner over 9 years ago

  • Status changed from New to Feedback

1) does the client send too slowly or the server process too slowly when the number of conn is large? what may happen?(i see the server does not take much cpu usage and is in "S" state while client always takes 100%)

Check the log (in particular the timestamps for individual threads to spot delays and if there are retransmits).

2) how can i improve the performance of ike nego when the number of conn is large? i have 24 cores and only one is used by charon. it is a waste.

Did you have a look at IkeSaTable and JobPriority? Also consider using load-tester instead of thousands of configs in ipsec.conf (which is not particularly efficient).

#2 Updated by Bin Liu over 9 years ago

thanks!

How to start the load-tester?
i tried but error returned. is there a description about how.

[root@localhost ln]# ipsec load-tester
Usage:
/usr/libexec/ipsec/load-tester initiate <count> [<delay in ms>]
[root@localhost ln]#
[root@localhost ln]# ipsec load-tester initiate 10000
connecting to /var/run/charon.ldt failed: No such file or directory

#3 Updated by Tobias Brunner over 9 years ago

You need to load and enable the load-tester plugin.

#4 Updated by Bin Liu over 9 years ago

OK. i got it.
but the ike nego failed.
logs:

07[ENC] generating IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ]
07[NET] sending packet: from 10.10.38.93[500] to 10.10.1.1[500] (300 bytes)
28[NET] received packet: from 10.10.38.93[500] to 10.10.1.1[500] (300 bytes)
28[ENC] parsed IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ]
28[IKE] received cert request for "CN=srv, OU=load-test, O=strongSwan" 
28[CFG] looking for peer configs matching 10.10.1.1[srv.strongswan.org]...10.10.38.93[c9822-r1.strongswan.org]
28[CFG] selected peer config 'load-test'
28[IKE] authentication of 'c9822-r1.strongswan.org' with pre-shared key successful
28[IKE] authentication of 'srv.strongswan.org' (myself) with pre-shared key
28[IKE] IKE_SA load-test[19745] established between 10.10.1.1[srv.strongswan.org]...10.10.38.93[c9822-r1.strongswan.org]
28[IKE] peer requested virtual IP %any
28[IKE] no virtual IP found for %any requested by 'c9822-r1.strongswan.org'
28[IKE] no virtual IP found, sending INTERNAL_ADDRESS_FAILURE
28[IKE] configuration payload negotiation failed, no CHILD_SA built
28[IKE] failed to establish CHILD_SA, keeping IKE_SA
28[ENC] generating IKE_AUTH response 1 [ IDr AUTH N(INT_ADDR_FAIL) ]
28[NET] sending packet: from 10.10.1.1[500] to 10.10.38.93[500] (124 bytes)
27[NET] received packet: from 10.10.1.1[500] to 10.10.38.93[500] (124 bytes)
27[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(INT_ADDR_FAIL) ]
27[IKE] authentication of 'srv.strongswan.org' with pre-shared key successful
27[IKE] IKE_SA load-test[19375] established between 10.10.38.93[c9822-r1.strongswan.org]...10.10.1.1[srv.strongswan.org]
27[IKE] received INTERNAL_ADDRESS_FAILURE notify, no CHILD_SA built
27[IKE] failed to establish CHILD_SA, keeping IKE_SA

the server side:

config setup

conn tt      
        reauth=no
        rekey=no
        dpdaction = none
        #replay_window = 0 
        keyingtries=%forever
        keyexchange=ikev2
        ike=modp2048
        left=10.10.1.1
        leftsubnet=0.0.0.0/0
        #leftikeport=500
        #leftsourceip=%config
        leftauth=psk
        right=%any
        rightauth=psk
        #rightsubnet=1.1.1.0/24
        rightsourceip=1.0.0.0/8
        esp=aes-sha1!
        mobike=no
        auto=add

the client:

load-tester {
                        # enable the plugin
            load = yes
            enable = yes
            # 10000 connections, ten in parallel
            initiators = 10
            iterations = 1000
            # use a delay of 100ms, overall time is: iterations * delay = 100s
            delay = 50
                        addrs {
                                # install ~32K addresses on eth0
                                ens255f0 = 10.10.0.0/18
                        }
                        # install all addresses with a /16 prefix
            addrs_prefix = 8
            # address of the gateway (releases before 5.0.2 used the "remote" keyword!)
            responder = 10.10.1.1
            # IKE-proposal to use
            ike = modp2048
            esp = aes128-sha1
            # use faster PSK authentication instead of 1024bit RSA
            initiator_auth = psk
            responder_auth = psk
            # request a virtual IP using configuration payloads
            request_virtual_ip = yes
            # disable IKE_SA rekeying (default)
            ike_rekey = 0
            # enable CHILD_SA every 60s
            child_rekey = 0
                        initiator_tsr = 192.0.0.0/8
            # do not delete the IKE_SA after it has been established (default)
            delete_after_established = no
            # do not shut down the daemon if all IKE_SAs established
            shutdown_when_complete = no

                }

why the client can not get the virtue ip? i configured rightsourceip=1.0.0.0/8.

#5 Updated by Tobias Brunner over 9 years ago

28[CFG] selected peer config 'load-test'

Looks like you enabled the load-tester plugin on the server. Either disable it or set load-tester.pool on the server to a named pool (could be 1.0.0.0/8 if you still load the config in ipsec.conf) that provides the IP addresses.

#6 Updated by Bin Liu over 9 years ago

thanks very much. But why the server can't find the peer.

client side log:

Starting strongSwan 5.3.0 IPsec [starter]...
Loading config setup
cachecrls=no
strictcrlpolicy=ifuri
uniqueids=keep
found netkey IPsec stack
Attempting to start charon...
00[DMN] Starting IKE charon daemon (strongSwan 5.3.0, Linux 3.10.0-327.el7.x86_64, x86_64)
00[CFG] loaded load-tester address pool 10.10.0.0/18 on ens255f0
00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
00[CFG] loading crls from '/etc/ipsec.d/crls'
00[CFG] loading secrets from '/etc/ipsec.secrets'
00[CFG] loaded IKE secret for %any
00[LIB] loaded plugins: charon aes des rc2 sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem fips-prf gmp xcbc cmac hmac attr load-tester kernel-netlink resolve socket-default stroke updown xauth-generic
00[JOB] spawning 32 worker threads
02[CFG] assigning new lease to 'ext-2'
02[CFG] installed load-tester IP 10.10.0.1 on ens255f0
02[IKE] initiating IKE_SA load-test1 to 10.11.1.1
charon (29891) started after 460 ms
02[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) ]
02[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
08[IKE] retransmit 1 of request with message ID 0
08[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
10[IKE] retransmit 2 of request with message ID 0
10[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
11[NET] received packet: from 10.11.1.1500 to 10.10.0.1500 (456 bytes)
11[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) N(MULT_AUTH) ]
11[IKE] sending cert request for "CN=srv, OU=load-test, O=strongSwan"
11[IKE] authentication of 'c2-r1.strongswan.org' (myself) with pre-shared key
11[IKE] establishing CHILD_SA load-test
11[ENC] generating IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ]
11[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (300 bytes)
12[NET] received packet: from 10.11.1.1500 to 10.10.0.1500 (76 bytes)
12[ENC] parsed IKE_AUTH response 1 [ N(AUTH_FAILED) ]
12[IKE] received AUTHENTICATION_FAILED notify error
12[CFG] lease 10.10.0.1 by 'ext-2' went offline

server side log:

Stopping strongSwan IPsec failed: starter is not running
Starting strongSwan 5.4.0 IPsec [starter]...
Loading config setup
Loading conn 'tt'
auto=add
dpdaction=none
esp=aes-sha1!
ike=aes128-sha1-modp2048
keyexchange=ikev2
keyingtries=%forever
left=10.11.1.1
leftauth=psk
leftsubnet=0.0.0.0/0
mobike=no
reauth=no
rekey=no
right=%any
rightauth=psk
rightsourceip=1.0.0.0/8
found netkey IPsec stack
Attempting to start charon...
00[DMN] Starting IKE charon daemon (strongSwan 5.4.0, Linux 3.10.0-327.el7.x86_64, x86_64)
00[CFG] loading ca certificates from '/etc/ipsec/ipsec.d/cacerts'
00[CFG] loading aa certificates from '/etc/ipsec/ipsec.d/aacerts'
00[CFG] loading ocsp signer certificates from '/etc/ipsec/ipsec.d/ocspcerts'
00[CFG] loading attribute certificates from '/etc/ipsec/ipsec.d/acerts'
00[CFG] loading crls from '/etc/ipsec/ipsec.d/crls'
00[CFG] loading secrets from '/etc/ipsec/ipsec.secrets'
00[CFG] loaded IKE secret for %any
00[LIB] loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem fips-prf gmp xcbc cmac hmac attr kernel-netlink resolve socket-default stroke vici updown error-notify
00[JOB] spawning 16 worker threads
charon (29210) started after 20 ms
08[CFG] received stroke: add connection 'tt'
08[CFG] adding virtual IP address pool 1.0.0.0/8
08[CFG] added configuration 'tt'
12[NET] received packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
12[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) ]
12[IKE] 10.10.0.1 is initiating an IKE_SA
12[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) N(MULT_AUTH) ]
12[NET] sending packet: from 10.11.1.1500 to 10.10.0.1500 (456 bytes)
13[NET] received packet: from 10.10.0.1500 to 10.11.1.1500 (300 bytes)
13[ENC] parsed IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ]
13[IKE] received 1 cert requests for an unknown ca
13[CFG] looking for peer configs matching 10.11.1.1[srv.strongswan.org]...10.10.0.1[c2-r1.strongswan.org]
13[CFG] no matching peer config found
13[ENC] generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]
13[NET] sending packet: from 10.11.1.1500 to 10.10.0.1500 (76 bytes)

server side config:

auto=add
dpdaction=none
esp=aes-sha1!
ike=aes128-sha1-modp2048
keyexchange=ikev2
keyingtries=%forever
left=10.11.1.1
leftauth=psk
leftsubnet=0.0.0.0/0
mobike=no
reauth=no
rekey=no
right=%any
rightauth=psk
rightsourceip=1.0.0.0/8

since i use psk, the id may be the ip. but there is a id named srv.strongswan.org on server and c2-r1.strongswan.org on client. I didn't config any fqdn and did not have a cert....

what's wrong?

#7 Updated by Bin Liu over 9 years ago

i configured the id on client: responder_id = 10.11.1.1

and the shared keys on both side are as follows:

client:
[root@localhost charon]# cat /etc/ipsec.secrets
: PSK 0x70736b

server:
[root@localhost ln]# cat /etc/ipsec.secrets
: PSK 0x70736b

They are the same.

But the server said that:
07[IKE] received 1 cert requests for an unknown ca
07[CFG] looking for peer configs matching 10.11.1.1[10.11.1.1]...10.10.0.1[c2-r1.strongswan.org]
07[CFG] selected peer config 'tt'
07[IKE] tried 1 shared key for '10.11.1.1' - 'c2-r1.strongswan.org', but MAC mismatched
07[ENC] generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]
07[NET] sending packet: from 10.11.1.1500 to 10.10.0.1500 (76 bytes)

what does the MAC mismatched, since the preshared keys i configured are the same! : PSK 0x70736b

#8 Updated by Tobias Brunner over 9 years ago

The PSK used by the client (or load-tester in general) is defined with load-tester.preshared_key (the default is "default-psk").

#9 Updated by Bin Liu over 9 years ago

i changed.

server:
[root@localhost ln]# cat /etc/ipsec.secrets
: PSK 123456

client:
preshared_key = 123456

but still error.same reason:tried 1 shared key for '10.11.1.1' - 'c2-r1.strongswan.org', but MAC mismatched

#10 Updated by Bin Liu over 9 years ago

i tried serval times as you said and changed different secrets. still MAC mismatched.pls help me to make it right! thanks very much!

for example:

client:

  1. strongswan.conf(5) for details.
    load-tester { # enable the plugin
    load = yes
    enable = yes # 10000 connections, ten in parallel
    initiators = 1
    iterations = 1 # use a delay of 100ms, overall time is: iterations * delay = 100s
    delay = 50
    addrs { # install ~32K addresses on eth0
    ens255f0 = 10.10.0.0/18
    } # install all addresses with a /16 prefix
    addrs_prefix = 8 # address of the gateway (releases before 5.0.2 used the "remote" keyword!)
    responder = 10.11.1.1 # IKE-proposal to use
    proposal = aes128-sha1-modp2048
    esp = aes128-sha1 # use faster PSK authentication instead of 1024bit RSA
    initiator_auth = psk
    responder_auth = psk
    responder_id = 10.11.1.1
    preshared_key = "default-psk" # request a virtual IP using configuration payloads
    request_virtual_ip = yes # disable IKE_SA rekeying (default)
    ike_rekey = 0 # enable CHILD_SA every 60s
    child_rekey = 0
    initiator_tsr = 192.0.0.0/8 # do not delete the IKE_SA after it has been established (default)
    delete_after_established = no # do not shut down the daemon if all IKE_SAs established
    shutdown_when_complete = no

    }

server:

[root@localhost home]# cat /etc/ipsec.secrets
: PSK "default-psk"

Loading conn 'tt'
auto=add
dpdaction=none
esp=aes-sha1!
ike=aes128-sha1-modp2048
keyexchange=ikev2
keyingtries=%forever
left=10.11.1.1
leftauth=psk
leftsubnet=0.0.0.0/0
mobike=no
reauth=no
rekey=no
right=%any
rightauth=psk
rightid=%any
rightsourceip=1.0.0.0/8

#11 Updated by Tobias Brunner over 9 years ago

  • Related to Issue #1519: SA establish failed using load-tester as client in PSK scenario, always reporting MAC mismatched on server. added

#12 Updated by Noel Kuntze over 8 years ago

Is this still relevant?

#13 Updated by Noel Kuntze over 8 years ago

  • Status changed from Feedback to Closed
  • Resolution set to No feedback