Issue #1518
Why does the performance of strongswan decreases dramatically when the number of connection is huge. pls help me!
Description
I test the performance of the ike negotiation by strongswan when the number of connection is large. And find the result is strange. Strongswan are both used by client and server. The client start many connections while the server (security gateway) has one with auto = add.
My OS is linux and board has 24 logical cores.
1) if client starts 1 conns, it takes 160ms more or less to establish all tunnels.
2) if client starts 16 conns, it takes only 370ms to establish all tunnels. because of the multi-threads it is very fast i think.
3) if client starts 1000 conns, it takes 25s more or less to establish all tunnels. nearly 400 tunnels up every 10 sec.
4) Howerver, if the client starts 5000 conns, it takes 50 minutes or more to establish all tunnels. It seems the performance decreases dramatically unbelievable. I see that from the beginning 40 tunnels up every 10 sec and soon 20 tunnels up every 10 sec. it becomes more and more slow as time passed by.
i can not understand why.
1) does the client send too slowly or the server process too slowly when the number of conn is large? what may happen?(i see the server does not take much cpu usage and is in "S" state while client always takes 100%)
2) how can i improve the performance of ike nego when the number of conn is large? i have 24 cores and only one is used by charon. it is a waste.
please help me. my hardware (24cores and NUMA) on machine is very strong and not a bottleneck i think.
the config file and packets captured by wireshark in 1000 and 5000 connections is provided in attachment.
Related issues
History
#1 Updated by Tobias Brunner over 9 years ago
- Status changed from New to Feedback
1) does the client send too slowly or the server process too slowly when the number of conn is large? what may happen?(i see the server does not take much cpu usage and is in "S" state while client always takes 100%)
Check the log (in particular the timestamps for individual threads to spot delays and if there are retransmits).
2) how can i improve the performance of ike nego when the number of conn is large? i have 24 cores and only one is used by charon. it is a waste.
Did you have a look at IkeSaTable and JobPriority? Also consider using load-tester instead of thousands of configs in ipsec.conf (which is not particularly efficient).
#2 Updated by Bin Liu over 9 years ago
thanks!
How to start the load-tester?
i tried but error returned. is there a description about how.
[root@localhost ln]# ipsec load-tester
Usage:
/usr/libexec/ipsec/load-tester initiate <count> [<delay in ms>]
[root@localhost ln]#
[root@localhost ln]# ipsec load-tester initiate 10000
connecting to /var/run/charon.ldt failed: No such file or directory
#3 Updated by Tobias Brunner over 9 years ago
You need to load and enable the load-tester plugin.
#4 Updated by Bin Liu over 9 years ago
OK. i got it.
but the ike nego failed.
logs:
07[ENC] generating IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ] 07[NET] sending packet: from 10.10.38.93[500] to 10.10.1.1[500] (300 bytes)
28[NET] received packet: from 10.10.38.93[500] to 10.10.1.1[500] (300 bytes) 28[ENC] parsed IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ] 28[IKE] received cert request for "CN=srv, OU=load-test, O=strongSwan" 28[CFG] looking for peer configs matching 10.10.1.1[srv.strongswan.org]...10.10.38.93[c9822-r1.strongswan.org] 28[CFG] selected peer config 'load-test' 28[IKE] authentication of 'c9822-r1.strongswan.org' with pre-shared key successful 28[IKE] authentication of 'srv.strongswan.org' (myself) with pre-shared key 28[IKE] IKE_SA load-test[19745] established between 10.10.1.1[srv.strongswan.org]...10.10.38.93[c9822-r1.strongswan.org] 28[IKE] peer requested virtual IP %any 28[IKE] no virtual IP found for %any requested by 'c9822-r1.strongswan.org' 28[IKE] no virtual IP found, sending INTERNAL_ADDRESS_FAILURE 28[IKE] configuration payload negotiation failed, no CHILD_SA built 28[IKE] failed to establish CHILD_SA, keeping IKE_SA 28[ENC] generating IKE_AUTH response 1 [ IDr AUTH N(INT_ADDR_FAIL) ] 28[NET] sending packet: from 10.10.1.1[500] to 10.10.38.93[500] (124 bytes)
27[NET] received packet: from 10.10.1.1[500] to 10.10.38.93[500] (124 bytes) 27[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(INT_ADDR_FAIL) ] 27[IKE] authentication of 'srv.strongswan.org' with pre-shared key successful 27[IKE] IKE_SA load-test[19375] established between 10.10.38.93[c9822-r1.strongswan.org]...10.10.1.1[srv.strongswan.org] 27[IKE] received INTERNAL_ADDRESS_FAILURE notify, no CHILD_SA built 27[IKE] failed to establish CHILD_SA, keeping IKE_SA
the server side:
config setup conn tt reauth=no rekey=no dpdaction = none #replay_window = 0 keyingtries=%forever keyexchange=ikev2 ike=modp2048 left=10.10.1.1 leftsubnet=0.0.0.0/0 #leftikeport=500 #leftsourceip=%config leftauth=psk right=%any rightauth=psk #rightsubnet=1.1.1.0/24 rightsourceip=1.0.0.0/8 esp=aes-sha1! mobike=no auto=add
the client:
load-tester { # enable the plugin load = yes enable = yes # 10000 connections, ten in parallel initiators = 10 iterations = 1000 # use a delay of 100ms, overall time is: iterations * delay = 100s delay = 50 addrs { # install ~32K addresses on eth0 ens255f0 = 10.10.0.0/18 } # install all addresses with a /16 prefix addrs_prefix = 8 # address of the gateway (releases before 5.0.2 used the "remote" keyword!) responder = 10.10.1.1 # IKE-proposal to use ike = modp2048 esp = aes128-sha1 # use faster PSK authentication instead of 1024bit RSA initiator_auth = psk responder_auth = psk # request a virtual IP using configuration payloads request_virtual_ip = yes # disable IKE_SA rekeying (default) ike_rekey = 0 # enable CHILD_SA every 60s child_rekey = 0 initiator_tsr = 192.0.0.0/8 # do not delete the IKE_SA after it has been established (default) delete_after_established = no # do not shut down the daemon if all IKE_SAs established shutdown_when_complete = no }
why the client can not get the virtue ip? i configured rightsourceip=1.0.0.0/8.
#5 Updated by Tobias Brunner over 9 years ago
28[CFG] selected peer config 'load-test'
Looks like you enabled the load-tester plugin on the server. Either disable it or set load-tester.pool on the server to a named pool (could be 1.0.0.0/8 if you still load the config in ipsec.conf) that provides the IP addresses.
#6 Updated by Bin Liu over 9 years ago
thanks very much. But why the server can't find the peer.
client side log:
Starting strongSwan 5.3.0 IPsec [starter]...
Loading config setup
cachecrls=no
strictcrlpolicy=ifuri
uniqueids=keep
found netkey IPsec stack
Attempting to start charon...
00[DMN] Starting IKE charon daemon (strongSwan 5.3.0, Linux 3.10.0-327.el7.x86_64, x86_64)
00[CFG] loaded load-tester address pool 10.10.0.0/18 on ens255f0
00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
00[CFG] loading crls from '/etc/ipsec.d/crls'
00[CFG] loading secrets from '/etc/ipsec.secrets'
00[CFG] loaded IKE secret for %any
00[LIB] loaded plugins: charon aes des rc2 sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem fips-prf gmp xcbc cmac hmac attr load-tester kernel-netlink resolve socket-default stroke updown xauth-generic
00[JOB] spawning 32 worker threads
02[CFG] assigning new lease to 'ext-2'
02[CFG] installed load-tester IP 10.10.0.1 on ens255f0
02[IKE] initiating IKE_SA load-test1 to 10.11.1.1
charon (29891) started after 460 ms
02[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) ]
02[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
08[IKE] retransmit 1 of request with message ID 0
08[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
10[IKE] retransmit 2 of request with message ID 0
10[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
11[NET] received packet: from 10.11.1.1500 to 10.10.0.1500 (456 bytes)
11[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) N(MULT_AUTH) ]
11[IKE] sending cert request for "CN=srv, OU=load-test, O=strongSwan"
11[IKE] authentication of 'c2-r1.strongswan.org' (myself) with pre-shared key
11[IKE] establishing CHILD_SA load-test
11[ENC] generating IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ]
11[NET] sending packet: from 10.10.0.1500 to 10.11.1.1500 (300 bytes)
12[NET] received packet: from 10.11.1.1500 to 10.10.0.1500 (76 bytes)
12[ENC] parsed IKE_AUTH response 1 [ N(AUTH_FAILED) ]
12[IKE] received AUTHENTICATION_FAILED notify error
12[CFG] lease 10.10.0.1 by 'ext-2' went offline
server side log:
Stopping strongSwan IPsec failed: starter is not running
Starting strongSwan 5.4.0 IPsec [starter]...
Loading config setup
Loading conn 'tt'
auto=add
dpdaction=none
esp=aes-sha1!
ike=aes128-sha1-modp2048
keyexchange=ikev2
keyingtries=%forever
left=10.11.1.1
leftauth=psk
leftsubnet=0.0.0.0/0
mobike=no
reauth=no
rekey=no
right=%any
rightauth=psk
rightsourceip=1.0.0.0/8
found netkey IPsec stack
Attempting to start charon...
00[DMN] Starting IKE charon daemon (strongSwan 5.4.0, Linux 3.10.0-327.el7.x86_64, x86_64)
00[CFG] loading ca certificates from '/etc/ipsec/ipsec.d/cacerts'
00[CFG] loading aa certificates from '/etc/ipsec/ipsec.d/aacerts'
00[CFG] loading ocsp signer certificates from '/etc/ipsec/ipsec.d/ocspcerts'
00[CFG] loading attribute certificates from '/etc/ipsec/ipsec.d/acerts'
00[CFG] loading crls from '/etc/ipsec/ipsec.d/crls'
00[CFG] loading secrets from '/etc/ipsec/ipsec.secrets'
00[CFG] loaded IKE secret for %any
00[LIB] loaded plugins: charon aes des rc2 sha2 sha1 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem fips-prf gmp xcbc cmac hmac attr kernel-netlink resolve socket-default stroke vici updown error-notify
00[JOB] spawning 16 worker threads
charon (29210) started after 20 ms
08[CFG] received stroke: add connection 'tt'
08[CFG] adding virtual IP address pool 1.0.0.0/8
08[CFG] added configuration 'tt'
12[NET] received packet: from 10.10.0.1500 to 10.11.1.1500 (448 bytes)
12[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) ]
12[IKE] 10.10.0.1 is initiating an IKE_SA
12[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(HASH_ALG) N(MULT_AUTH) ]
12[NET] sending packet: from 10.11.1.1500 to 10.10.0.1500 (456 bytes)
13[NET] received packet: from 10.10.0.1500 to 10.11.1.1500 (300 bytes)
13[ENC] parsed IKE_AUTH request 1 [ IDi CERTREQ IDr AUTH CPRQ SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) ]
13[IKE] received 1 cert requests for an unknown ca
13[CFG] looking for peer configs matching 10.11.1.1[srv.strongswan.org]...10.10.0.1[c2-r1.strongswan.org]
13[CFG] no matching peer config found
13[ENC] generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]
13[NET] sending packet: from 10.11.1.1500 to 10.10.0.1500 (76 bytes)
server side config:
auto=add
dpdaction=none
esp=aes-sha1!
ike=aes128-sha1-modp2048
keyexchange=ikev2
keyingtries=%forever
left=10.11.1.1
leftauth=psk
leftsubnet=0.0.0.0/0
mobike=no
reauth=no
rekey=no
right=%any
rightauth=psk
rightsourceip=1.0.0.0/8
since i use psk, the id may be the ip. but there is a id named srv.strongswan.org on server and c2-r1.strongswan.org on client. I didn't config any fqdn and did not have a cert....
what's wrong?
#7 Updated by Bin Liu over 9 years ago
i configured the id on client: responder_id = 10.11.1.1
and the shared keys on both side are as follows:
client:
[root@localhost charon]# cat /etc/ipsec.secrets
: PSK 0x70736b
server:
[root@localhost ln]# cat /etc/ipsec.secrets
: PSK 0x70736b
They are the same.
But the server said that:
07[IKE] received 1 cert requests for an unknown ca
07[CFG] looking for peer configs matching 10.11.1.1[10.11.1.1]...10.10.0.1[c2-r1.strongswan.org]
07[CFG] selected peer config 'tt'
07[IKE] tried 1 shared key for '10.11.1.1' - 'c2-r1.strongswan.org', but MAC mismatched
07[ENC] generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]
07[NET] sending packet: from 10.11.1.1500 to 10.10.0.1500 (76 bytes)
what does the MAC mismatched, since the preshared keys i configured are the same! : PSK 0x70736b
#8 Updated by Tobias Brunner over 9 years ago
The PSK used by the client (or load-tester in general) is defined with load-tester.preshared_key (the default is "default-psk").
#9 Updated by Bin Liu over 9 years ago
i changed.
server:
[root@localhost ln]# cat /etc/ipsec.secrets
: PSK 123456
client:
preshared_key = 123456
but still error.same reason:tried 1 shared key for '10.11.1.1' - 'c2-r1.strongswan.org', but MAC mismatched
#10 Updated by Bin Liu over 9 years ago
i tried serval times as you said and changed different secrets. still MAC mismatched.pls help me to make it right! thanks very much!
for example:
client:
- strongswan.conf(5) for details.
load-tester { # enable the plugin
load = yes
enable = yes # 10000 connections, ten in parallel
initiators = 1
iterations = 1 # use a delay of 100ms, overall time is: iterations * delay = 100s
delay = 50
addrs { # install ~32K addresses on eth0
ens255f0 = 10.10.0.0/18
} # install all addresses with a /16 prefix
addrs_prefix = 8 # address of the gateway (releases before 5.0.2 used the "remote" keyword!)
responder = 10.11.1.1 # IKE-proposal to use
proposal = aes128-sha1-modp2048
esp = aes128-sha1 # use faster PSK authentication instead of 1024bit RSA
initiator_auth = psk
responder_auth = psk
responder_id = 10.11.1.1
preshared_key = "default-psk" # request a virtual IP using configuration payloads
request_virtual_ip = yes # disable IKE_SA rekeying (default)
ike_rekey = 0 # enable CHILD_SA every 60s
child_rekey = 0
initiator_tsr = 192.0.0.0/8 # do not delete the IKE_SA after it has been established (default)
delete_after_established = no # do not shut down the daemon if all IKE_SAs established
shutdown_when_complete = no}
server:
[root@localhost home]# cat /etc/ipsec.secrets
: PSK "default-psk"
Loading conn 'tt'
auto=add
dpdaction=none
esp=aes-sha1!
ike=aes128-sha1-modp2048
keyexchange=ikev2
keyingtries=%forever
left=10.11.1.1
leftauth=psk
leftsubnet=0.0.0.0/0
mobike=no
reauth=no
rekey=no
right=%any
rightauth=psk
rightid=%any
rightsourceip=1.0.0.0/8
#11 Updated by Tobias Brunner over 9 years ago
- Related to Issue #1519: SA establish failed using load-tester as client in PSK scenario, always reporting MAC mismatched on server. added
#12 Updated by Noel Kuntze over 8 years ago
Is this still relevant?
#13 Updated by Noel Kuntze over 8 years ago
- Status changed from Feedback to Closed
- Resolution set to No feedback