Project

General

Profile

Issue #3169

Load test issue - signature validation failed

Added by Krishnamurthy Daulatabad 10 days ago. Updated 9 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
testing
Affected version:
5.7.1
Resolution:

Description

I am using strongswan on both client and server. This is a test setup to do the IKE load tests. I have generated 1000 client certs and a ClientCA file (using PKI). So a single client, with correct entries in ipsec.conf attempts to make 1000 tunnels to the server.

Observation:
1. If I have auto=add in ipsec.conf and then use a script with "ipsec up clientN" commands in a loop, all the tunnels are established (slow though)
2. But when I make auto=start in the ipsec.conf Around 850 tunnels gets established (pretty fast). But remaining clients get a AUTH_FAILED error. This is due to error returned by verify_emsa_pkcs1_signature() calling verify_signature() and openssl is returning below error:
error:04091068:lib(4):func(145):reason(104) .
which indicates "error:04091068:rsa routines:int_rsa_verify:bad signature"
and IKE error is "signature validation failed, looking for another key"

But if I restart the same tunnel from the client side (ipsec up clientN), it establishes successfully.
So, do you see any issue as to why there is a signature validation failure when tunnel establishment rate in high.

I am using default 16 IKE worker threads.
Please let me know if you need any information from my end.

History

#1 Updated by Tobias Brunner 10 days ago

  • Category set to testing
  • Status changed from New to Feedback
  • Priority changed from Urgent to Normal

I have generated 1000 client certs and a ClientCA file (using PKI). So a single client, with correct entries in ipsec.conf attempts to make 1000 tunnels to the server.

You know that the load-tester plugin can generate certificates on the fly?

1. If I have auto=add in ipsec.conf and then use a script with "ipsec up clientN" commands in a loop, all the tunnels are established (slow though)

That configuration backend is not really intended to manage that many connections (also applies when using auto=start). The stroke plugin limits concurrent socket connections (see charon.plugins.stroke.max_concurrent in strongswan.conf). You could use ipsec stroke up-nb clientN to initiate without logging and waiting.

2. But when I make auto=start in the ipsec.conf Around 850 tunnels gets established (pretty fast).

This way connections are initiated internally, so there is no limit as above.

So, do you see any issue as to why there is a signature validation failure when tunnel establishment rate in high.

Nope, no idea.

#2 Updated by Krishnamurthy Daulatabad 9 days ago

Thanks Tobias for your quick response. I will try the max_concurrent option and up-nb option to speed up stroke initiation.

I have tried load-tester plugin also and we had some other usage model which needed direct client/server testing like I described above.

So with the client/server test above, is openssl error due to any multi-threading issue ( I am not sure if openssl is really multi-thread safe). As I mentioned above, I don't see the issue when the tunnel establishment is slow which points to some timing issue. When the connection establishment is fast, I see this error for 100-150 tunnels and they upon retry they all get established fine (no cert issue as such)

#3 Updated by Tobias Brunner 9 days ago

I have tried load-tester plugin also and we had some other usage model which needed direct client/server testing like I described above.

What do you mean?

So with the client/server test above, is openssl error due to any multi-threading issue ( I am not sure if openssl is really multi-thread safe). As I mentioned above, I don't see the issue when the tunnel establishment is slow which points to some timing issue.

Possible, but I really have no idea.

#4 Updated by Krishnamurthy Daulatabad 9 days ago

Tobias Brunner wrote:

I have tried load-tester plugin also and we had some other usage model which needed direct client/server testing like I described above.

What do you mean?

We have a different agent (and other data path daemons) that drives the server side configuration (CA certs, keys etc) and hence we moved to standard way of initiating the tunnel (instead of load-tester)

So with the client/server test above, is openssl error due to any multi-threading issue ( I am not sure if openssl is really multi-thread safe). As I mentioned above, I don't see the issue when the tunnel establishment is slow which points to some timing issue.

Possible, but I really have no idea.

From IKE code point of view, is there any common memory area that is used by multiple threads while validating the client CA signature?

#5 Updated by Tobias Brunner 9 days ago

From IKE code point of view, is there any common memory area that is used by multiple threads while validating the client CA signature?

The CA certificate (and public key) will be shared by all IKE_SAs/threads. However, each verification uses its own context objects in the openssl plugin, so other than that the same public key is read into the context there shouldn't be any shared data.

Also available in: Atom PDF