strongSwan

Yes, it's generating extra messages and we'd like to use our embedded HSM only when it's necessary.

Yes, it's generating extra messages

You mean the two C_Sign() failed: DATA_LEN_RANGE messages?

and we'd like to use our embedded HSM only when it's necessary.

What do you mean? Handling that error should not have that much of an impact (checking the input data is probably one of the the first things it does).

By the way, did you try the current master branch (or the current developers release)? It contains a change in the pkcs11 plugin that should filter out unsupported signature mechanisms (at least if the HSM reports only the ones it actually supports via C_GetMechanismList - of course, that's a global command, so it does not work if the limitation depends on the key).

You mean the two C_Sign() failed: DATA_LEN_RANGE messages?

No I was wrong, I thought it was generating extra IKE_AUTH request message

What do you mean? Handling that error should not have that much of an impact (checking the input data is probably one of the the first things it does).

Well it's an embedded HSM, its lifetime also depend on usage, that's why we'd like to use it only when it's necessary.
Checking the input data, is not done, it's more a kind of try / failure / succeed

I used a pkcs11 proxy to see what's happening to the HSM.
Raw signature is supported with key size limited to 256.

Calling C_GetMechanismInfo
Input
 slotID: 1
 type: 4161 (CKM_ECDSA)
 pInfo: 0xbe88308c
Output
 pInfo: 0xbe88308c
  ulMinKeySize: 256
  ulMaxKeySize: 256
  flags: 10241
   CKF_HW: TRUE
   CKF_ENCRYPT: FALSE
   CKF_DECRYPT: FALSE
   CKF_DIGEST: FALSE
   CKF_SIGN: TRUE
   CKF_SIGN_RECOVER: FALSE
   CKF_VERIFY: TRUE
   CKF_VERIFY_RECOVER: FALSE
   CKF_GENERATE: FALSE
   CKF_GENERATE_KEY_PAIR: FALSE
   CKF_WRAP: FALSE
   CKF_UNWRAP: FALSE
   CKF_DERIVE: FALSE
   CKF_EC_F_P: FALSE
   CKF_EC_F_2M: FALSE
   CKF_EC_ECPARAMETERS: FALSE
   CKF_EC_NAMEDCURVE: FALSE
   CKF_EC_UNCOMPRESS: FALSE
   CKF_EC_COMPRESS: FALSE
   CKF_EXTENSION: FALSE
Returning 0 (CKR_OK)

Actually, signature mechanism try different hash algorithm till it manage to generate one

Calling C_SignInit
Input
 hSession: 2922383072
 pMechanism: 0xb6ce755c
  mechanism: 4161 (CKM_ECDSA)
  pParameter: (nil)
  ulParameterLen: (nil)
 hKey: 7153408
Returning 0 (CKR_OK)

First try (512 bits) => Failed

Calling C_Sign
Input
 hSession: 2922383072
 pData: 0xae3043e8
 *pData: HEX(89C83E94AF9B3FC7302C1F6E0D9E67708BD235F0B46CE94CD6D74DB79AFFD9803951806B3FF9F3B9D0EE77F54A87E18F6BA474C8852710050A29941DEFBB84DF)
 ulDataLen: 64
 pSignature: 0xae3047b8
 pulSignatureLen: 0xb0cc6318
 *pulSignatureLen: 64
Returning 33 (CKR_DATA_LEN_RANGE)

Second try (384 bits) => Failed

Calling C_Sign
Input
 hSession: 2922383072
 pData: 0xae300d30
 *pData: HEX(88E59669E2894C21EF5BC70756DA0E9DE6B220F85B8D245F38819B7CCE75F92BCE9C1682501D30263BB6DF39F69C134E)
 ulDataLen: 48
 pSignature: 0xae3043e8
 pulSignatureLen: 0xb0cc6318
 *pulSignatureLen: 64
Returning 33 (CKR_DATA_LEN_RANGE)

Thirs try (256 bits) => Ok

Calling C_Sign
Input
 hSession: 2922383072
 pData: 0xae304ba8
 *pData: HEX(E619C44A9773797519B362B3B2E3A2C298AA55029C2A9FC233B9B2150220ECBD)
 ulDataLen: 32
 pSignature: 0xae3047b8
 pulSignatureLen: 0xb0cc6318
 *pulSignatureLen: 64
Output
 pSignature: 0xae3047b8
 pulSignatureLen: 0xb0cc6318
 *pSignature: HEX(1590D51F55DDC1754B042F9B0B8AEA9A95F48C91AF764F1A51BD4E17782CFC337404A666619B0C74C4AA6B563EF00EC88F1DEECF66833460395831511217C02C)
 *pulSignatureLen: 64
Returning 0 (CKR_OK)

I also use these parameters :

strongswan.conf

charon {
    tls {
        suites = TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
    }
...

ipsec.conf

leftauth=eap-tls:ecdsa-256-sha256 

So I don't understand why SHA512 / SHA384 are tried for ECDSA signature before SHA256

Checking the input data, is not done, it's more a kind of try / failure / succeed

Are you sure about that. It seems it bails out because it feels the input data length is too long (CKR_DATA_LEN_RANGE), which is the length of the hashed data (64 for SHA-512, 48 for SHA-384 and finally 32 for SHA-256, which it accepts).

Raw signature is supported with key size limited to 256.

Hm, I see. The key length is not the problem, though. It's the data length, for which the mechanisms can't report a supported range. Actually, the PKCS#11 standard in section 2.3.6 has a table that explains that C_Sign for CKM_ECDSA accepts any input length with a footnote that says: "Input the entire raw digest. Internally, this will be truncated to the appropriate number of bits.", which seems what the HSM should do instead of reporting CKR_DATA_LEN_RANGE. Section 2.3.1 on EC signatures says it similarly: "If the length of the hash value is larger than the bit length of n, only the leftmost bits of the hash up to the length of n will be used. Any truncation is done by the token.".

Actually, signature mechanism try different hash algorithm till it manage to generate one

Yes, the TLS library tries all hash/signature-combinations the server supports in the order it sent them in the SignatureAndHashAlgorithm extension.

leftauth=eap-tls:ecdsa-256-sha256 

So I don't understand why SHA512 / SHA384 are tried for ECDSA signature before SHA256

Such constraints currently have no effect on the TLS signatures whatsoever (they are only used during certificate chain validation, which makes not much sense in leftauth).