Frequently Asked Questions (FAQ) » History » Version 9

« Previous - Version 9/64 (diff) - Next » - Current version
Daniel Mentz, 07.12.2009 11:17
"next payload type of ISAKMP Message has an unknown value" caused by psk mismatch

Frequently Asked Questions

Q: I'm trying to set up a VPN tunnel with a ZyXELL/Linksys/X router but the other side keeps on telling me "no proposal chosen" when strongSwan initiates the connection.

A: Make sure that the peer supports all the algorithms (including the key lengths) which strongSwan proposes for IKE and ESP. In terms of IKE, the proposal consists of the following parts: Encryption algorithm, hash algorithm (PRF) and DH group. In terms of ESP the proposal includes the following: Encryption algorithm, hash algorithm, pfs group (DH group) and compression algorithm. There are lots of IPsec implementations out there that do not support compression or have implemented it erronously. So the first thing to try in this situation is to switch compression off on the peer. strongSwan's default setting is


See also Chapter 14.1 Authentication and encryption algorithms of the strongSwan documentation. It has good information about the relevant parameters.

Q: I'm getting the error message "no RSA public key known for '....' ". What am I doing wrong?

A: If you are using RSA based signatures for authentication strongSwan needs to have the peer's RSA public key in order to verify its authentication. This public key can be provided either by using the rightrsasigkey directive in the configuration file which was popular with FreeS/WAN or it can be extracted from the peer's X.509 certificate. This certificate can in turn be preloaded via the rightcert directive if it is available locally or it can be requested from the remote end by a certificate request. Now if the certificate is missing one reason might be that the remote end refused to send it. Another reason could be that strongSwan did not send a certificate request. This happens if you set the nocrsend option to yes. The Astaro Security Gateway which uses strongSwan behind the scene is known to do that. In order to make the IPsec connection work in that scenario you need to set leftsendcert to yes on the other end. With leftsendcert=yes strongSwan sends its certificate across even if no certificate request was received. This helps to interoperate with some misconfigured peers.

Q: I want to set up strongSwan to interoperate with Microsoft Windows using L2TP/IPsec. I'm getting the error message "NAT-Traversal: Transport mode disabled due to security concerns" which results in strongSwan sending an encrypted notification BAD_PROPOSAL_SYNTAX

A: Here is a quote from strongSwan lead developer Andreas Steffen on how to deal with this problem:

NAT-Traversal with IPsec transport mode has some inherent security risks. Since Microsoft doesn't care about this please compile strongSwan with the option

./configure  --enable-nat-transport

Q: Does strongSwan support IKEv1 Aggressive Mode?

A: Quote from Andreas Steffen:
bq. No, strongSwan does not support IKEv1 Aggressive Mode and never will. Otherwise we would have called the project "weakSwan".

Q: strongSwan fails to initiate a connection to a peer. I'm using RSA authentication and I noticed the two error messages:

discarding duplicate packet; already STATE_MAIN_I3

on the initiator side and
max number of retransmissions (2) reached STATE_MAIN_R2

on the responder side.

A: This problem might be related to the Path MTU (Maximum Transmission Unit). The IKE protocol is transported in UDP datagrams. As result the UDP datagrams also contain the X.509 certificate you are using. Now, if you're using a large certificate the UDP datagram might get bigger than the PMTU. That's the point where IP fragmentation kicks in and cuts your IP packet / UDP datagram in two or more pieces. There are some firewalls out there that strictly block IP fragments and therefore hamper your IKE connection. Large X.509 certificates could result from long Distinguished names or from long RSA keys (2048 bit). As a workaround you can reconfigure your firewall, try to make your certificates smaller or preload the certificates on both sides and thereby get away without transmitting the certificates over UDP.

Q: How can I turn off NAT traversal in charon (IKEv2) ?

A: NAT traversal cannot be disabled in the IKEv2 charon daemon. If you don't like automatic port floating to UDP/4500 due to the MOBIKE protocol (RFC 4555) which happens even if no NAT situation exists then you can disable MOBIKE by adding


to ipsec.conf in the connection definition.

Q: I'm trying to setup strongSwan to interop with a device from Juniper. The connection setup fails. I found the following message in the log file:

ignoring CERT_PKCS7_WRAPPED_X509 certificate request payload

A: The problem is that Juniper expects strongSwan to send its certificate[s] in CERT_PKCS7_WRAPPED_X509 format which is quite unusual. strongSwan can parse such payloads (e.g. Windows XP sends them if there is a multi-level certificate chain) but currently cannot construct them since there was never a need. We have full PKCS#7 functionality in our scepclient tool but it hasn't be integrated into the pluto daemon.

Are you using a multi-level certificate hierarchy and if yes could you import the root and all intermediate CA certificates statically on your Juniper box? Or just use a simple certificate hierarchy with path length 0?

Q: I'm trying to set up a connection using a pre-shared key configuration. I get the following error message:

pluto[37]: packet from 10.x.x.30:500: next payload type of ISAKMP Message has an unknown value: 33

A: This error message usually points to a difference in the pre-shared key configured on the two server. With the wrong key the receiver is not able to correctly decrypt the incoming traffic. Please check the configured PSKs in /etc/ipsec.secrets.