strongSwan

h1. High Availability

{{>toc}}

Starting with the upcoming 4.4.0 release, the IKEv2 daemon will experimentally support pseudo active/active High Availability and Load Sharing capabilities using a cluster of [initially] two nodes.

The source code has been merged into the master branch and will be included in the 4.4.0 release.

> *The development of the High Availability functionality has been sponsored by "secunet":http://www.secunet.com.*

h2. Problem statement

The IKEv2/IPsec protocol is not well suited for operation in an active/active cluster. While it is possible to share the state of IKE_SAs over high-speed links in a cluster, sharing the kernel-maintained IPsec ESP SAs is very difficult. Due to the strict sequence numbering of ESP packets, the overhead for synchronizing ESP sequence numbers would be very high.

The IETF ipsecme working group "currently discusses":http://www.ietf.org/id/draft-ietf-ipsecme-ipsec-ha-00.txt the problems in more details and will probably standardize a solution which involves the client to work around the problems. This will, however, require an extension to the IKEv2 protocol. Clients connecting to a highly available cluster will benefit from this features only if they support this extension, existing clients (such as the one shipped with Windows 7) will not be able to take advantage of this efforts.

h2. Possible approaches

h3. Node-to-node synchronization

While the synchronization of IKE state including sequence numbers is realistic between two nodes, exchanging state information of an ESP security association is difficult.

Synchronizing the state for each processed IPsec packet will put a high load on the nodes. Synchronization after a certain amount of packets and/or after a certain timeout can reduce the load, but will make fail-over handling much more difficult, as we have to guess on the taking-over node how many packets the failing node has processed, but could not synchronize before it failed.

Another problem with such an approach is that there is no way of doing load sharing between nodes. An SA is strictly bound to a single node until the event of a failure.

h3. Client-to-cluster synchronization

Another approach to consider is requesting state information from the client. In the event of a failure, the taking-over node can request sequence numbers from the client. But this approach has the same deficiencies as previously discussed. Further, it requires extensions to the IKE protocol between client and gateway, making existing implementations incompatible with this approach.

h2. Functional specification

strongSwan uses a slightly different approach. Our solution should provide:

* Failure detection: If a node fails due to power loss, hardware failures, kernel oops or daemon crashes, the node will be removed from the cluster.

* State synchronization: If a node is removed due to failure or administrative purposes, the cluster should already have an up-to-date copy of the nodes state to take over.

* Takeover: Node failure detection and state takeover should happen within 1-3 seconds.

* Transparent migration: TCP or application sessions should not be interrupted during take over.

* Load sharing: Load should be shared across all actives nodes in a cluster.

* Reintgration: A repaired node can be (re-)added to an existing cluster, taking over a part of the load.

* Legacy clients: No protocol extension, any IKEv2 client should be able to benefit from High Availability if connected to a cluster.

Migration of clients to another node does not affect the connection, a client does usually not detect a takeover. This allows a gateway administrator to e.g. remove a node from the load sharing cluster, apply security updates, reboot and reintegrate the node. 

h2. Selected solution

The selected solution is based on the idea of "ClusterIP":http://lwn.net/Articles/108078, a Linux kernel module allowing a set of nodes to provide a service under a single virtual IP.

h3. How ClusterIP works

All nodes in a ClusterIP-based setup act under a single virtual IP address. The nodes spoof ARP requests with a multicast MAC address. This will make the switch forwarding the packet to each node in the cluster.

The received packet is associated to a segment by calculating a hash value of it. In the simplest setup, the source address is hashed and the hash value modulo the number of segments results in the responsible segment number. Each segment is handled by exactly one node in the cluster.

The node responsible for the packet will pass it to upper layers, where all others just drop the packet in the netfilter code. Depending on the hash value, e.g. TCP connections are kept on the same node. If a node fails, a remaining node will take over the segment and process packets for it.

h3. IPsec with ClusterIP

While the ClusterIP module itself is not designed to handle IPsec traffic or even act as a forwarding router, the principle of ClusterIP is. If the IKE daemons in the cluster can synchronize the IKE state and the basic IPsec SA state without sequence numbers, a modified ClusterIP module can do the rest:

For traffic to decrypt, the SPI value of the ESP packet can be included in the hash calculation, resulting in a spread of the packet flow across all nodes. As each node processes a packet of an SA from time to time, sequence numbers are automatically incremented if a packet is processed. Sequence numbers are not mangled if a packet is not verified using the IPsec authentication algorithm, as an attacker would be able to manipulate the SA state otherwise. Paket flows are partitioned in a dozen of packets to avoid too much packet reordering.

For traffic to encrypt on the cluster, the SA is looked up and the hash value is fed with the SPI of the found SA. If the segment matches, the packet is further processed. If not, only the sequence number is incremented. To avoid assigning the same sequence number to different packets on multiple nodes, additional logic is required.

h2. Kernel Implementation

The ClusterIP Netfilter module uses an additional PREROUTING hook to mark received packets for forwarding. Two new Netfilter hooks are included in the IPsec processing, exactly before the decryption/encryption process (XFRM_IN/XFRM_OUT).

<pre>

                 v        PLAIN        ^

    +------------------------------------------------+

    |            |                     |             |

    |     +--------------+      +--------------+     |

    |  4. |  PREROUTING  |      |   DECRYPT    |  3. |

    |     +--------------+      +--------------+     |

    |            |                     ^             |

    |            v                     |             |

    |     +--------------+      +--------------+     |

    |  5. |   XFRM_OUT   |      |   XFRM_IN    |  2. |

    |     +--------------+      +--------------+     |

    |            |                     ^             |

    |            v           ^         | ESP/AH      |

    |     +--------------+   |  +--------------+     |

    |  6. |   ENCRYPT    |   +--|    INPUT     |  1. |

    |     +--------------+      +--------------+     |

    |            |                     |             |

    +------------------------------------------------+

                 v       CRYPTED       ^

</pre>

#  AH, ESP and UDP-Encapsulated ESP packets are all accepted. Other traffic is subject to the ClusterIP selection algorithm based on the source IP address (e.g. IKE traffic).

# Undecrypted IPsec traffic gets dropped using a ClusterIP algorithm, based on the IPsec SA.

# Decryption process is done on the responsible node only.

# Traffic is received on ClusterIP multicast MAC and must be tagged as unicast traffic to advance through IP forwarding.

# After IPsec policy lookup, unencrypted traffic gets dropped using a ClusterIP algorithm, based on IPsec SA. Outgoing sequence numbers are assigned before the packed drop, this will keep outgoing sequence numbers in sync on all nodes.

# Encryption process is done on the responsible node only.

Patches are available for the "XFRM input/output hooks":http://wiki.strongswan.org/attachments/111/0001-Netfilter-hooks-before-XFRM-input-and-output-process.patch and to the "ClusterIP module":http://wiki.strongswan.org/attachments/112/0002-Extended-the-CLUSTERIP-module-to-use-it-on-a-IPSec-g.patch that allows ClusterIP to work with IPsec.

The patches currently break the Netfilter ABI, iptables needs "a patch":http://wiki.strongswan.org/attachments/113/0001-Added-XFRM-hooks-to-iptables-headers.patch to work on top of a patched kernel.

h2. IKE daemon implementation

A separate high availability plugin implemented for the IKEv2 daemon charon is responsible for state synchronization between the nodes in a cluster and simple monitoring functionality. It is currently designed for two nodes, but will be extended to synchronize larger clusters in the future.

To enable the high availability plugin, build strongSwan with --enable-ha.

h3. Daemon hooks

The plugin registers itself at several hooks in the daemon. These hooks are used for notifications about SA state changes and push information to the plugin. The following hooks are used:

* ike_keys(): receives IKE key material (DH, nonces, proposals)

* ike_updown()/ike_rekey(): monitor state changes of IKE SAs

* message(): used to update IKE message IDs

* child_keys(): receives CHILD key material

* child_state_change(): monitor state changes of CHILD SAs

The plugin registers its hook functions at the daemon bus. These hooks are sufficient to synchronize all IKE- and CHILD SAs with all the state required to do a fail-over of IKE and ESP SAs.

h3. Synchronization messages

The hook functions collect the required synchronization data and prepare messages to be sent to other nodes in the cluster. Messages are sent in unencrypted UDP datagrams, sent and received on port 4510. As these messages contain sensitive key material, securing the messages by IPsec is recommended.

No packet acknowledge/retransmit scheme is currently implemented, the cluster needs a reliable network with very few packet losses. It might be necessary to use a more reliable transport protocol in the future, especially if nodes start to drop packets due to an overloaded CPU.

Messages contain a protocol version, a message type and different attributes. The following synchronization message types are currently defined:

* IKE_ADD: A new IKE_SA has been established. This message contains all information to derive key material. If the message contains a REKEY attribute, the IKE_SA inherits all required parameters from the old SA.

* IKE_UPDATE: Update IKE_SA with newer information (e.g. Identities when authentication is complete).

* IKE_DELETE: Delete an established IKE_SA.

* CHILD_ADD: CHILD_SA has been established, contains keying material.

* CHILD_DELETE: CHILD_SA has been deleted.

h3. State synchronization

Received synchronization messages are parsed, mirrored IKE and CHILD_SAs are created from this information. Mirrored CHILD_SAs do not differ from normally exchanged ones; they are installed in the kernel and handle packets if ClusterIP feels responsible for it.

IKE_SAs are installed in a special PASSIVE state. They do not handle traffic, but accept state changes from sync messages only. PASSIVE IKE_SAs are managed in the IKE_SA manager as any other SA and are accessible through e.g. "ipsec statusall".

Key derivation is repeated on mirrored SAs the same way as it is done on the real SAs. This allows the reuse of existing installation routines and the HA plugin to be very unobtrusive. A node responsible for an IKE_SA does not keep the keying material in memory, it just pushes the exchanged secret to other nodes and forgets the secrets afterwards.

h3. Control messages

In addition to the synchronization messages, the HA plugin uses control messages to notify about segment changes and optionally messages for simple monitoring functions:

* SEGMENT_DROP: List of segments the sending node is dropping responsibility.

* SEGMENT_TAKE: List of segments the sending node is taking responsibility.

* STATUS: Heartbeat message containing a list of segments the sending node is responsible.

* RESYNC: Request for resynchronization of a list of segments.

The take/drop messages are sent to notify other nodes about changes done by the daemon automatically or the administrator manually. The receiving node will automatically do the opposite action to handle all segment exactly once.

If heartbeat monitoring is enabled, the status message is periodically sent. This allows to detect the activity of the remote node and take over segments the remote node is not serving. It also implements node failure detection for simple errors.

If a replacement of a failing node is installed, reintegration of the node can be speed up by sending the resynchronization message. The active node

will start resyncing all SAs, allowing the administrator to rebalance the load distribution in the cluster afterwards.

h3. Failover

In the failover case, responsibility for complete ClusterIP segments are moved from one node to another. Responsibility for a segment can be enabled or disabled on each node. For this purpose, the plugin uses the same hashing algorithm to calculate responsibility based on the source IP address.

If a segment is activated, the plugin searches for IKE_SAs in this segment and sets the state of all PASSIVE IKE_SAs to ESTABLISHED. No further action is required: The daemon handles the IKE_SA as every other one and sends out synchronization messages for state changes.

On segment deactivation, the plugin searches for IKE_SAs in the ESTABLISHED state in this segments and sets the state to PASSIVE.

CHILD_SAs are completely unaffected from activation and deactivation: They are always active and handle traffic assigned by ClusterIP.

h3. Reintegration

To reintegrate a failed node into a cluster, the node needs state information from scratch. If all the required state has been synced, the reintegrated node can be used as failover node again. Segments can be activated on the reintegrated node only after all required state has been exchanged.

SA state automatically gets synchronized during rekeying. Each rekeying procedure provides fresh keying material which can be used to build the

mirrored IKE and CHILD_SA state. Rekeying is currently the only way to push the required state to a reintegrated node, as the key material is not stored directly on an active node.

To speed up the reintegration process, the plugin can trigger the rekeying of existing IKE- and CHILD_SAs in a segment, allowing a reintegration process to complete within seconds.

h2. Configuration

Configuration is done in two places. The necessary virtual IPs and the ClusterIP rules are installed manually. This is explicitly not done by the daemon, as the rules must stay active after daemon shutdown or error conditions.

The HA plugin requires a configuration matching to the installed ClusterIP rules. All nodes in the cluster need an identical connection configuration and credentials; IP addresses assigned to clients using configuration payloads must be set carefully using a central or two distinct address pools.

h3. ClusterIP

The configuration of the extended ClusterIP module is similar to a default ClusterIP setup. For a traffic forwarding IPsec gateway, a cluster usually

needs an internal virtual IP/MAC and an external virtual IP/MAC on each node.

<pre>

ip address add 192.168.0.200/24 dev eth0

iptables -A INPUT -i eth0 -d 192.168.0.200 -j CLUSTERIP --new \

   --hashmode sourceip --clustermac 01:00:5e:00:00:20 \

   --total-nodes 2 --local-node 1

</pre>

This example installs the virtual IP 192.168.0.200 on interface eth0 and adds a corresponding ClusterIP rule. ClusterIP rules are always added to the INPUT chain. To get the same result for segment responsibility calculation in the kernel and the HA plugin, the sourceip hashmode and a hash init value of 0 must be used (default).

The _total-nodes_ option must match the configuration of the HA plugin, and all nodes require the same virtual IP/MAC and ClusterIP configuration.

ClusterIP requires the _local-node_ option to be present. However, the HA plugin reassigns segment responsibility during daemon startup.

h3. HA plugin

The HA plugin configuration is handled in the _strongswan.conf_ file.

<pre>

charon {

    # ...

    ha {

        local = 10.0.0.2

        remote = 10.0.0.1

        segment_count = 2

        # secret = s!ronG-P5K-s3cret

        fifo_interface = yes

        monitor = yes

        resync = yes

    }

}

</pre>

The _local_ and _remote_ addresses are used to send and receive sync messages, _segment_count_ defines the number of segments to use.

If a _secret_ option is specified, the nodes automatically establish a pre-shared key authenticated IPsec tunnel for HA sync and control messages

(experimental).

The segment responsibility administration interface is enabled with the _fifo_interface_ option. The _monitor_ parameter enables the heartbeat based remote node monitoring, the _resync_ option enables automatic state resynchronization if a node joins the cluster.

h3. Administrating segment responsibility

Changing the segment responsibility is done for the daemon, where it will propagate the changes in segment responsibility to the kernel.

The HA plugin uses a very similar interface for segment control as ClusterIP. Instead of a proc entry, it uses a FIFO located at _/var/run/charon.ha_ .

Echoing +1/-1 will activate/deactivate responsibility for segment 1, while an additional command *3 will enforce a resynchronization by triggering a rekey of all SAs in segment 3.