Project

General

Profile

Issue #3119

GRE tunnel does not work when both VPN servers are behind NAT

Added by Florin Andrei 13 days ago. Updated 12 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
configuration
Affected version:
5.7.2
Resolution:

Description

Environment:
RHEL8, kernel 4.18.0-80.4.2.el8_0.x86_64, strongSwan 5.7.2, iproute2-ss180813 (4.18.0)
Two VPN servers, both are behind NAT (EC2 instances with private IPs assigned to eth0, and public EIPs attached to them).

Goal:
Establish an IPSec tunnel between the two VPN instances, through NAT. Within the IPSec tunnel, create a GRE tunnel. Routing at the IPSec level needs to be minimal, just enough to establish GRE. Make sure GRE works, and setup site-to-site routing through GRE. (Later on do dynamic routing with BGP, but make sure GRE works first.)

Test to show GRE works:
From site1-vpn try to ping site2-host

Problem:
The GRE tunnel does not appear to work via IPSec. I get the GRE packets at the other end of the tunnel, but the payload is not extracted from the GRE envelope (or the payload is not routed correctly to the destination).

Network topology:
Two separate VPCs, no peering. Each VPN instance is the default gateway for several subnets on each VPC.

VPC1: 10.0.1.0/24                                                               VPC2: 10.0.2.0/24

+------------+                                                                     +------------+
|            |eth0                              VPN                            eth0|            |
| site1-vpn  |=====================================================================| site2-vpn  |
|            |10.0.1.254/25 | 35.155.151.175           52.25.225.42 | 10.0.2.254/25|            |
+------------+                                                                     +------------+

subnetPub                                                                               subnetPub
----------------------------                                         ----------------------------
subnetPriv                                                                             subnetPriv

+------------+                                                                     +------------+
|            |eth0                                                             eth0|            |
| site1-host |                                                                     | site2-host |
|            |10.0.1.126/25                                           10.0.2.126/25|            |
+------------+                                                                     +------------+

ipsec.conf:

site1:

#######################################################################
config setup
  strictcrlpolicy=yes
  uniqueids = no

conn %default
  authby = psk
  auto = route
  compress = yes
  dpdaction = restart
  dpddelay = 5s
  dpdtimeout = 30s
  esp = aes256-sha512-modp4096
  forceencaps = yes
  fragmentation = yes
  ike = aes256-sha512-modp4096
  keyexchange = ikev2
  type = tunnel

conn site1-site2
  left = 10.0.1.254
  leftid = 10.0.1.254
  leftsubnet = 10.0.1.254/32
  right = 52.25.225.42
  rightid = 10.0.2.254
  rightsubnet = 10.0.2.254/32
#######################################################################

site2:

#######################################################################
config setup
  strictcrlpolicy=yes
  uniqueids = no

conn %default
  authby = psk
  auto = route
  compress = yes
  dpdaction = restart
  dpddelay = 5s
  dpdtimeout = 30s
  esp = aes256-sha512-modp4096
  forceencaps = yes
  fragmentation = yes
  ike = aes256-sha512-modp4096
  keyexchange = ikev2
  type = tunnel

conn site2-site1
  left = 10.0.2.254
  leftid = 10.0.2.254
  leftsubnet = 10.0.2.254/32
  right = 35.155.151.175
  rightid = 10.0.1.254
  rightsubnet = 10.0.1.254/32
#######################################################################

strongswan statusall:

site1:

#######################################################################
Status of IKE charon daemon (strongSwan 5.7.2, Linux 4.18.0-80.4.2.el8_0.x86_64, x86_64):
  uptime: 27 minutes, since Jul 10 23:49:15 2019
  malloc: sbrk 2834432, mmap 0, used 960144, free 1874288
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 5
  loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
Listening IP addresses:
  10.0.1.254
Connections:
 site1-site2:  10.0.1.254...52.25.225.42  IKEv2, dpddelay=5s
 site1-site2:   local:  [10.0.1.254] uses pre-shared key authentication
 site1-site2:   remote: [10.0.2.254] uses pre-shared key authentication
 site1-site2:   child:  10.0.1.254/32 === 10.0.2.254/32 TUNNEL, dpdaction=restart
Routed Connections:
 site1-site2{1}:  ROUTED, TUNNEL, reqid 1
 site1-site2{1}:   10.0.1.254/32 === 10.0.2.254/32
Security Associations (1 up, 0 connecting):
 site1-site2[1]: ESTABLISHED 25 minutes ago, 10.0.1.254[10.0.1.254]...52.25.225.42[10.0.2.254]
 site1-site2[1]: IKEv2 SPIs: 8a29b0f64a71dcfd_i* 46db87418da51f39_r, pre-shared key reauthentication in 2 hours
 site1-site2[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_4096
 site1-site2{2}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: cc11064f_i c47401ae_o, IPCOMP CPIs: 4cab_i 2bca_o
 site1-site2{2}:  AES_CBC_256/HMAC_SHA2_512_256, 124824 bytes_i (1486 pkts, 0s ago), 124824 bytes_o (1486 pkts, 0s ago), rekeying in 23 minutes
 site1-site2{2}:   10.0.1.254/32 === 10.0.2.254/32
#######################################################################

site2:

#######################################################################
Status of IKE charon daemon (strongSwan 5.7.2, Linux 4.18.0-80.4.2.el8_0.x86_64, x86_64):
  uptime: 28 minutes, since Jul 10 23:49:25 2019
  malloc: sbrk 1888256, mmap 0, used 931376, free 956880
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
  loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
Listening IP addresses:
  10.0.2.254
Connections:
 site2-site1:  10.0.2.254...35.155.151.175  IKEv2, dpddelay=5s
 site2-site1:   local:  [10.0.2.254] uses pre-shared key authentication
 site2-site1:   remote: [10.0.1.254] uses pre-shared key authentication
 site2-site1:   child:  10.0.2.254/32 === 10.0.1.254/32 TUNNEL, dpdaction=restart
Routed Connections:
 site2-site1{1}:  ROUTED, TUNNEL, reqid 1
 site2-site1{1}:   10.0.2.254/32 === 10.0.1.254/32
Security Associations (1 up, 0 connecting):
 site2-site1[1]: ESTABLISHED 26 minutes ago, 10.0.2.254[10.0.2.254]...35.155.151.175[10.0.1.254]
 site2-site1[1]: IKEv2 SPIs: 8a29b0f64a71dcfd_i 46db87418da51f39_r*, pre-shared key reauthentication in 2 hours
 site2-site1[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_4096
 site2-site1{2}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c47401ae_i cc11064f_o, IPCOMP CPIs: 2bca_i 4cab_o
 site2-site1{2}:  AES_CBC_256/HMAC_SHA2_512_256, 130536 bytes_i (1554 pkts, 1s ago), 130536 bytes_o (1554 pkts, 1s ago), rekeying in 18 minutes
 site2-site1{2}:   10.0.2.254/32 === 10.0.1.254/32
#######################################################################

iptables:

site1:

#######################################################################
[root@site1-vpn ~]# iptables -L -n -v -t filter
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@site1-vpn ~]# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
#######################################################################

site2:

#######################################################################
[root@site2-vpn ~]# iptables -L -n -v -t filter
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@site2-vpn ~]# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
#######################################################################

sysctl (both VPN instances)

net.ipv4.ip_forward = 1

From site1-vpn I can ping site2-vpn:

[root@site1-vpn ~]# ping -n 10.0.2.254
PING 10.0.2.254 (10.0.2.254) 56(84) bytes of data.
64 bytes from 10.0.2.254: icmp_seq=1 ttl=64 time=0.444 ms
64 bytes from 10.0.2.254: icmp_seq=2 ttl=64 time=0.535 ms
64 bytes from 10.0.2.254: icmp_seq=3 ttl=64 time=0.469 ms

Now create the GRE tunnel:

# site1
ip tunnel add mytunnel mode gre local 10.0.1.254 remote 10.0.2.254 ttl 255
ip link set mytunnel up
ip route add 10.0.2.0/24 dev mytunnel

# site2
ip tunnel add mytunnel mode gre local 10.0.2.254 remote 10.0.1.254 ttl 255
ip link set mytunnel up
ip route add 10.0.1.0/24 dev mytunnel

ip address:

#######################################################################
[root@site1-vpn ~]# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:07:4e:61:6a:94 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.254/25 brd 10.0.1.255 scope global dynamic noprefixroute eth0
       valid_lft 3164sec preferred_lft 3164sec
    inet6 fe80::7:4eff:fe61:6a94/64 scope link 
       valid_lft forever preferred_lft forever
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: mytunnel@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8977 qdisc noqueue state UNKNOWN group default qlen 1000
    link/gre 10.0.1.254 peer 10.0.2.254
    inet6 fe80::5efe:a00:1fe/64 scope link 
       valid_lft forever preferred_lft forever

[root@site2-vpn ~]# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:bd:b5:ba:f7:a4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.254/25 brd 10.0.2.255 scope global dynamic noprefixroute eth0
       valid_lft 3132sec preferred_lft 3132sec
    inet6 fe80::bd:b5ff:feba:f7a4/64 scope link 
       valid_lft forever preferred_lft forever
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: mytunnel@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8977 qdisc noqueue state UNKNOWN group default qlen 1000
    link/gre 10.0.2.254 peer 10.0.1.254
    inet6 fe80::5efe:a00:2fe/64 scope link 
       valid_lft forever preferred_lft forever
#######################################################################

ip route show

#######################################################################
[root@site1-vpn ~]# ip route show
default via 10.0.1.129 dev eth0 proto dhcp metric 100 
10.0.1.128/25 dev eth0 proto kernel scope link src 10.0.1.254 metric 100 
10.0.2.0/24 dev mytunnel scope link

[root@site2-vpn ~]# ip route show
default via 10.0.2.129 dev eth0 proto dhcp metric 100 
10.0.1.0/24 dev mytunnel scope link 
10.0.2.128/25 dev eth0 proto kernel scope link src 10.0.2.254 metric 100
#######################################################################

ip tunnel show

#######################################################################
[root@site1-vpn ~]# ip tunnel show
gre0: gre/ip remote any local any ttl inherit nopmtudisc
mytunnel: gre/ip remote 10.0.2.254 local 10.0.1.254 ttl 255

[root@site2-vpn ~]# ip tunnel show
gre0: gre/ip remote any local any ttl inherit nopmtudisc
mytunnel: gre/ip remote 10.0.1.254 local 10.0.2.254 ttl 255
#######################################################################

ssh to site1-vpn, try to ping site2-host (across the GRE/VPN tunnels, through site2-vpn), not working:

[root@site1-vpn ~]# ping -n 10.0.2.126
PING 10.0.2.126 (10.0.2.126) 56(84) bytes of data.

tcpdump on site2-vpn:

[root@site2-vpn ~]# tcpdump -n -s 0 -i eth0 not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:31:44.420295 IP 35.155.151.175.ipsec-nat-t > 10.0.2.254.ipsec-nat-t: UDP-encap: ESP(spi=0xc47401ae,seq=0x973), length 168
00:31:44.420379 IP 10.0.1.254 > 10.0.2.254: GREv0, length 88: IP 10.0.1.254 > 10.0.2.126: ICMP echo request, id 9678, seq 76, length 64
00:31:45.444307 IP 35.155.151.175.ipsec-nat-t > 10.0.2.254.ipsec-nat-t: UDP-encap: ESP(spi=0xc47401ae,seq=0x974), length 168
00:31:45.444368 IP 10.0.1.254 > 10.0.2.254: GREv0, length 88: IP 10.0.1.254 > 10.0.2.126: ICMP echo request, id 9678, seq 77, length 64

[root@site2-vpn ~]# tcpdump -n -s 0 -i mytunnel not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mytunnel, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
00:32:28.452417 IP 10.0.1.254 > 10.0.2.126: ICMP echo request, id 9678, seq 119, length 64
00:32:29.476432 IP 10.0.1.254 > 10.0.2.126: ICMP echo request, id 9678, seq 120, length 64

tcpdump on site2-host (the destination of ICMP echo request) shows no activity whatsoever.

From site2-vpn I can ping site2-host just fine:

[root@site2-vpn ~]# ping -n 10.0.2.126
PING 10.0.2.126 (10.0.2.126) 56(84) bytes of data.
64 bytes from 10.0.2.126: icmp_seq=1 ttl=64 time=0.495 ms
64 bytes from 10.0.2.126: icmp_seq=2 ttl=64 time=0.496 ms

On site2-host, the security groups allow ICMP from 0.0.0.0/0

If I erase the GRE tunnel, modify ipsec.conf on both sides so that leftsubnet and rightsubnet are 10.0.1.0/24 and 10.0.2.0/24 (and viceversa on site2), then from site1-vpn I can ping site2-host just fine. This is no GRE, just IPSec, and static routing done by strongSwan. This means the problem is not routing or security groups in site2.

History

#1 Updated by Tobias Brunner 12 days ago

  • Category set to configuration
  • Status changed from New to Feedback

Could be an rp_filter problem. Also, that the remote endpoint of the GRE device is part of the subnet routed via it doesn't seem ideal.

#2 Updated by Noel Kuntze 12 days ago

Pastebin the output of `iptables-save`, please and sysctl net.ipv4.conf | grep -F .rp_filter.

Make sure that the remote peer's network is routed over the EC2 instance that provides the tunnel to that network.

EDIT: You do that in the AWS webinterface, not on the host.

Also available in: Atom PDF