strongSwan

Hi Guys - Sorry to have to ask for help on something that is probably pretty straight forwards, but I have got well and truly stuck!

I have a digital ocean machine acting as a gateway to a mysql server. I have been providing access to the gateway and sql sever via an openvpn server on the box. Open VPN provides access to the gateway via virtual network (10.8.0.0/24) and allows onwards connection to the digital ocean private network. This is on port 443 via tcp and all works great!

I have subsequently tried to connect 3 physical sites to the sql server, again via the gatway box. I have done this via strongswan IP sec tunnels on the gateway box, connecting to fortigate devices at each of the 3 sites. The tunnels connect, and I can connect to devices at the physical sites when ssh'd into the gateway box. However I cant connect to any device at the 3 sites via a RW warrior/openvpn connection, or from the sql box. When I connect to the fortigate devices I cant ping the gateway box through the tunnels - so there is defiantly something wrong with what I have done. All the routes (on all ends) seem to be ok/correct.

I based my configuration on the site to site psk page, and have spent lots of time reading other peoples issues - I have a feeling I have made a stupid mistake somewhere.

I have spent some time reading the splittunnel page, but being honest I am not 100% sure I understand exactly what I should be doing, and not quite sure what to look at next. I think it is to do with the firewall on the gateway box, but I am noway near an iptables expert. Rather than try and explain further by words I drew a simple network diagram which is attached to the post.

Config / troubleshooting info:

ipsec.conf

config setup

conn %default
    ikelifetime=60m
    keylife=60m
    rekeymargin=3m
    keyingtries=1
    authby=secret
    keyexchange=ikev2
    mobike=no
    forceencaps = yes

#left is internal net
#right is external

# connection to MUJ
conn muj
    left=134.209.30.189
    leftsubnet=10.131.113.114/16
    right=212.76.91.43
    rightsubnet=192.168.1.0/24
    auto=start

# connection to MDX
conn mdx
    left=134.209.30.189
    leftsubnet=10.131.113.114/16
    right=91.72.218.234
    rightsubnet=192.168.5.0/24
    auto=start

# connection to MUD
conn mud
    left=134.209.30.189
    leftsubnet=10.131.113.114/16
    right=213.210.247.98
    rightsubnet=192.168.10.0/24
    auto=start

IPSsec secrets

134.209.30.189 91.72.218.234 : PSK "secret" 
134.209.30.189 212.76.91.43 : PSK "secret" 
134.209.30.189 213.210.247.98 : PSK "secret" 

IPsec statusall output

sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-48-generic, x86_64):
  uptime: 35 hours, since May 04 21:07:10 2019
  malloc: sbrk 2195456, mmap 0, used 1224816, free 970640
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 7
  loaded plugins: charon aesni aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default connmark stroke vici updown eap-mschapv2 xauth-generic counters
Listening IP addresses:
  134.209.30.189
  10.16.0.5
  10.131.113.114
  10.8.0.1
Connections:
         mdx:  134.209.30.189...91.72.218.234  IKEv2
         mdx:   local:  [134.209.30.189] uses pre-shared key authentication
         mdx:   remote: [91.72.218.234] uses pre-shared key authentication
         mdx:   child:  10.131.0.0/16 === 192.168.5.0/24 TUNNEL
         muj:  134.209.30.189...212.76.91.43  IKEv2
         muj:   local:  [134.209.30.189] uses pre-shared key authentication
         muj:   remote: [212.76.91.43] uses pre-shared key authentication
         muj:   child:  10.131.0.0/16 === 192.168.1.0/24 TUNNEL
         mud:  134.209.30.189...213.210.247.98  IKEv2
         mud:   local:  [134.209.30.189] uses pre-shared key authentication
         mud:   remote: [213.210.247.98] uses pre-shared key authentication
         mud:   child:  10.131.0.0/16 === 192.168.10.0/24 TUNNEL
Security Associations (3 up, 0 connecting):
         muj[122]: ESTABLISHED 52 seconds ago, 134.209.30.189[134.209.30.189]...212.76.91.43[212.76.91.43]
         muj[122]: IKEv2 SPIs: 9b9ed3ec587e2412_i 49b426e198294710_r*, pre-shared key reauthentication in 54 minutes
         muj[122]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_384
         muj{722}:  INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: c8f7c18f_i e0e9a16f_o
         muj{722}:  AES_CBC_128/HMAC_SHA1_96, 252 bytes_i (3 pkts, 10s ago), 252 bytes_o (3 pkts, 10s ago), rekeying in 54 minutes
         muj{722}:   10.131.0.0/16 === 192.168.1.0/24
         mdx[121]: ESTABLISHED 23 minutes ago, 134.209.30.189[134.209.30.189]...91.72.218.234[91.72.218.234]
         mdx[121]: IKEv2 SPIs: 82b9671f5eb81709_i* 21dd5be3561b27fe_r, pre-shared key reauthentication in 31 minutes
         mdx[121]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_384
         mdx{721}:  INSTALLED, TUNNEL, reqid 80, ESP in UDP SPIs: cb57c031_i 81cac89f_o
         mdx{721}:  AES_CBC_256/HMAC_SHA2_256_128, 252 bytes_i (3 pkts, 20s ago), 252 bytes_o (3 pkts, 20s ago), rekeying in 32 minutes
         mdx{721}:   10.131.0.0/16 === 192.168.5.0/24
         mud[120]: ESTABLISHED 24 minutes ago, 134.209.30.189[134.209.30.189]...213.210.247.98[213.210.247.98]
         mud[120]: IKEv2 SPIs: 7588baff1e0f1aee_i* 85597c1e2d6fcea0_r, pre-shared key reauthentication in 31 minutes
         mud[120]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_384
         mud{712}:  INSTALLED, TUNNEL, reqid 79, ESP in UDP SPIs: c53986cc_i 0f86850a_o
         mud{712}:  AES_CBC_256/HMAC_SHA2_256_128, 252 bytes_i (3 pkts, 3s ago), 336 bytes_o (4 pkts, 3s ago), rekeying in 30 minutes
         mud{712}:   10.131.0.0/16 === 192.168.10.0/24

IP route show:

default via 134.209.16.1 dev eth0 proto static 
10.8.0.0/24 via 10.8.0.2 dev tun0 
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1 
10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.5 
10.131.0.0/16 dev eth1 proto kernel scope link src 10.131.113.114 
134.209.16.0/20 dev eth0 proto kernel scope link src 134.209.30.189 

IP route table 220:

192.168.1.0/24 via 134.209.16.1 dev eth0 proto static src 10.131.113.114 
192.168.5.0/24 via 134.209.16.1 dev eth0 proto static src 10.131.113.114 
192.168.10.0/24 via 134.209.16.1 dev eth0 proto static src 10.131.113.114 

iptables-save
note I currently have UFW on. I have tried it with it on and off. I have played around with the order of the POSTROUTING rules, and added the first 3 rules after reading the split tunnel page. The 4th POSTROUTING rule is for the openvpn virtual net. I get the impression my problem is either here or in my config.

# Generated by iptables-save v1.6.1 on Mon May  6 08:36:24 2019
*nat
:PREROUTING ACCEPT [5685:1298674]
:INPUT ACCEPT [1188:70846]
:OUTPUT ACCEPT [247:19780]
:POSTROUTING ACCEPT [247:19780]
-A POSTROUTING -s 192.168.1.0/24 -o eth0 -m policy --dir out --pol ipsec -j ACCEPT
-A POSTROUTING -s 192.168.5.0/24 -o eth0 -m policy --dir out --pol ipsec -j ACCEPT
-A POSTROUTING -s 192.168.10.0/24 -o eth0 -m policy --dir out --pol ipsec -j ACCEPT
-A POSTROUTING -s 10.8.0.0/24 -j MASQUERADE
COMMIT
# Completed on Mon May  6 08:36:24 2019
# Generated by iptables-save v1.6.1 on Mon May  6 08:36:24 2019
*filter
:INPUT DROP [25:2411]
:FORWARD ACCEPT [284:20568]
:OUTPUT ACCEPT [7:588]
:ufw-after-forward - [0:0]
:ufw-after-input - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-before-input - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-output - [0:0]
:ufw-logging-allow - [0:0]
:ufw-logging-deny - [0:0]
:ufw-not-local - [0:0]
:ufw-reject-forward - [0:0]
:ufw-reject-input - [0:0]
:ufw-reject-output - [0:0]
:ufw-skip-to-policy-forward - [0:0]
:ufw-skip-to-policy-input - [0:0]
:ufw-skip-to-policy-output - [0:0]
:ufw-track-forward - [0:0]
:ufw-track-input - [0:0]
:ufw-track-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-user-input - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-output - [0:0]
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] " 
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-forward -j ufw-user-forward
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -j ufw-not-local
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-input -j ufw-user-input
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -j ufw-user-output
-A ufw-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] " 
-A ufw-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] " 
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
-A ufw-skip-to-policy-forward -j ACCEPT
-A ufw-skip-to-policy-input -j DROP
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-track-forward -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-forward -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 22 -m comment --comment "\'dapp_OpenSSH\'" -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 443 -j ACCEPT
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] " 
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT
COMMIT

Ip forward enabled

$ cat /proc/sys/net/ipv4/ip_forward
1

When I run a traceroute on the RW device to a remote site IP (say 192.168.5.200) it gets as far as the gateway box but then gets stuck:

 1  10.8.0.1 (10.8.0.1)  23.384 ms  48.531 ms  48.570 ms
 2  134.209.16.254 (134.209.16.254)  48.603 ms 134.209.16.253 (134.209.16.253)  48.643 ms 134.209.16.254 (134.209.16.254)  48.582 ms
 3 * * *
 4 * * *
etc

TCP dump from gateway box when pinging physical site from RW box. (ping 192.168.5.200 from RW client connected via openVPN. Potentially seems like the ICMP packet is not being routed down the tunnel by the gatway box?

On gateway:
sudo tcpdump port not 22 -n | grep ICMP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:41:42.999500 IP 134.209.30.189 > 192.168.5.200: ICMP echo request, id 7128, seq 1, length 64
08:41:44.020472 IP 134.209.30.189 > 192.168.5.200: ICMP echo request, id 7128, seq 2, length 64
08:41:45.033813 IP 134.209.30.189 > 192.168.5.200: ICMP echo request, id 7128, seq 3, length 64
08:41:46.052167 IP 134.209.30.189 > 192.168.5.200: ICMP echo request, id 7128, seq 4, length 64
08:41:47.074133 IP 134.209.30.189 > 192.168.5.200: ICMP echo request, id 7128, seq 5, length 64

Pinging box (virtual ip 10.8.0.6):

$ ping 192.168.5.200
PING 192.168.5.200 (192.168.5.200) 56(84) bytes of data.
^C
--- 192.168.5.200 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 178ms

Thanks for your help!

Toby