Project

General

Profile

Issue #3346

fwmark doesn't work in client-to-site connection

Added by kay kay about 1 month ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Category:
configuration
Affected version:
5.6.2
Resolution:
No change required

Description

I'm trying to route the traffic using the fwmark, but for some reason it looks like fwmark is not taken into consideration at all.

Here is my ipsec.conf:

conn babel
        keyexchange=ikev2
        type=tunnel
        leftfirewall=yes
        left=%any
        leftcert=myCert.pem
        right=ipsec-server.com
        rightid=%ipsec-server.com
        rightsubnet=0.0.0.0/0,::/0
        leftsourceip=%config4,%config6
        auto=add

charon/kernel-netlink.conf:

kernel-netlink {
    fwmark = !0x42
    mtu = 1350
}
$ ip a show dev enp0s25
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:11:22:77:39:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.148/24 brd 192.168.1.255 scope global dynamic noprefixroute enp0s25
       valid_lft 39936sec preferred_lft 39936sec
    inet 10.25.9.1/32 scope global enp0s25
       valid_lft forever preferred_lft forever
$ ip rule
0:      from all lookup local 
220:    not from all fwmark 0x42 lookup 220 
32766:  from all lookup main 
32767:  from all lookup default
$ ip r show table 220
default via 192.168.1.1 dev enp0s25 proto static src 10.25.9.1 mtu 1350 
192.168.1.0/24 dev enp0s25 proto static src 192.168.1.148 

In this case all the traffic should go via ipsec tunnel if it is not marked:

$ curl -4 ifconfig.co
%IPSEC-SERVER-IP%

This is correct. But if I want to route a particular destination via my local ISP, I need to do the following:

iptables -t mangle -I OUTPUT -d 104.28.18.94,104.28.19.94 -j MARK --set-mark 0x42

However curl still gets an ipsec-server IP address.

I tried POSTROUTING and PREROUTING as well. Looks like the mark is ignored. Do I miss something?

History

#1 Updated by Tobias Brunner about 1 month ago

  • Category set to configuration
  • Status changed from New to Feedback

Your firewall rule probably is not considered when the initial routing decision happens and the source address for the TCP socket in curl is determined, i.e. when that happens no mark is set and the source address specified by the route in table 220 will be used. So the IPsec policy will match and the packet is tunneled. I suppose you'd have to get curl to set that mark on the socket itself (via SO_MARK) or perhaps do something via NAT to avoid that the source address changes so the IPsec policy doesn't match.

But why not use passthrough policies for these destinations?

I tried POSTROUTING and PREROUTING as well.

The former is only used when forwarding packets, not for traffic that originates locally. The latter is way too late.

#2 Updated by kay kay about 1 month ago

I tried POSTROUTING and PREROUTING as well.

The former is only used when forwarding packets, not for traffic that originates locally. The latter is way too late.

I performed tests, therefore I tried to run curl locally on the ipsec client (router), and on a remote device, which has an ipsec client in default gatevay.

Nevertheless, I enabled kernel-libipsec plugin and it created an ipsec0 tun interface. With the tun interface the routing works fine:

$ ip -4 a show dev ipsec0
14: ipsec0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 10.25.9.1/32 scope global ipsec0
       valid_lft forever preferred_lft forever

The following iptables rule allows to forward traffic for the router itself:

iptables -t mangle -D OUTPUT -d 104.28.18.94,104.28.19.94 -j MARK --set-xmark 0x42

And this rule to forward the traffic according to the routable source IP address:

iptables -t mangle -I PREROUTING -s 192.168.1.148 -j MARK --set-xmark 0x42

All the traffic from the "192.168.1.148" will be routed through ipsec.

Now the question, what are the pros and cons of using a separate TUN device except the ability to route the traffic? Does it work slower?

#3 Updated by Tobias Brunner about 1 month ago

I performed tests, therefore I tried to run curl locally on the ipsec client (router), and on a remote device, which has an ipsec client in default gatevay.

When forwarding traffic from other hosts while using virtual IPs you have to use SNAT rules in order to tunnel traffic (if you don't, no forwarded traffic will ever be tunneled as the source IP won't match the IPsec policies).

Now the question, what are the pros and cons of using a separate TUN device except the ability to route the traffic? Does it work slower?

The TUN device is not the only difference, see kernel-libipsec for caveats.

As I said before, just use passthrough policies to exclude specific traffic from the tunnel.

#4 Updated by kay kay about 1 month ago

I appreciate your help, Tobias and I tried to get rid of libipsec. However I still miss something... Let me try to explain my config:

1) I have an openwrt based router (19.07, strongswan 5.8.2), it uses pppoe to connect to an ISP. All the problems I have relate to this router.
2) I also have the same strongswan config on my laptop. When I run ipsec client with it on my laptop, it forwards all the traffic via ipsec without a problem. It works like a VPN client. curl ifconfig.co shows IPSec server address.
3) openwrt with the same config stuck once an IPSec tunnel is established, because it doesn't create the local net rule:

$ ip r show table 220
default via 1.2.3.4 dev pppoe proto static src 10.25.9.2 mtu 1300
#192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 # <<< this actually exists on my laptop, but not on the openwrt and once the connection is established, I'm completely losing a local connection to my router, because its 220 table contains only one entry

4) If I add an 192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 entry manually, I can reach the router after the ipsec connection is up and running on the openwrt router.
5) If I run a curl ifconfig.co directly on the router, I receive my local ISP IP address, which is weird, because all the traffic must be forwarded via an IPSec.
6) If I run a curl ifconfig.co on my laptop (when no ipsec is running on the laptop) I also receive my local ISP IP address. I expect to see remote IPSec server address.

There are no very specific iptables rules on the openwrt:

$ iptables-save
# Generated by iptables-save v1.8.3 on Sat Feb 22 22:18:41 2020
*nat
:PREROUTING ACCEPT [1377:200253]
:INPUT ACCEPT [544:73740]
:OUTPUT ACCEPT [734:61661]
:POSTROUTING ACCEPT [180:20684]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
-A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
-A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
-A PREROUTING -i pppoe -m comment --comment "!fw3" -j zone_wan_prerouting
-A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
-A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
-A POSTROUTING -o pppoe -m comment --comment "!fw3" -j zone_wan_postrouting
-A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
-A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
-A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
-A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
COMMIT
# Completed on Sat Feb 22 22:18:41 2020
# Generated by iptables-save v1.8.3 on Sat Feb 22 22:18:41 2020
*mangle
:PREROUTING ACCEPT [262809:223561981]
:INPUT ACCEPT [14772:2779089]
:FORWARD ACCEPT [248003:220773198]
:OUTPUT ACCEPT [10888:2981519]
:POSTROUTING ACCEPT [258866:223753417]
-A FORWARD -o pppoe -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Sat Feb 22 22:18:41 2020
# Generated by iptables-save v1.8.3 on Sat Feb 22 22:18:41 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_wan_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_wan_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_wan_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
-A INPUT -d 10.25.9.2/32 -i pppoe -m policy --dir in --pol ipsec --reqid 2 --proto esp -j ACCEPT
-A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
-A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
-A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
-A INPUT -i pppoe -m comment --comment "!fw3" -j zone_wan_input
-A FORWARD -d 10.25.9.2/32 -i pppoe -m policy --dir in --pol ipsec --reqid 2 --proto esp -j ACCEPT
-A FORWARD -s 10.25.9.2/32 -o pppoe -m policy --dir out --pol ipsec --reqid 2 --proto esp -j ACCEPT
-A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
-A FORWARD -i pppoe -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -m comment --comment "!fw3" -j reject
-A OUTPUT -s 10.25.9.2/32 -o pppoe -m policy --dir out --pol ipsec --reqid 2 --proto esp -j ACCEPT
-A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
-A OUTPUT -o pppoe -m comment --comment "!fw3" -j zone_wan_output
-A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
-A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
-A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
-A syn_flood -m comment --comment "!fw3" -j DROP
-A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
-A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
-A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
-A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
-A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
-A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o pppoe -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
-A zone_wan_dest_ACCEPT -o pppoe -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_REJECT -o pppoe -m comment --comment "!fw3" -j reject
-A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
-A zone_wan_forward -p esp -m comment --comment "!fw3: Allow-IPSec-ESP" -j zone_lan_dest_ACCEPT
-A zone_wan_forward -p udp -m udp --dport 500 -m comment --comment "!fw3: Allow-ISAKMP" -j zone_lan_dest_ACCEPT
-A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
-A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
-A zone_wan_input -p udp -m udp --dport 68 -m comment --comment "!fw3: Allow-DHCP-Renew" -j ACCEPT
-A zone_wan_input -p icmp -m icmp --icmp-type 8 -m comment --comment "!fw3: Allow-Ping" -j ACCEPT
-A zone_wan_input -p igmp -m comment --comment "!fw3: Allow-IGMP" -j ACCEPT
-A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
-A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
-A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
-A zone_wan_src_REJECT -i pppoe -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Sat Feb 22 22:18:41 2020
$ ip r
default via 1.2.3.1 dev pppoe proto static 
1.2.3.1 dev pppoe proto kernel scope link src 1.2.3.67 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
$ ip rule
0:      from all lookup local 
220:    from all lookup 220 
32766:  from all lookup main 
32767:  from all lookup default
$ ip r show table 220
default via 1.2.3.1 dev pppoe proto static src 10.25.9.2 mtu 1300 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1

Could you please help me to understand why it doesn't work, when I get rid of user-space libipsec? And why it works fine if I run the same config directly on my leptop?

#5 Updated by kay kay about 1 month ago

$ ipsec statusall
Status of IKE charon daemon (strongSwan 5.8.2, Linux 4.14.167, mips):
  uptime: 9 minutes, since Feb 22 22:34:54 2020
  worker threads: 10 of 16 idle, 6/0/0/0 working, job queue: 0/0/0/0, scheduled: 7
  loaded plugins: charon test-vectors aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac ctr gcm attr kernel-netlink resolve socket-default connmark forecast farp stroke vici smp updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap dhcp whitelist led duplicheck addrblock unity
Listening IP addresses:
  192.168.1.1
  1.2.3.67
Connections:
       babel:  %any...ipsec-server.com  IKEv2
       babel:   local:  [C=US, O=Babel, CN=kay] uses public key authentication
       babel:    cert:  "C=US, O=Babel, CN=kay" 
       babel:   remote: [ipsec-server.com] uses public key authentication
       babel:   child:  dynamic === 0.0.0.0/0 ::/0 TUNNEL
Security Associations (1 up, 0 connecting):
       babel[2]: ESTABLISHED 49 seconds ago, 1.2.3.67[C=US, O=Babel, CN=kay]...4.4.4.4[C=US, O=Babel, CN=server]
       babel[2]: IKEv2 SPIs: 32b4602409da328b_i* 589a20f6e4c83757_r, public key reauthentication in 2 hours
       babel[2]: IKE proposal: AES_CBC_128/HMAC_SHA2_256_128/PRF_AES128_XCBC/MODP_3072
       babel{2}:  INSTALLED, TUNNEL, reqid 2, ESP in UDP SPIs: c020cb98_i cbe1b3e8_o
       babel{2}:  AES_CBC_128/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o (0 pkts, 29s ago), rekeying in 41 minutes
       babel{2}:   10.25.9.2/32 === 0.0.0.0/0

#6 Updated by kay kay about 1 month ago

I made a test with a x86_64 docker image same strongswan version, where I used the same parameters, same modules, same configs.

When I run curl ifconfig.co on docker x86_64, I get the remote VPN IP.

When I run curl ifconfig.co on openwrt arm/mips, I get the local VPN IP.

I compared ip xfrm policy outputs, they are the same. I still cannot get why IPSec route doesn't work on openwrt(arm/mips), but works in the same docker env.

#7 Updated by Tobias Brunner about 1 month ago

3) openwrt with the same config stuck once an IPSec tunnel is established, because it doesn't create the local net rule:

[...]

As I said before, you can configure a passthrough policy, which will also create a route for the configured subnet. On your laptop you might have the bypass-lan plugin enabled, which automatically creates passthrough policies for locally attached subnets.

5) If I run a curl ifconfig.co directly on the router, I receive my local ISP IP address, which is weird, because all the traffic must be forwarded via an IPSec.

Might depend on how curl selects a source address. But with the route in table 220 the virtual IP address should be selected (you can check with ip route get <IP of ifconfig.co>).

6) If I run a curl ifconfig.co on my laptop (when no ipsec is running on the laptop) I also receive my local ISP IP address. I expect to see remote IPSec server address.

That will not work unless you SNAT forwarded traffic to your virtual IP. The IPsec policies are for only that IP, not for any IP address of your clients behind the router. If the virtual IP is not static, you'll have to use a custom updown script to install appropriate firewall rules. The NAT rule should also fix the local source address issue you described above.

#8 Updated by kay kay about 1 month ago

6) If I run a curl ifconfig.co on my laptop (when no ipsec is running on the laptop) I also receive my local ISP IP address. I expect to see remote IPSec server address.

That will not work unless you SNAT forwarded traffic to your virtual IP.

Well, this did work fine, when I configured strongSwan to use a tun interface, which was a part of the wan zone. All further problems occurred, when I decided to disable tun.

$ ip r get 104.28.19.94
10.8.25.20 via 10.8.25.1 dev pppoe table 220 src 10.25.9.1 uid 0
    cache mtu 1300

Even curl with the forced source IP didn't help:

$ curl --interface 10.25.9.1 ifconfig.co
%local_isp_ip%

But you were right, forcing the source IP as a SNAT solved the issue:

iptables -t nat -I POSTROUTING -d 104.28.18.94,104.28.19.94 -j SNAT --to 10.25.9.1

I still don't understand why in docker env it works out-of-the-box. Probably the pppoe interface has some specifics.

Now, almost all previous problems solved, but unfortunately I faced a new one.

Download speed of the ipsec tunnel is ~15mbit. However an upload speed is very low ~50kbit. I'm setting 1300 MTU to the tunnel, because higher MTU causes issues. When I'm setting up a connection via a tun interface, I get higher CPU load, but symmetric ~6mbit speed. Do you have an idea why upload is so low, when I don't use tun interface?

#9 Updated by Tobias Brunner about 1 month ago

Well, this did work fine, when I configured strongSwan to use a tun interface, which was a part of the wan zone. All further problems occurred, when I decided to disable tun.

These are two completely different approaches (route-based vs. policy-based).

Even curl with the forced source IP didn't help:

[...]

No idea. Maybe just curl being weird (old version, different version).

I still don't understand why in docker env it works out-of-the-box. Probably the pppoe interface has some specifics.

I don't think it is caused by the interface. You simply are not forwarding any traffic in the container and curl might behave differently for local traffic there (different version, might even depend on the kernel version).

Do you have an idea why upload is so low, when I don't use tun interface?

Maybe requires MSS clamping (see ForwardingAndSplitTunneling).

#10 Updated by kay kay about 1 month ago

OpenWRT set mss clamping by default, see a rule in mangle table:

-A FORWARD -o pppoe -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu

I also set mss/mtu in kernel-netlink:

# egrep 'mss|mtu' /etc/strongswan.d/charon/kernel-netlink.conf 
    mss = 1200
    mtu = 1300

And I even tried to set my laptop MTU to 1200, I still have the same low performance. Probably there is an interference between pppoe mtu and ipsec tunnel mtu, since they share the same interface.

ip r show table 220
default via 10.8.25.1 dev pppoe proto static src 10.25.9.1 mtu 1300 advmss 1200 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1

Adding these rules also didn't give an effect:

iptables -t mangle -A FORWARD -m policy --pol ipsec --dir in -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1201:1536 -j TCPMSS --set-mss 1200
iptables -t mangle -A FORWARD -m policy --pol ipsec --dir out -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1201:1536 -j TCPMSS --set-mss 1200

#11 Updated by kay kay about 1 month ago

UPD: I removed mtu/mss settings from the kernel-netlink.conf and now the tunnel works fine, the speed is 16 mbit/s with a 100% CPU load.

#12 Updated by kay kay about 1 month ago

Thanks for you help, Tobias. This issue can be closed. I'll open a new one about an ARM router.

#13 Updated by Tobias Brunner about 1 month ago

  • Status changed from Feedback to Closed
  • Assignee set to Tobias Brunner
  • Resolution set to No change required

Also available in: Atom PDF