Project

General

Profile

Issue #3480

StrongSwan process Ping replies

Added by TAHER BAHASHWAN 22 days ago. Updated 13 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
-
Affected version:
5.8.4
Resolution:

Description

Hi

I have configured StrongSwan & Palo Alto FW, and I found strange issue, is that, ping replies or requests are not processed in StronSwan, I have decrypted ESPs and I also collected a logs with@ tcpdump -s 0 -n -i nflog:5 -w@
When I open wireshark and I decrypt the ESP for the reply that received by StrongSwan I can see it, but on console the ping is not function it is hung on

 ping 10.10.101.254
PING 10.10.101.254 (10.10.101.254) 56(84) bytes of data.

Note: the StrongSwan is running on Ubuntu 18.04
Here is the strongswan config file

cat /etc/ipsec.conf
config setup
        charondebug="all" 
        uniqueids=yes
        strictcrlpolicy=no

conn IPSec-To-PaloAlto
   #aggressive = no
   #fragmentation = yes
    keyexchange = ikev2
    authby=secret
    installpolicy = yes
    type = tunnel
    left=167.xx.xx.xx
    right=95.xx.xx.xx
    leftid=167.xx.xx.xx
    rightid=95.xx.xx.xx
    leftsubnet=10.0.10.1/32
    rightsubnet=10.10.101.254/32
    ike=aes128-sha1-modp1024!
    esp=aes128-sha1-modp1024!
#    ah=sha1--modp1024!
    forceencaps = yes
    keyingtries=0
    ikelifetime=28800s
    lifetime=3600s
    dpddelay=10s
    dpdtimeout=60s
    dpdaction=restart
    auto=route

here is the decrypted ESPs ping traffic that was captured on StrongSwan, it shows the reply from Palo Alto (10.10.101.254)

No.    Time    DeltaTime    Source    Destination    Protocol    Dst Port    Length    Bytes in Flight    MTU    Info
7    2020-06-10 11:11:17.884017    0.000000    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=362/27137, ttl=64
8    2020-06-10 11:11:17.884018    0.000001    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=362/27137, ttl=64
15    2020-06-10 11:11:18.907988    0.000000    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=363/27393, ttl=64
16    2020-06-10 11:11:18.907988    0.000000    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=363/27393, ttl=64
27    2020-06-10 11:11:19.931956    0.000001    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=364/27649, ttl=64
28    2020-06-10 11:11:19.931956    0.000000    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=364/27649, ttl=64
35    2020-06-10 11:11:20.955980    0.000000    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=365/27905, ttl=64
36    2020-06-10 11:11:20.955981    0.000001    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=365/27905, ttl=64
43    2020-06-10 11:11:21.980036    0.000042    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=366/28161, ttl=64
44    2020-06-10 11:11:21.980036    0.000000    10.10.101.254    10.0.10.1    ICMP    4500    264        264    Echo (ping) reply    id=0x7961, seq=366/28161, ttl=64

Note: 10.0.10.1 is a loopback interface on StronSwan machine (Ubuntu 18.04)

History

#1 Updated by Tobias Brunner 22 days ago

  • Description updated (diff)
  • Status changed from New to Feedback

Maybe the firewall block these packets before or after decryption, or the peers disagree on UDP encapsulation. Also check /proc/net/xfrm_stat for errors.

#2 Updated by TAHER BAHASHWAN 22 days ago

Tobias Brunner wrote:

Maybe the firewall block these packets before or after decryption, or the peers disagree on UDP encapsulation. Also check /proc/net/xfrm_stat for errors.

Hi

Yes I can see that there are errors can check it please

at /proc/net/xfrm_stat | grep Error
XfrmInError                     0
XfrmInBufferError               0
XfrmInHdrError                  0
XfrmInStateProtoError           1738
XfrmInStateModeError            0
XfrmInStateSeqError             0
XfrmInPolError                  0
XfrmOutError                    0
XfrmOutBundleGenError           0
XfrmOutBundleCheckError         0
XfrmOutStateProtoError          0
XfrmOutStateModeError           362
XfrmOutStateSeqError            0
XfrmOutPolError                 0
XfrmFwdHdrError                 0
XfrmAcquireError                0

No Firewall enabled on Ubuntu except iptables and below is iptables current rules

iptables -L -v -n --line-n
Chain INPUT (policy ACCEPT 724K packets, 100M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       78  6552 NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL policy match dir in pol ipsec nflog-group 5
2       78  6552 NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL policy match dir in pol ipsec nflog-group 5
3       78  6552 NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL policy match dir in pol ipsec nflog-group 5

Chain FORWARD (policy ACCEPT 42893 packets, 126M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    18190  818K NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type !LOCAL policy match dir in pol ipsec nflog-group 5
2    18190  818K NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type !LOCAL policy match dir in pol ipsec nflog-group 5

Chain OUTPUT (policy ACCEPT 807K packets, 288M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     2017  169K NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir out pol ipsec nflog-group 5
2     2017  169K NFLOG      all  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir out pol ipsec nflog-group 5

#3 Updated by TAHER BAHASHWAN 22 days ago

Here is the syslog as well I did a fresh test

~# tail -f /var/log/syslog
Jun 10 13:59:59 ubuntu-test charon: 12[ENC] generating INFORMATIONAL response 46 [ ]
Jun 10 13:59:59 ubuntu-test charon: 12[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)
Jun 10 14:00:01 ubuntu-test charon: 14[CFG] received stroke: terminate 'IPSec-To-PaloAlto'
Jun 10 14:00:01 ubuntu-test charon: 06[IKE] deleting IKE_SA IPSec-To-PaloAlto[3] between 167.xx.xx.xx[167.xx.xx.xx]...95.xx.xx.xx[95.xx.xx.xx]
Jun 10 14:00:01 ubuntu-test charon: 06[IKE] sending DELETE for IKE_SA IPSec-To-PaloAlto[3]
Jun 10 14:00:01 ubuntu-test charon: 06[ENC] generating INFORMATIONAL request 2 [ D ]
Jun 10 14:00:01 ubuntu-test charon: 06[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)
Jun 10 14:00:01 ubuntu-test charon: 15[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (76 bytes)
Jun 10 14:00:01 ubuntu-test charon: 15[ENC] parsed INFORMATIONAL response 2 [ ]
Jun 10 14:00:01 ubuntu-test charon: 15[IKE] IKE_SA deleted
Jun 10 14:00:13 ubuntu-test charon: 09[KNL] creating acquire job for policy 10.0.10.1/32[udp/43526] === 10.10.101.254/32[udp/1025] with reqid {1}
Jun 10 14:00:13 ubuntu-test charon: 09[IKE] initiating IKE_SA IPSec-To-PaloAlto[4] to 95.xx.xx.xx
Jun 10 14:00:13 ubuntu-test charon: 09[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Jun 10 14:00:13 ubuntu-test charon: 09[NET] sending packet: from 167.xx.xx.xx[500] to 95.xx.xx.xx[500] (334 bytes)
Jun 10 14:00:13 ubuntu-test charon: 07[NET] received packet: from 95.xx.xx.xx[500] to 167.xx.xx.xx[500] (304 bytes)
Jun 10 14:00:13 ubuntu-test charon: 07[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) ]
Jun 10 14:00:13 ubuntu-test charon: 07[IKE] remote host is behind NAT
Jun 10 14:00:13 ubuntu-test charon: 07[IKE] authentication of '167.xx.xx.xx' (myself) with pre-shared key
Jun 10 14:00:13 ubuntu-test charon: 07[IKE] establishing CHILD_SA IPSec-To-PaloAlto{7} reqid 1
Jun 10 14:00:13 ubuntu-test charon: 07[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Jun 10 14:00:13 ubuntu-test charon: 07[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (300 bytes)
Jun 10 14:00:13 ubuntu-test charon: 10[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (204 bytes)
Jun 10 14:00:13 ubuntu-test charon: 10[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
Jun 10 14:00:13 ubuntu-test charon: 10[IKE] authentication of '95.xx.xx.xx' with pre-shared key successful
Jun 10 14:00:13 ubuntu-test charon: 10[IKE] IKE_SA IPSec-To-PaloAlto[4] established between 167.xx.xx.xx[167.xx.xx.xx]...95.xx.xx.xx[95.xx.xx.xx]
Jun 10 14:00:13 ubuntu-test charon: 10[IKE] scheduling reauthentication in 27839s
Jun 10 14:00:13 ubuntu-test charon: 10[IKE] maximum IKE_SA lifetime 28379s
Jun 10 14:00:13 ubuntu-test charon: 10[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Jun 10 14:00:13 ubuntu-test charon: 10[IKE] CHILD_SA IPSec-To-PaloAlto{7} established with SPIs c94809e2_i a69e096f_o and TS 10.0.10.1/32 === 10.10.101.254/32
Jun 10 14:00:19 ubuntu-test charon: 11[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (76 bytes)
Jun 10 14:00:19 ubuntu-test charon: 11[ENC] parsed INFORMATIONAL request 0 [ ]
Jun 10 14:00:19 ubuntu-test charon: 11[ENC] generating INFORMATIONAL response 0 [ ]
Jun 10 14:00:19 ubuntu-test charon: 11[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)
Jun 10 14:00:24 ubuntu-test charon: 15[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (76 bytes)
Jun 10 14:00:24 ubuntu-test charon: 15[ENC] parsed INFORMATIONAL request 1 [ ]
Jun 10 14:00:24 ubuntu-test charon: 15[ENC] generating INFORMATIONAL response 1 [ ]
Jun 10 14:00:24 ubuntu-test charon: 15[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)

#4 Updated by Tobias Brunner 22 days ago

Yes I can see that there are errors can check it please

...
XfrmInStateProtoError           1738
...
XfrmOutStateModeError           362
...

Yes, that's not good. Seems to indicate a problem with your kernel (in particular because you also see XfrmOutStateModeError, however, at least some outbound packets seem to get processed correctly). What kernel version are you using? What modules are loaded? Any patches?

#5 Updated by TAHER BAHASHWAN 22 days ago

Tobias Brunner wrote:

Yes I can see that there are errors can check it please
[...]

Yes, that's not good. Seems to indicate a problem with your kernel (in particular because you also see XfrmOutStateModeError, however, at least some outbound packets seem to get processed correctly). What kernel version are you using? What modules are loaded? Any patches?

Many thanks for your support, and Here is the requested information, BTW no patches applied this is a new VM installed on Digital Ocean Cloud


root@ubuntu-test:~# uname -r
5.3.0-51-generic
root@ubuntu-s-3vcpu-1gb-sfo2-01:~#
root@ubuntu-s-3vcpu-1gb-sfo2-01:~# cd /sys
root@ubuntu-s-3vcpu-1gb-sfo2-01:/sys# find . |wc -l
71398

root@ubuntu-test:/sys#  lsmod
Module                  Size  Used by
ip6table_filter        16384  0
ip6_tables             32768  1 ip6table_filter
xt_MASQUERADE          20480  1
xt_nat                 16384  0
binfmt_misc            24576  1
ufs                    81920  0
qnx4                   16384  0
hfsplus               110592  0
hfs                    61440  0
minix                  36864  0
ntfs                  106496  0
msdos                  20480  0
jfs                   192512  0
xfs                  1273856  0
cpuid                  16384  0
iptable_filter         16384  1
xt_addrtype            16384  5
xt_policy              16384  12
iptable_mangle         16384  1
xt_multiport           20480  4
xt_NFLOG               16384  24
iptable_raw            16384  1
nfnetlink_queue        24576  0
nfnetlink_log          20480  24
nfnetlink              16384  2 nfnetlink_queue,nfnetlink_log
bluetooth             573440  0
ecdh_generic           16384  1 bluetooth
ecc                    32768  1 ecdh_generic
xt_REDIRECT            20480  0
xt_tcpudp              20480  0
iptable_nat            16384  1
nf_nat                 40960  4 xt_nat,iptable_nat,xt_MASQUERADE,xt_REDIRECT
nf_conntrack          139264  4 nf_nat,xt_nat,xt_MASQUERADE,xt_REDIRECT
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
bpfilter               24576  0
dummy                  16384  0
authenc                16384  4
echainiv               16384  4
xfrm_user              36864  2
xfrm4_tunnel           16384  0
tunnel4                16384  1 xfrm4_tunnel
ipcomp                 16384  0
xfrm_ipcomp            16384  1 ipcomp
esp4                   24576  4
ah4                    20480  0
af_key                 36864  0
xfrm_algo              16384  5 af_key,esp4,xfrm_ipcomp,ah4,xfrm_user
nls_iso8859_1          16384  1
intel_rapl_msr         20480  0
intel_rapl_common      24576  1 intel_rapl_msr
isst_if_common         16384  0
nfit                   65536  0
kvm_intel             245760  0
kvm                   655360  1 kvm_intel
irqbypass              16384  1 kvm
joydev                 28672  0
input_leds             16384  0
serio_raw              20480  0
mac_hid                16384  0
sch_fq_codel           20480  2
ib_iser                49152  0
rdma_cm                61440  1 ib_iser
iw_cm                  49152  1 rdma_cm
ib_cm                  57344  1 rdma_cm
ib_core               299008  4 rdma_cm,iw_cm,ib_iser,ib_cm
iscsi_tcp              24576  0
libiscsi_tcp           28672  1 iscsi_tcp
libiscsi               57344  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   110592  4 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi
ip_tables              32768  4 iptable_filter,iptable_raw,iptable_nat,iptable_mangle
x_tables               40960  14 ip6table_filter,iptable_filter,xt_multiport,xt_NFLOG,xt_tcpudp,xt_addrtype,xt_nat,xt_policy,ip6_tables,iptable_raw,ip_tables,xt_MASQUERADE,iptable_mangle,xt_REDIRECT
autofs4                45056  2
btrfs                1236992  0
zstd_compress         163840  1 btrfs
raid10                 57344  0
raid456               155648  0
async_raid6_recov      24576  1 raid456
async_memcpy           20480  2 raid456,async_raid6_recov
async_pq               24576  2 raid456,async_raid6_recov
async_xor              20480  3 async_pq,raid456,async_raid6_recov
async_tx               20480  5 async_pq,async_memcpy,async_xor,raid456,async_raid6_recov
xor                    24576  2 async_xor,btrfs
raid6_pq              114688  4 async_pq,btrfs,raid456,async_raid6_recov
libcrc32c              16384  5 nf_conntrack,nf_nat,btrfs,xfs,raid456
raid1                  45056  0
raid0                  24576  0
multipath              20480  0
linear                 20480  0
crct10dif_pclmul       16384  1
crc32_pclmul           16384  0
ghash_clmulni_intel    16384  0
aesni_intel           372736  8
qxl                    65536  0
ttm                   102400  1 qxl
drm_kms_helper        180224  3 qxl
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
aes_x86_64             20480  1 aesni_intel
crypto_simd            16384  1 aesni_intel
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm                   491520  4 drm_kms_helper,qxl,ttm
psmouse               151552  0
cryptd                 24576  6 crypto_simd,ghash_clmulni_intel
glue_helper            16384  1 aesni_intel
virtio_net             57344  0
net_failover           20480  1 virtio_net
virtio_scsi            24576  0
virtio_blk             20480  3
failover               16384  1 net_failover
pata_acpi              16384  0
i2c_piix4              28672  0
floppy                 81920  0

#6 Updated by Tobias Brunner 22 days ago

Not sure about 5.3 kernels, but there have definitely been issues with aesni_intel on occasion (might depend on the hardware/virtualization). So you could try to disable that.

#7 Updated by TAHER BAHASHWAN 22 days ago

Tobias Brunner wrote:

Not sure about 5.3 kernels, but there have definitely been issues with aesni_intel on occasion (might depend on the hardware/virtualization). So you could try to disable that.

I am not able to remove it, it seems it was installed with different tool, I will try to blacklist it

root@ubuntu-test:/sys# sudo modprobe -r aesni_intel
modprobe: FATAL: Module aesni_intel is in use.
root@ubuntu-test:/sys# lsmod | grep aesni_intel
aesni_intel           372736  8
aes_x86_64             20480  1 aesni_intel
crypto_simd            16384  1 aesni_intel
glue_helper            16384  1 aesni_intel
root@ubuntu-test:/sys# sudo modprobe -r aes_x86_64
modprobe: FATAL: Module aes_x86_64 is in use.
root@ubuntu-test:/sys# lsmod | grep aes_x86_64
aes_x86_64             20480  1 aesni_intel
root@ubuntu-test:/sys# sudo modprobe -r crypto_simd
modprobe: FATAL: Module crypto_simd is in use.
root@ubuntu-test:/sys# sudo modprobe -r crypto_simd
modprobe: FATAL: Module crypto_simd is in use.
root@ubuntu-test:/sys# sudo modprobe -r aesni_intel
modprobe: FATAL: Module aesni_intel is in use.
root@ubuntu-test:/sys# sudo modprobe -r aesni_intel aes_x86_64  crypto_simd glue_helper
modprobe: FATAL: Module aesni_intel is in use.

#8 Updated by TAHER BAHASHWAN 22 days ago

I disabled aesni_intel module with "rmmod aesni_intel" and I tested again this time no errors but still not working


root@ubuntu-test:~# cat /proc/net/xfrm_stat
XfrmInError                     0
XfrmInBufferError               0
XfrmInHdrError                  0
XfrmInNoStates                  0
XfrmInStateProtoError           0
XfrmInStateModeError            0
XfrmInStateSeqError             0
XfrmInStateExpired              0
XfrmInStateMismatch             0
XfrmInStateInvalid              0
XfrmInTmplMismatch              0
XfrmInNoPols                    0
XfrmInPolBlock                  0
XfrmInPolError                  0
XfrmOutError                    0
XfrmOutBundleGenError           0
XfrmOutBundleCheckError         0
XfrmOutNoStates                 6
XfrmOutStateProtoError          0
XfrmOutStateModeError           0
XfrmOutStateSeqError            0
XfrmOutStateExpired             0
XfrmOutPolBlock                 0
XfrmOutPolDead                  0
XfrmOutPolError                 0
XfrmFwdHdrError                 0
XfrmOutStateInvalid             0
XfrmAcquireError                0


root@ubuntu-test:~# lsmod | grep aesni_intel
root@ubuntu-test:~# lsmod
Module                  Size  Used by
authenc                16384  2
echainiv               16384  2
xfrm_user              36864  2
xfrm4_tunnel           16384  0
tunnel4                16384  1 xfrm4_tunnel
ipcomp                 16384  0
xfrm_ipcomp            16384  1 ipcomp
esp4                   24576  2
ah4                    20480  0
af_key                 36864  0
xfrm_algo              16384  5 af_key,esp4,xfrm_ipcomp,ah4,xfrm_user
ip6table_filter        16384  0
ip6_tables             32768  1 ip6table_filter
xt_MASQUERADE          20480  1
iptable_nat            16384  1
nf_nat                 40960  2 iptable_nat,xt_MASQUERADE
nf_conntrack          139264  2 nf_nat,xt_MASQUERADE
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
xt_multiport           20480  4
iptable_raw            16384  1
iptable_mangle         16384  1
nfnetlink_log          20480  24
nfnetlink              16384  1 nfnetlink_log
xt_NFLOG               16384  24
xt_policy              16384  12
xt_addrtype            16384  5
iptable_filter         16384  1
bpfilter               24576  0
nls_iso8859_1          16384  1
intel_rapl_msr         20480  0
intel_rapl_common      24576  1 intel_rapl_msr
isst_if_common         16384  0
nfit                   65536  0
kvm_intel             245760  0
kvm                   655360  1 kvm_intel
irqbypass              16384  1 kvm
input_leds             16384  0
joydev                 28672  0
serio_raw              20480  0
mac_hid                16384  0
sch_fq_codel           20480  2
ib_iser                49152  0
rdma_cm                61440  1 ib_iser
iw_cm                  49152  1 rdma_cm
ib_cm                  57344  1 rdma_cm
ib_core               299008  4 rdma_cm,iw_cm,ib_iser,ib_cm
iscsi_tcp              24576  0
libiscsi_tcp           28672  1 iscsi_tcp
libiscsi               57344  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   110592  4 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi
ip_tables              32768  4 iptable_filter,iptable_raw,iptable_nat,iptable_mangle
x_tables               40960  11 ip6table_filter,iptable_filter,xt_multiport,xt_NFLOG,xt_addrtype,xt_policy,ip6_tables,iptable_raw,ip_tables,xt_MASQUERADE,iptable_mangle
autofs4                45056  2
btrfs                1236992  0
zstd_compress         163840  1 btrfs
raid10                 57344  0
raid456               155648  0
async_raid6_recov      24576  1 raid456
async_memcpy           20480  2 raid456,async_raid6_recov
async_pq               24576  2 raid456,async_raid6_recov
async_xor              20480  3 async_pq,raid456,async_raid6_recov
async_tx               20480  5 async_pq,async_memcpy,async_xor,raid456,async_raid6_recov
xor                    24576  2 async_xor,btrfs
raid6_pq              114688  4 async_pq,btrfs,raid456,async_raid6_recov
libcrc32c              16384  4 nf_conntrack,nf_nat,btrfs,raid456
raid1                  45056  0
raid0                  24576  0
multipath              20480  0
linear                 20480  0
crct10dif_pclmul       16384  1
crc32_pclmul           16384  0
ghash_clmulni_intel    16384  0
qxl                    65536  0
ttm                   102400  1 qxl
drm_kms_helper        180224  3 qxl
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
aes_x86_64             20480  2
crypto_simd            16384  0
cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
glue_helper            16384  0
virtio_blk             20480  3
sysimgblt              16384  1 drm_kms_helper
virtio_scsi            24576  0
psmouse               151552  0
virtio_net             57344  0
net_failover           20480  1 virtio_net
failover               16384  1 net_failover
fb_sys_fops            16384  1 drm_kms_helper
drm                   491520  4 drm_kms_helper,qxl,ttm
i2c_piix4              28672  0
pata_acpi              16384  0
floppy                 81920  0

Jun 11 01:00:31 ubuntu-test ipsec[1032]: 13[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (284 bytes)
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (204 bytes)
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[IKE] authentication of '95.xx.xx.xx' with pre-shared key successful
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[IKE] IKE_SA IPSec-To-COB[5] established between 167.xx.xx.xx[167.xx.xx.xx]...95.xx.xx.xx[95.xx.xx.xx]
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[IKE] scheduling reauthentication in 28014s
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[IKE] maximum IKE_SA lifetime 28554s
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 16[IKE] CHILD_SA IPSec-To-COB{8} established with SPIs c147f9df_i 8e76117a_o and TS 10.0.10.1/32 === 10.10.101.254/32
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 07[IKE] retransmit 1 of request with message ID 0
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 08[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (76 bytes)
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 08[ENC] parsed INFORMATIONAL request 0 [ ]
Jun 11 01:00:31 ubuntu-test charon: 10[IKE] retransmit 2 of request with message ID 0
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 08[ENC] generating INFORMATIONAL response 0 [ ]
Jun 11 01:00:31 ubuntu-test ipsec[1032]: 08[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)
Jun 11 01:00:33 ubuntu-test charon: 12[NET] received packet: from 95.xx.xx.xx[4500] to 167.xx.xx.xx[4500] (76 bytes)
Jun 11 01:00:33 ubuntu-test charon: 12[ENC] parsed INFORMATIONAL request 1 [ ]
Jun 11 01:00:33 ubuntu-test charon: 12[ENC] generating INFORMATIONAL response 1 [ ]
Jun 11 01:00:33 ubuntu-test charon: 12[NET] sending packet: from 167.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)

#9 Updated by TAHER BAHASHWAN 22 days ago

BTW I did the same test on different CentOS7 machine and got the same issue

[root@CENTOS-TEST-IPSec-SS ~]# uname -r
3.10.0-957.27.2.el7.x86_64

[root@CENTOS-TEST-IPSec-SS ~]# cat /proc/net/xfrm_stat
XfrmInError                     0
XfrmInBufferError               0
XfrmInHdrError                  0
XfrmInNoStates                  0
XfrmInStateProtoError           0
XfrmInStateModeError            0
XfrmInStateSeqError             0
XfrmInStateExpired              0
XfrmInStateMismatch             0
XfrmInStateInvalid              0
XfrmInTmplMismatch              0
XfrmInNoPols                    0
XfrmInPolBlock                  0
XfrmInPolError                  0
XfrmOutError                    0
XfrmOutBundleGenError           0
XfrmOutBundleCheckError         0
XfrmOutNoStates                 3
XfrmOutStateProtoError          0
XfrmOutStateModeError           0
XfrmOutStateSeqError            0
XfrmOutStateExpired             0
XfrmOutPolBlock                 0
XfrmOutPolDead                  0
XfrmOutPolError                 0
XfrmFwdHdrError                 0
XfrmOutStateInvalid             0
[root@CENTOS-TEST-IPSec-SS ~]#


Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 13[KNL] creating acquire job for policy 10.20.0.5/32[udp/nimspooler] === 10.10.101.254/32[udp/blackjack] with reqid {1}
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 13[IKE] initiating IKE_SA IPSec-To-COB[2] to 95.xx.xx.xx
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 13[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 13[NET] sending packet: from 159.xx.xx.xx[500] to 95.xx.xx.xx[500] (336 bytes)
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[NET] received packet: from 95.xx.xx.xx[500] to 159.xx.xx.xx[500] (304 bytes)
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) ]
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[CFG] selected proposal: IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[IKE] remote host is behind NAT
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[IKE] authentication of '159.xx.xx.xx' (myself) with pre-shared key
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[IKE] establishing CHILD_SA IPSec-To-COB{3} reqid 1
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 15[NET] sending packet: from 159.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (300 bytes)
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[NET] received packet: from 95.xx.xx.xx[4500] to 159.xx.xx.xx[4500] (204 bytes)
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[IKE] authentication of '95.xx.xx.xx' with pre-shared key successful
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[IKE] IKE_SA IPSec-To-COB[2] established between 159.xx.xx.xx[159.xx.xx.xx]...95.xx.xx.xx[95.xx.xx.xx]
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[IKE] scheduling reauthentication in 28245s
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[IKE] maximum IKE_SA lifetime 28785s
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[CFG] selected proposal: ESP:AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ
Jun 10 22:35:15 CENTOS-TEST-IPSec-SS charon: 16[IKE] CHILD_SA IPSec-To-COB{3} established with SPIs cc033492_i ed61d7af_o and TS 10.20.0.5/32 === 10.10.101.254/32
Jun 10 22:35:21 CENTOS-TEST-IPSec-SS charon: 08[NET] received packet: from 95.xx.xx.xx[4500] to 159.xx.xx.xx[4500] (76 bytes)
Jun 10 22:35:21 CENTOS-TEST-IPSec-SS charon: 08[ENC] parsed INFORMATIONAL request 0 [ ]
Jun 10 22:35:21 CENTOS-TEST-IPSec-SS charon: 08[ENC] generating INFORMATIONAL response 0 [ ]
Jun 10 22:35:21 CENTOS-TEST-IPSec-SS charon: 08[NET] sending packet: from 159.xx.xx.xx[4500] to 95.xx.xx.xx[4500] (76 bytes)

loaded modules

[root@CENTOS-TEST-IPSec-SS ~]# lsmod
Module                  Size  Used by
drbg                   30186  1
ansi_cprng             12989  0
authenc                17776  3
xfrm6_mode_tunnel      13227  3
xfrm4_mode_tunnel      12605  6
xfrm4_tunnel           12857  0
tunnel4                13252  1 xfrm4_tunnel
ipcomp                 12700  0
xfrm_ipcomp            13413  1 ipcomp
esp4                   17247  3
ah4                    17247  0
af_key                 40225  0
isofs                  39844  0
sb_edac                31940  0
iosf_mbi               15582  0
kvm_intel             188683  0
kvm                   621392  1 kvm_intel
qxl                    59823  1
irqbypass              13503  1 kvm
crc32_pclmul           13133  0
ttm                   114635  1 qxl
ghash_clmulni_intel    13273  0
drm_kms_helper        179394  1 qxl
syscopyarea            12529  1 drm_kms_helper
sysfillrect            12701  1 drm_kms_helper
sysimgblt              12640  1 drm_kms_helper
fb_sys_fops            12703  1 drm_kms_helper
drm                   429744  4 qxl,ttm,drm_kms_helper
aesni_intel           189415  6
ppdev                  17671  0
lrw                    13286  1 aesni_intel
gf128mul               15139  1 lrw
glue_helper            13990  1 aesni_intel
ablk_helper            13597  1 aesni_intel
cryptd                 21190  6 ghash_clmulni_intel,aesni_intel,ablk_helper
pcspkr                 12718  0
virtio_balloon         18015  0
joydev                 17389  0
parport_pc             28205  0
i2c_piix4              22401  0
drm_panel_orientation_quirks    12957  1 drm
parport                46395  2 ppdev,parport_pc
ip_tables              27126  0
xfs                   996949  1
libcrc32c              12644  1 xfs
ata_generic            12923  0
pata_acpi              13053  0
virtio_net             28063  0
virtio_blk             18222  2
virtio_scsi            18463  0
ata_piix               35052  0
libata                243133  3 pata_acpi,ata_generic,ata_piix
crct10dif_pclmul       14307  0
crct10dif_common       12595  1 crct10dif_pclmul
crc32c_intel           22094  1
serio_raw              13434  0
virtio_pci             22985  0
floppy                 69432  0
virtio_ring            22746  5 virtio_blk,virtio_net,virtio_pci,virtio_balloon,virtio_scsi
virtio                 14959  5 virtio_blk,virtio_net,virtio_pci,virtio_balloon,virtio_scsi
sunrpc                353103  1

#10 Updated by TAHER BAHASHWAN 22 days ago

Only this counterXfrmOutNoStates increasing in /proc/net/xfrm_stat

#11 Updated by TAHER BAHASHWAN 22 days ago

I tried it also on different kernel version and different cloud provider (vultr) but still same issue

root@UBUNTU-VULTR-TEST:~# uname -r
4.15.0-101-generic

root@UBUNTU-VULTR-TEST:~# cat /proc/net/xfrm_stat
XfrmInError                     0
XfrmInBufferError               0
XfrmInHdrError                  0
XfrmInNoStates                  0
XfrmInStateProtoError           0
XfrmInStateModeError            0
XfrmInStateSeqError             0
XfrmInStateExpired              0
XfrmInStateMismatch             0
XfrmInStateInvalid              0
XfrmInTmplMismatch              0
XfrmInNoPols                    0
XfrmInPolBlock                  0
XfrmInPolError                  0
XfrmOutError                    0
XfrmOutBundleGenError           0
XfrmOutBundleCheckError         0
XfrmOutNoStates                 6
XfrmOutStateProtoError          0
XfrmOutStateModeError           0
XfrmOutStateSeqError            0
XfrmOutStateExpired             0
XfrmOutPolBlock                 0
XfrmOutPolDead                  0
XfrmOutPolError                 0
XfrmFwdHdrError                 0
XfrmOutStateInvalid             0
XfrmAcquireError                0

#12 Updated by Noel Kuntze 22 days ago

Please provide us the output of `ip address`, `ipsec statusall` and `iptables-save`.

#13 Updated by TAHER BAHASHWAN 22 days ago

Noel Kuntze wrote:

Please provide us the output of `ip address`, `ipsec statusall` and `iptables-save`.

Here it is, this output is from the last ipsec tunnel I created on this (vultr) (root@UBUNTU-VULTR-TEST:~# uname -r
4.15.0-101-generic)


root@UBUNTU-VULTR-TEST:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 56:00:02:d3:23:78 brd ff:ff:ff:ff:ff:ff
    inet 45.77.201.249/23 brd 45.77.201.255 scope global dynamic ens3
       valid_lft 78070sec preferred_lft 78070sec
    inet6 2001:19f0:5:fea:5400:2ff:fed3:2378/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591600sec preferred_lft 604400sec
    inet6 fe80::5400:2ff:fed3:2378/64 scope link
       valid_lft forever preferred_lft forever
3: lo1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ca:4a:e6:3f:1a:b7 brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.5/32 scope global lo1
       valid_lft forever preferred_lft forever
    inet6 fe80::c84a:e6ff:fe3f:1ab7/64 scope link
       valid_lft forever preferred_lft forever


root@UBUNTU-VULTR-TEST:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-101-generic, x86_64):
  uptime: 2 hours, since Jun 10 22:59:30 2020
  malloc: sbrk 1622016, mmap 0, used 762768, free 859248
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 5
  loaded plugins: charon aesni aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default connmark stroke updown eap-mschapv2 xauth-generic counters
Listening IP addresses:
  45.77.201.249
  2001:19f0:5:fea:5400:2ff:fed3:2378
  10.20.0.5
Connections:
IPSec-To-PaloAlto:  45.77.201.249...95.xx.xx.xx  IKEv2, dpddelay=10s
IPSec-To-PaloAlto:   local:  [45.77.201.249] uses pre-shared key authentication
IPSec-To-PaloAlto:   remote: [95.xx.xx.xx] uses pre-shared key authentication
IPSec-To-PaloAlto:   child:  10.20.0.5/32 === 10.10.101.254/32 TUNNEL, dpdaction=restart
Routed Connections:
IPSec-To-PaloAlto{1}:  ROUTED, TUNNEL, reqid 1
IPSec-To-PaloAlto{1}:   10.20.0.5/32 === 10.10.101.254/32
Security Associations (1 up, 0 connecting):
IPSec-To-PaloAlto[2]: ESTABLISHED 2 hours ago, 45.77.201.249[45.77.201.249]...95.xx.xx.xx[95.xx.xx.xx]
IPSec-To-PaloAlto[2]: IKEv2 SPIs: 26187947e8064967_i* fee7ec6da04fafaf_r, pre-shared key reauthentication in 5 hours
IPSec-To-PaloAlto[2]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
IPSec-To-PaloAlto{5}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: ca0ff8dc_i 9bb5c3bb_o
IPSec-To-PaloAlto{5}:  AES_CBC_128/HMAC_SHA1_96/MODP_1024, 0 bytes_i, 0 bytes_o, rekeying in 4 minutes
IPSec-To-PaloAlto{5}:   10.20.0.5/32 === 10.10.101.254/32


root@UBUNTU-VULTR-TEST:~# iptables -L -v -n --line-n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
root@UBUNTU-VULTR-TEST:~#

#14 Updated by Noel Kuntze 21 days ago

Thank you for the output of `ip address` and `ipsec statusall`, I really need the output of `iptables-save` though, not `iptables -L`.

#15 Updated by TAHER BAHASHWAN 21 days ago

Noel Kuntze wrote:

Thank you for the output of `ip address` and `ipsec statusall`, I really need the output of `iptables-save` though, not `iptables -L`.

Sure, here it is


root@UBUNTU-VULTR-TEST:~# iptables-save > /etc/iptables/rules.v4
root@UBUNTU-VULTR-TEST:~# cat /etc/iptables/rules.v4
# Generated by iptables-save v1.6.1 on Thu Jun 11 02:04:25 2020
*filter
:INPUT ACCEPT [1124:94188]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1043:131311]
COMMIT
# Completed on Thu Jun 11 02:04:25 2020

#16 Updated by Noel Kuntze 21 days ago

Thank you. Seems like it's not exactly what I thought it could be. Would you kindly provide the output of `ip -s xfrm policy` and `ip -s xfrm state` immediately after you tried to ping and the response wasn't decapsulated? Thank you.

#17 Updated by TAHER BAHASHWAN 21 days ago

Noel Kuntze wrote:

Thank you. Seems like it's not exactly what I thought it could be. Would you kindly provide the output of `ip -s xfrm policy` and `ip -s xfrm state` immediately after you tried to ping and the response wasn't decapsulated? Thank you.

Sure
Here it is

root@UBUNTU-VULTR-TEST:~# ip -s xfrm policy && ip -s xfrm state
src 10.20.0.5/32 dst 10.10.101.254/32 uid 0
        dir out action allow index 233 priority 367231 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use 2020-06-11 02:25:03
        tmpl src 45.77.201.249 dst 95.xx.xx.xx
                proto esp spi 0x80b62bf2(2159422450) reqid 1(0x00000001) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 10.10.101.254/32 dst 10.20.0.5/32 uid 0
        dir fwd action allow index 226 priority 367231 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use -
        tmpl src 95.xx.xx.xx dst 45.77.201.249
                proto esp spi 0x00000000(0) reqid 1(0x00000001) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 10.10.101.254/32 dst 10.20.0.5/32 uid 0
        dir in action allow index 216 priority 367231 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use -
        tmpl src 95.xx.xx.xx dst 45.77.201.249
                proto esp spi 0x00000000(0) reqid 1(0x00000001) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket in action allow index 211 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:25:03
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket out action allow index 204 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:25:02
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket in action allow index 195 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:25:02
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket out action allow index 188 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:25:02
src ::/0 dst ::/0 uid 0
        socket in action allow index 179 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src ::/0 dst ::/0 uid 0
        socket out action allow index 172 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src ::/0 dst ::/0 uid 0
        socket in action allow index 163 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src ::/0 dst ::/0 uid 0
        socket out action allow index 156 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -

State
===========          
src 45.77.201.249 dst 95.xx.xx.xx
        proto esp spi 0x80b62bf2(2159422450) reqid 1(0x00000001) mode tunnel
        replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
        auth-trunc hmac(sha1) 0xffbe56373a7e8063404d3e3b995f3dd302685ebb (160 bits) 96
        enc cbc(aes) 0xca0fefdeced4dc0dc86af822be2f3e4f (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x1, bitmap 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 2764(sec), hard 3600(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          84(bytes), 1(packets)
          add 2020-06-11 02:25:02 use 2020-06-11 02:25:03
        stats:
          replay-window 0 replay 0 failed 0
src 95.xx.xx.xx dst 45.77.201.249
        proto esp spi 0xce2de55d(3459114333) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        auth-trunc hmac(sha1) 0xe15074fb0f41a72e0b6007bdb82e89ca255d319d (160 bits) 96
        enc cbc(aes) 0x43605b41708b304f09f9952816e1a623 (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 2635(sec), hard 3600(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use -
        stats:
          replay-window 0 replay 0 failed 0
root@UBUNTU-VULTR-TEST:~#

The ping is continues actually,
So also here another output after few minutes while ping is running

root@UBUNTU-VULTR-TEST:~# ip -s xfrm policy && ip -s xfrm state
src 10.20.0.5/32 dst 10.10.101.254/32 uid 0
        dir out action allow index 233 priority 367231 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use 2020-06-11 02:27:19
        tmpl src 45.77.201.249 dst 95.xx.xx.xx
                proto esp spi 0x80b62bf2(2159422450) reqid 1(0x00000001) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 10.10.101.254/32 dst 10.20.0.5/32 uid 0
        dir fwd action allow index 226 priority 367231 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use -
        tmpl src 95.xx.xx.xx dst 45.77.201.249
                proto esp spi 0x00000000(0) reqid 1(0x00000001) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 10.10.101.254/32 dst 10.20.0.5/32 uid 0
        dir in action allow index 216 priority 367231 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use -
        tmpl src 95.xx.xx.xx dst 45.77.201.249
                proto esp spi 0x00000000(0) reqid 1(0x00000001) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket in action allow index 211 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:27:19
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket out action allow index 204 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:27:18
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket in action allow index 195 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:25:02
src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
        socket out action allow index 188 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use 2020-06-11 02:25:02
src ::/0 dst ::/0 uid 0
        socket in action allow index 179 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src ::/0 dst ::/0 uid 0
        socket out action allow index 172 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src ::/0 dst ::/0 uid 0
        socket in action allow index 163 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src ::/0 dst ::/0 uid 0
        socket out action allow index 156 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft 0(bytes), hard 0(bytes)
          limit: soft 0(packets), hard 0(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:24:28 use -
src 45.77.201.249 dst 95.xx.xx.xx
        proto esp spi 0x80b62bf2(2159422450) reqid 1(0x00000001) mode tunnel
        replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
        auth-trunc hmac(sha1) 0xffbe56373a7e8063404d3e3b995f3dd302685ebb (160 bits) 96
        enc cbc(aes) 0xca0fefdeced4dc0dc86af822be2f3e4f (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x86, bitmap 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 2764(sec), hard 3600(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          11256(bytes), 134(packets)
          add 2020-06-11 02:25:02 use 2020-06-11 02:25:03
        stats:
          replay-window 0 replay 0 failed 0
src 95.xx.xx.xx dst 45.77.201.249
        proto esp spi 0xce2de55d(3459114333) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        auth-trunc hmac(sha1) 0xe15074fb0f41a72e0b6007bdb82e89ca255d319d (160 bits) 96
        enc cbc(aes) 0x43605b41708b304f09f9952816e1a623 (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 2635(sec), hard 3600(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2020-06-11 02:25:02 use -
        stats:
          replay-window 0 replay 0 failed 0

#18 Updated by Tobias Brunner 21 days ago

Do you still see inbound ESP packets when you capture traffic (i.e. does the peer receive the request and respond)? If so, are those UDP-encapsulated or plain ESP?

#19 Updated by TAHER BAHASHWAN 21 days ago

Tobias Brunner wrote:

Do you still see inbound ESP packets when you capture traffic (i.e. does the peer receive the request and respond)? If so, are those UDP-encapsulated or plain ESP?

all are plain ESP

BTW
Hi all of a sudden I got only one ping packet successful

root@UBUNTU-VULTR-TEST:~# ping 10.10.101.254
PING 10.10.101.254 (10.10.101.254) 56(84) bytes of data.
64 bytes from 10.10.101.254: icmp_seq=29804 ttl=64 time=164 ms
^C
--- 10.10.101.254 ping statistics ---
41750 packets transmitted, 1 received, 99% packet loss, time 42750389ms
rtt min/avg/max/mdev = 164.516/164.516/164.516/0.000 ms

Here is the packet capture before decryption

No.    Time    DeltaTime    Source    Destination    Protocol    Dst Port    Length    Bytes in Flight    MTU    Info
1    2020-06-11 17:20:55.614595    0.000000    45.77.201.249    95.xx.xx.xx    ISAKMP    500    412        412    IKE_SA_INIT MID=00 Initiator Request
2    2020-06-11 17:20:55.614602    0.000007    95.xx.xx.xx    45.77.201.249    ISAKMP    500    436        436    IKE_SA_INIT MID=00 Responder Response
3    2020-06-11 17:20:55.614603    0.000001    45.77.201.249    95.xx.xx.xx    ISAKMP    4500    396        396    IKE_AUTH MID=01 Initiator Request
4    2020-06-11 17:20:55.614604    0.000001    95.xx.xx.xx    45.77.201.249    ISAKMP    4500    340        340    IKE_AUTH MID=01 Responder Response
5    2020-06-11 17:20:56.638597    1.023993    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=2/512, ttl=64 (no response found!)
6    2020-06-11 17:20:56.638606    0.000009    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=2/512, ttl=64 (no response found!)
7    2020-06-11 17:20:56.638606    0.000000    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
8    2020-06-11 17:20:56.638606    0.000000    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
9    2020-06-11 17:20:57.662760    1.024154    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=3/768, ttl=64 (no response found!)
10    2020-06-11 17:20:57.662770    0.000010    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=3/768, ttl=64 (no response found!)
11    2020-06-11 17:20:57.662771    0.000001    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
12    2020-06-11 17:20:57.662771    0.000000    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
13    2020-06-11 17:20:58.686591    1.023820    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=4/1024, ttl=64 (no response found!)
14    2020-06-11 17:20:58.686600    0.000009    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=4/1024, ttl=64 (no response found!)
15    2020-06-11 17:20:58.686601    0.000001    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
16    2020-06-11 17:20:58.686602    0.000001    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
17    2020-06-11 17:20:59.710571    1.023969    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=5/1280, ttl=64 (no response found!)
18    2020-06-11 17:20:59.710658    0.000087    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=5/1280, ttl=64 (no response found!)
19    2020-06-11 17:20:59.710659    0.000001    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
20    2020-06-11 17:20:59.710659    0.000000    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
21    2020-06-11 17:21:00.734574    1.023915    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=6/1536, ttl=64 (no response found!)
22    2020-06-11 17:21:00.734583    0.000009    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=6/1536, ttl=64 (no response found!)
23    2020-06-11 17:21:00.734583    0.000000    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
24    2020-06-11 17:21:00.734584    0.000001    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
25    2020-06-11 17:21:00.734584    0.000000    95.xx.xx.xx    45.77.201.249    ISAKMP    4500    212        212    INFORMATIONAL MID=00 Responder Request
26    2020-06-11 17:21:00.734585    0.000001    45.77.201.249    95.xx.xx.xx    ISAKMP    4500    156        156    INFORMATIONAL MID=00 Initiator Response
27    2020-06-11 17:21:01.758667    1.024082    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=7/1792, ttl=64 (no response found!)
28    2020-06-11 17:21:01.758677    0.000010    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=7/1792, ttl=64 (no response found!)
29    2020-06-11 17:21:01.758678    0.000001    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
30    2020-06-11 17:21:01.758678    0.000000    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
31    2020-06-11 17:21:02.782564    1.023886    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=8/2048, ttl=64 (no response found!)
32    2020-06-11 17:21:02.782572    0.000008    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=8/2048, ttl=64 (no response found!)
33    2020-06-11 17:21:02.782573    0.000001    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
34    2020-06-11 17:21:02.782574    0.000001    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
35    2020-06-11 17:21:03.806554    1.023980    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=9/2304, ttl=64 (no response found!)
36    2020-06-11 17:21:03.806566    0.000012    10.20.0.5    10.10.101.254    ICMP        132        132    Echo (ping) request  id=0x11ee, seq=9/2304, ttl=64 (no response found!)
37    2020-06-11 17:21:03.806567    0.000001    45.77.201.249    95.xx.xx.xx    ESP    4500    208        208    ESP (SPI=0x8a192794)
38    2020-06-11 17:21:03.806567    0.000000    95.xx.xx.xx    45.77.201.249    ESP    4500    264        264    ESP (SPI=0xc965c3a2)
39    2020-06-11 17:21:06.110594    2.304027    95.xx.xx.xx    45.77.201.249    ISAKMP    4500    212        212    INFORMATIONAL MID=01 Responder Request
40    2020-06-11 17:21:06.110603    0.000009    45.77.201.249    95.xx.xx.xx    ISAKMP    4500    156        156    INFORMATIONAL MID=01 Initiator Response
41    2020-06-11 17:21:07.646513    1.535910    45.77.201.249    95.xx.xx.xx    ISAKMP    4500    156        156    INFORMATIONAL MID=02 Initiator Request
42    2020-06-11 17:21:07.646521    0.000008    95.xx.xx.xx    45.77.201.249    ISAKMP    4500    212        212    INFORMATIONAL MID=02 Responder Response

Here is the packet after decryption for SPI 0xc965c3a2

No.    Time    DeltaTime    Source    Destination    Protocol    Dst Port    Length    Bytes in Flight    MTU    Info
8    2020-06-11 17:20:56.638606    0.000000    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=2/512, ttl=64
12    2020-06-11 17:20:57.662771    0.000000    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=3/768, ttl=64
16    2020-06-11 17:20:58.686602    0.000001    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=4/1024, ttl=64
20    2020-06-11 17:20:59.710659    0.000000    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=5/1280, ttl=64
24    2020-06-11 17:21:00.734584    0.000001    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=6/1536, ttl=64
30    2020-06-11 17:21:01.758678    0.000000    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=7/1792, ttl=64
34    2020-06-11 17:21:02.782574    0.000001    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=8/2048, ttl=64
38    2020-06-11 17:21:03.806567    0.000000    10.10.101.254    10.20.0.5    ICMP    4500    264        264    Echo (ping) reply    id=0x11ee, seq=9/2304, ttl=64

#20 Updated by Noel Kuntze 21 days ago

Does it work between two Linux peers?

#21 Updated by Tobias Brunner 21 days ago

all are plain ESP

That won't work. As you can see in the output of ip xfrm state (and the output of ipsec statusall), your SAs are configured with UDP encapsulation. The Linux kernel can't handle plain ESP packets for such SAs. So make sure the peers agree that there is a NAT situation and both enable UDP encapsulation.

#22 Updated by Noel Kuntze 21 days ago

If they actually are plain ESP, then that output is wrong, because 4500 is evidently the NAT-T port:

No.    Time                          DeltaTime   Source           Destination    Protocol    Dst Port    Length    Bytes in Flight    MTU    Info
7      2020-06-11 17:20:56.638606    0.000000    45.77.201.249    95.xx.xx.xx    ESP         4500        208       208                       ESP (SPI=0x8a192794)

#23 Updated by Tobias Brunner 21 days ago

If they actually are plain ESP, then that output is wrong, because 4500 is evidently the NAT-T port:

You're right. Unlike tcpdump, Wireshark does not explicitly mention the UDP-encapsulation in the Info (or Protocol) field (tcpdump adds "UDP-encap:").

#24 Updated by TAHER BAHASHWAN 21 days ago

Noel Kuntze wrote:

If they actually are plain ESP, then that output is wrong, because 4500 is evidently the NAT-T port:
[...]

That

Tobias Brunner wrote:

all are plain ESP

That won't work. As you can see in the output of ip xfrm state (and the output of ipsec statusall), your SAs are configured with UDP encapsulation. The Linux kernel can't handle plain ESP packets for such SAs. So make sure the peers agree that there is a NAT situation and both enable UDP encapsulation.

Noel Kuntze wrote:

Does it work between two Linux peers?

Yes it is working, my issue now is between Palo Alto & StrongSwan

#25 Updated by TAHER BAHASHWAN 21 days ago

Noel Kuntze wrote:

If they actually are plain ESP, then that output is wrong, because 4500 is evidently the NAT-T port:
[...]

This is what made me not sure how things mixed,

Palo Alto is configured with NAT-T, Palo Alto itself is behind a NATed IP but there is no Port transalation happening front of Palo Alto, just IP to IP translation
So it is strange thing I am also wondering why packets are ESP whereas NAT-T is enabled

#26 Updated by TAHER BAHASHWAN 21 days ago

TAHER BAHASHWAN wrote:

Noel Kuntze wrote:

If they actually are plain ESP, then that output is wrong, because 4500 is evidently the NAT-T port:
[...]

This is what made me not sure how things mixed,

Palo Alto is configured with NAT-T, Palo Alto itself is behind a NATed IP but there is no Port transalation happening front of Palo Alto, just IP to IP translation
So it is strange thing I am also wondering why packets are ESP whereas NAT-T is enabled

If I disable NAT-T on Palo Alto it will work, but what is the justification for this behavior since Palo Alto behind a NAT

#27 Updated by Noel Kuntze 20 days ago

TAHER BAHASHWAN wrote:

If I disable NAT-T on Palo Alto it will work, but what is the justification for this behavior since Palo Alto behind a NAT

So it does work if it's disabled, or you think it will work with NAT-T disabled on the Palo Alto side?

#28 Updated by TAHER BAHASHWAN 20 days ago

Noel Kuntze wrote:

TAHER BAHASHWAN wrote:

If I disable NAT-T on Palo Alto it will work, but what is the justification for this behavior since Palo Alto behind a NAT

So it does work if it's disabled, or you think it will work with NAT-T disabled on the Palo Alto side?

with NAT-Disabled on Palo Alto , it is working fine with StrongSwan, but why is this scenario happening, Palo Alto is behind a NATed IP "yes no port NAT is happening only layer 3 NAT"
So with NAT-T enabled we can see packets received on StrongSwan side and it decrypted but it is not normal ping,
Please look at below results on strongswan after I diabled NAT-T on Palo Alto VW FW.

root@UBUNTU-VULTR-TEST:~# ping 10.10.101.254
PING 10.10.101.254 (10.10.101.254) 56(84) bytes of data.
64 bytes from 10.10.101.254: icmp_seq=2 ttl=64 time=159 ms
64 bytes from 10.10.101.254: icmp_seq=3 ttl=64 time=159 ms
64 bytes from 10.10.101.254: icmp_seq=4 ttl=64 time=159 ms
64 bytes from 10.10.101.254: icmp_seq=5 ttl=64 time=159 ms
^C
--- 10.10.101.254 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4017ms
rtt min/avg/max/mdev = 159.123/159.251/159.416/0.114 ms
root@UBUNTU-VULTR-TEST:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-101-generic, x86_64):
  uptime: 38 hours, since Jun 11 02:24:28 2020
  malloc: sbrk 1712128, mmap 0, used 927888, free 784240
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 7
  loaded plugins: charon aesni aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm attr kernel-netlink resolve socket-default connmark stroke updown eap-mschapv2 xauth-generic counters
Listening IP addresses:
  45.77.201.249
  2001:19f0:5:fea:5400:2ff:fed3:2378
  10.20.0.5
Connections:
IPSec-To-PaloAlto:  45.77.201.249...95.xx.xx.xx  IKEv2, dpddelay=10s
IPSec-To-PaloAlto:   local:  [45.77.201.249] uses pre-shared key authentication
IPSec-To-PaloAlto:   remote: [95.xx.xx.xx] uses pre-shared key authentication
IPSec-To-PaloAlto:   child:  10.20.0.5/32 === 10.10.101.254/32 TUNNEL, dpdaction=restart
Routed Connections:
IPSec-To-PaloAlto{1}:  ROUTED, TUNNEL, reqid 1
IPSec-To-PaloAlto{1}:   10.20.0.5/32 === 10.10.101.254/32
Security Associations (1 up, 0 connecting):
IPSec-To-PaloAlto[14]: ESTABLISHED 16 seconds ago, 45.77.201.249[45.77.201.249]...95.xx.xx.xx[95.xx.xx.xx]
IPSec-To-PaloAlto[14]: IKEv2 SPIs: d32b3d7dab2dda43_i* 5050390f9761f0b9_r, pre-shared key reauthentication in 7 hours
IPSec-To-PaloAlto[14]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
IPSec-To-PaloAlto{58}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: cc9cde5b_i cddb4367_o
IPSec-To-PaloAlto{58}:  AES_CBC_128/HMAC_SHA1_96, 336 bytes_i (4 pkts, 11s ago), 336 bytes_o (4 pkts, 12s ago), rekeying in 45 minutes
IPSec-To-PaloAlto{58}:   10.20.0.5/32 === 10.10.101.254/32
root@UBUNTU-VULTR-TEST:~# ip x s
src 45.77.201.249 dst 95.xx.xx.xx
        proto esp spi 0xcddb4367 reqid 1 mode tunnel
        replay-window 0 flag af-unspec
        auth-trunc hmac(sha1) 0xa48c7c4795836cd3c08b6a66d6b09df8ae74f9e2 96
        enc cbc(aes) 0xc48e11d25ea9e14974621a8c299aaf71
        anti-replay context: seq 0x0, oseq 0x4, bitmap 0x00000000
src 95.xx.xx.xx dst 45.77.201.249
        proto esp spi 0xcc9cde5b reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha1) 0x6b0d8c088de2d2cfd2dd9b4ad8a8b19686c8da88 96
        enc cbc(aes) 0x6505b800d85e2dc963c96ce1f2d87b03
        anti-replay context: seq 0x4, oseq 0x0, bitmap 0x0000000f
root@UBUNTU-VULTR-TEST:~# tail -f /var/log/syslog
Jun 12 17:19:39 UBUNTU-VULTR-TEST charon: 06[ENC] generating INFORMATIONAL response 4 [ ]
Jun 12 17:19:39 UBUNTU-VULTR-TEST charon: 06[NET] sending packet: from 45.77.201.249[500] to 95.xx.xx.xx[500] (76 bytes)
Jun 12 17:19:44 UBUNTU-VULTR-TEST charon: 16[NET] received packet: from 95.xx.xx.xx[500] to 45.77.201.249[500] (76 bytes)
Jun 12 17:19:44 UBUNTU-VULTR-TEST charon: 16[ENC] parsed INFORMATIONAL request 5 [ ]
Jun 12 17:19:44 UBUNTU-VULTR-TEST charon: 16[ENC] generating INFORMATIONAL response 5 [ ]
Jun 12 17:19:44 UBUNTU-VULTR-TEST charon: 16[NET] sending packet: from 45.77.201.249[500] to 95.xx.xx.xx[500] (76 bytes)
Jun 12 17:19:49 UBUNTU-VULTR-TEST charon: 13[NET] received packet: from 95.xx.xx.xx[500] to 45.77.201.249[500] (76 bytes)
Jun 12 17:19:49 UBUNTU-VULTR-TEST charon: 13[ENC] parsed INFORMATIONAL request 6 [ ]
Jun 12 17:19:49 UBUNTU-VULTR-TEST charon: 13[ENC] generating INFORMATIONAL response 6 [ ]
Jun 12 17:19:49 UBUNTU-VULTR-TEST charon: 13[NET] sending packet: from 45.77.201.249[500] to 95.xx.xx.xx[500] (76 bytes)
^C

root@UBUNTU-VULTR-TEST:~# tcpdump -s 0 -n -i nflog:5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on nflog:5, link-type NFLOG (Linux netfilter log messages), capture size 262144 bytes
17:44:53.374669 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 14, length 64
17:44:53.377153 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 14, length 64
17:44:53.377162 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x12), length 132
17:44:53.377200 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x12), length 132
17:44:53.377206 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 14, length 64
17:44:53.377211 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 14, length 64
17:44:53.377216 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 15, length 64
17:44:53.377220 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 15, length 64
17:44:53.377225 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x13), length 132
17:44:54.526545 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x13), length 132
17:44:54.526596 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 15, length 64
17:44:54.526606 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 15, length 64
17:44:54.526612 IP 95.xx.xx.xx.500 > 45.77.201.249.500: isakmp: child_sa  inf2
17:44:54.526742 IP 45.77.201.249.500 > 95.xx.xx.xx.500: isakmp: child_sa  inf2[IR]
17:44:54.526754 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 16, length 64
17:44:54.526764 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 16, length 64
17:44:54.526771 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x14), length 132
17:44:54.526777 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x14), length 132
17:44:54.526782 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 16, length 64
17:44:54.526788 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 16, length 64
17:44:56.382533 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 17, length 64
17:44:56.382812 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 17, length 64
17:44:56.382959 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x15), length 132
17:44:56.383138 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x15), length 132
17:44:56.383276 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 17, length 64
17:44:56.383406 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 17, length 64
17:44:56.383578 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 18, length 64
17:44:56.383721 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 18, length 64
17:44:56.383864 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x16), length 132
17:44:57.534549 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x16), length 132
17:44:57.534834 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 18, length 64
17:44:57.534945 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 18, length 64
17:44:57.535091 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 19, length 64
17:44:57.535214 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 19, length 64
17:44:57.535305 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x17), length 132
17:44:57.535394 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x17), length 132
17:44:57.535649 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 19, length 64
17:44:57.535792 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 19, length 64
17:44:59.358978 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 20, length 64
17:44:59.359392 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 20, length 64
17:44:59.359654 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x18), length 132
17:44:59.359929 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x18), length 132
17:44:59.360136 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 20, length 64
17:44:59.360306 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 20, length 64
17:44:59.360448 IP 95.xx.xx.xx.500 > 45.77.201.249.500: isakmp: child_sa  inf2
17:44:59.360626 IP 45.77.201.249.500 > 95.xx.xx.xx.500: isakmp: child_sa  inf2[IR]
17:45:00.382589 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 21, length 64
17:45:00.383041 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 21, length 64
17:45:00.383356 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x19), length 132
17:45:00.383619 IP 95.xx.xx.xx > 45.77.201.249: ESP(spi=0xcc9cde5b,seq=0x19), length 132
17:45:00.383666 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 21, length 64
17:45:00.383695 IP 10.10.101.254 > 10.20.0.5: ICMP echo reply, id 12052, seq 21, length 64
17:45:00.383721 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 22, length 64
17:45:00.383899 IP 10.20.0.5 > 10.10.101.254: ICMP echo request, id 12052, seq 22, length 64
17:45:00.383935 IP 45.77.201.249 > 95.xx.xx.xx: ESP(spi=0xcddb4367,seq=0x1a), length 132

#29 Updated by Tobias Brunner 17 days ago

with NAT-Disabled on Palo Alto , it is working fine with StrongSwan, but why is this scenario happening, Palo Alto is behind a NATed IP "yes no port NAT is happening only layer 3 NAT"

IPsec works fine over this type of NAT as there is no confusion where to forward ESP packets for a specific IP to (and traffic selectors are also not a problem with your config). With NAT-T disabled, neither side will detect a NAT and UDP-encapsulation is not enabled.

Why it fails with NAT-T enabled, I don't know. From the previous captures it looks like UDP-encapsulation is enabled on both host, which would otherwise be problematic. Why the Linux box fails to process these packets, while Wireshark can decrypt them successfully seems strange.

So with NAT-T enabled we can see packets received on StrongSwan side and it decrypted but it is not normal ping,

What do you mean with "is not normal ping"?

#30 Updated by TAHER BAHASHWAN 14 days ago

Tobias Brunner wrote:

with NAT-Disabled on Palo Alto , it is working fine with StrongSwan, but why is this scenario happening, Palo Alto is behind a NATed IP "yes no port NAT is happening only layer 3 NAT"

IPsec works fine over this type of NAT as there is no confusion where to forward ESP packets for a specific IP to (and traffic selectors are also not a problem with your config). With NAT-T disabled, neither side will detect a NAT and UDP-encapsulation is not enabled.

Why it fails with NAT-T enabled, I don't know. From the previous captures it looks like UDP-encapsulation is enabled on both host, which would otherwise be problematic. Why the Linux box fails to process these packets, while Wireshark can decrypt them successfully seems strange.

So with NAT-T enabled we can see packets received on StrongSwan side and it decrypted but it is not normal ping,

What do you mean with "is not normal ping"?

Thanks Tobias from above comments, I have these few question, I would appreciate to get your feedback on them:
1- How to disable or enable "UDP-encapsulation" on StrongSwang?
2- Is NAT-T required if no port translation happening but only IP NAT is happening?

What do you mean with "is not normal ping"?

Here I mean that we can see ping packets in tcpdump but on terminal ping is hung and it does not give any result.
Just like this below

 ping 10.10.101.254
PING 10.10.101.254 (10.10.101.254) 56(84) bytes of data.

#31 Updated by Tobias Brunner 13 days ago

1- How to disable or enable "UDP-encapsulation" on StrongSwang?

You can only force UDP-encapsulation via forceencaps option (you enabled that in your config above). It currently can't be disabled (i.e. strongSwan always sends NAT-D payloads and uses UDP-encapsulation if a NAT is detected). See NatTraversal.

2- Is NAT-T required if no port translation happening but only IP NAT is happening?

No, but only if have a 1:1 mapping between public and private IPs (if the server knows this, it could respond with a NAT-D payload that matches the public IP, which would hide the NAT from the client).

What do you mean with "is not normal ping"?

Here I mean that we can see ping packets in tcpdump but on terminal ping is hung and it does not give any result.

Could you attach a pcap file and the SPIs, algorithms, keys etc. (i.e. output of ip -s xfrm state) of such a session?

Also available in: Atom PDF