Part 5 established the tunnel and ended with a question - why does one of the traffic flows fail?
Setting up a Site-to-Site VPN is one thing; understanding exactly how a packet traverses from an Azure VNet down to a local KVM bridge is another. This part takes a deeper look at the underlying networking layers, looking specifically at how XFRM policies, routing decisions, and iptables chains interact to manage hybrid traffic flow.
TL;DR
Most traffic flows between Azure and an on-premises KVM network work automatically once the S2S tunnel is up. The only flow that doesn’t work automatically is Azure VM → on-premises KVM VM. It fails because libvirt’s default iptables rules block forwarded traffic from the Azure address range into libvirt bridge (virbr0). The fix is a single iptables rule — but understanding why that rule is needed is what this article is really about.
Recap - There are 3 scenarios of bi-directional traffic flow in our setup between -
- Azure Virtual Machines and On-premises Virtual Machines
- Azure Virtual Machines and On-premises VPN Gateway
- On-premises VPN Gateway and the On-premises Virtual Machines
All scenarios except Azure virtual machine to on-premises virtual machine will pass without any additional configuration.
Let’s see why each of the below scenarios play out. Here is a quick reference of the IP addresses we will be using in this discussion.
192.168.122.0/24— on-premises KVM Virtual Network (libvirt bridge
virbr0)192.168.1.106 — on-premises VPN gateway (StrongSwan)
192.168.1.0/24 — on-premises Wi-Fi LAN
61.69.136.49 — on-premises public IP
20.219.67.227 — Azure VPN Gateway Public IP
10.66.0.0/16 — Azure VNet
10.66.0.0/24 — Azure GatewaySubnet
10.66.5.0/24 — Azure WorkloadSubnet
Throughout this article, Mint refers to the Linux Mint host (192.168.1.106) running StrongSwan as the on-premises VPN gateway and DC1 refers to one of the KVM virtual machines in the on-premises virtual network hosted in the Linux Mint host using libvirt.
Scenario 1 - Mint [192.168.1.106] → Azure VM [10.66.5.4]
Consider the case where the on-premises VPN Gateway (Mint) is sending data packets to Azure Virtual Machine behind the Azure VPN Gateway using its private IP.
Step 1 - Determine the next hop
Mint prepares the packet with destination IP as 10.66.5.4, and consults its route table for the next hop. Since the destination IP is outside any specific local route, it matches the default route and the next hop is established as the Wi-Fi gateway 192.168.1.1.
Step 2 - Evaluate XFRM policy
But before the packet leaves Mint, the kernel checks the (src, dst, protocol) tuple against installed XFRM out policies. A match is found for src=192.168.1.0/24 dst=10.66.0.0/16. The match is found because StrongSwan installed matching XFRM policies derived from the traffic selectors configured in ipsec.conf
You can run the below command and inspect the output of XFRM policy:
```
sudo ip xfrm policy show
```
```
src 192.168.122.0/24 dst 10.66.0.0/16
dir out priority 379519
tmpl src 192.168.1.106 dst 20.219.67.227
proto esp spi 0x82a62bd6 reqid 1 mode tunnel`
src 192.168.1.0/24 dst 10.66.0.0/16
dir out priority 379519
tmpl src 192.168.1.106 dst 20.219.67.227
proto esp spi 0x82a62bd6 reqid 1 mode tunnel
src 10.66.0.0/16 dst 192.168.122.0/24
dir fwd priority 379519
tmpl src 20.219.67.227 dst 192.168.1.106
proto esp reqid 1 mode tunnel
src 10.66.0.0/16 dst 192.168.122.0/24
dir in priority 379519
tmpl src 20.219.67.227 dst 192.168.1.106
proto esp reqid 1 mode tunnel
src 10.66.0.0/16 dst 192.168.1.0/24
dir fwd priority 379519
tmpl src 20.219.67.227 dst 192.168.1.106
proto esp reqid 1 mode tunnel
src 10.66.0.0/16 dst 192.168.1.0/24
dir in priority 379519
tmpl src 20.219.67.227 dst 192.168.1.106
proto esp reqid 1 mode tunnel
```
Note - routing decides where the packet goes, XFRM decides how it leaves. If no out policy matches, the packet leaves as plaintext.
Step 3 - Encrypt the packet
Next, the packet is encrypted with ciphers as per the metadata got by looking up the SPI attributes in XFRM state linked to the XFRM policy that just got evaluated. After the kernel module encrypts the packet the original IP packet becomes the ESP payload, a new IP header is prepended with src=192.168.1.106 dst=20.219.67.227 (Azure VPN Gateway IP address).
Step 4 - Send the packet to Azure VPN Gateway
At this point the re-wrapped packet is encapsulated in UDP Port 4500 because NAT-Traversal (NAT-T) is automatically engaged (since our StrongSwan gateway sits behind a home Wi-Fi NAT router). Routing runs again on this encapsulated ESP packet. With the src as 192.168.1.106 (Mint’s IP) and dst as Azure VPN Gateway’s public IP, it matches to the default route via 192.168.1.1 (Wi-Fi router). The Wi-Fi router NATs the source to the public IP before forwarding to the internet, towards the Azure VPN Gateway public IP.
Step 5 - Azure VPN Gateway receives the packet
The Azure VPN Gateway receives the UDP:4500 packet, identifies it as belonging to an established S2S tunnel via the SPI in the ESP header, which maps to an established Security Association negotiated during IKE handshake. It then decrypts the ESP payload, and strips the outer headers to reveal the original inner packet with dst=10.66.5.4
Step 6 - Azure VM receives the packet
The Azure VPN Gateway decapsulates the packet and hands it back to the VNet’s routing layer, which uses system routes to deliver it to 10.66.5.4.
Scenario 2 - Azure VM [10.66.5.4] → Mint [192.168.1.106]
The Azure VM constructs the packet with src=10.66.5.4(a VM in Azure workload VNet) and dst=192.168.1.106 (Mint) and sends it to the NIC of the VM.
Step 1 - Calculate Effective Route
Azure Networking fabric picks up the outgoing packet and uses the Effective Route from that particular NIC. Based on this, the packet is sent to the next hop. In this case the effective route looks like this:
Source | State | Address Prefixes | Next Hop Type | NextHop IP
Default | Active | 10.66.0.0/16 | Virtual network
Virtual network gateway | Active | 192.168.122.0/24 | Virtual network gateway | 10.66.0.5
Virtual network gateway | Active | 192.168.1.0/24 | Virtual network gateway | 10.66.0.5
Default | Active | 0.0.0.0/0 | Internet
Since the packet is for 192.168.1.0/24, the next hop is Virtual Network Gateway (10.66.0.5)
Step 2 - Packet encryption
Once the packet reaches the VPN Gateway, it matches the destination 192.168.1.0/24 against its SA database, identifies the tunnel endpoint as 61.69.136.49 (home Wi-Fi’s public IP), encrypts the packet and stamps dst=61.69.136.49 src=20.219.67.227 and sends the packet to on-premises public IP.
Step 3 - Packet received by on-premises
Once the packet reaches the on-premises Wi-Fi router, its NAT entry is consulted and will have a NAT-T entry for UDP:4500 between the Mint (machine running StrongSwan).
The NAT-T (NAT Traversal) entry is created when StrongSwan initiates the tunnel - the outbound UDP:4500 packet from
192.168.1.106creates a NAT binding in the router’s conntrack table. (this entry is created during tunnel establishment, not on first data packet) When Azure replies on the same port, the router consults its conntrack entry and translates the destination back to192.168.1.106. It’s standard UDP NAT pinhole behavior, not anything IPSec-specific.
This NAT pinhole behavior assumes the tunnel was initiated from on-premises. If Azure initiates, the router has no conntrack entry for the inbound IKE packet and will drop it - unless a static port-forward rule exists for UDP:500 and UDP:4500 pointing to the StrongSwan host. Once the handshake completes via the port-forward, the conntrack entry is created and the tunnel behaves identically from that point on. This is why Azure-initiated connectivity requires the explicit port-forward rule that on-premises-initiated tunnels don’t need.
The Wi-Fi router sees only the outer UDP:4500 wrapper — the ESP payload is opaque to it. It performs NAT translation here (dst=61.69.136.49 → 192.168.1.106) based on conntrack entry and forwards it on to Mint (192.168.1.106)
Step 4 - Packet arrives at Mint
The tunnel termination happens here. When Mint (192.168.1.106) receives the packet, the XFRM in policy fires here because dst=192.168.1.106 is Mint itself rather than a forwarded destination, the kernel identifies the Security Association using the SPI in the ESP header, decrypts the packet and delivers it locally.
Scenario 3 - Mint [192.168.1.106] to DC1 [192.168.122.10]
This flow needs the packet from on-premises virtual network gateway machine to cross over to the virtual network that it hosts.
Two different flows are possible here. In both cases Mint consults its routing table and finds 192.168.122.0/24 dev virbr0, so traffic is sent through the libvirt bridge.
Subflow A - Non encrypted communication outside of the tunnel between Mint and the libvirt virtual network
In this case, Mint originates a packet. It goes through the OUTPUT chain, not the FORWARD chain. libvirt installs default rules permitting all host-originated traffic, so the packet is accepted and delivered to DC1 via libvirt bridge without any additional configuration.
Subflow B - ESP Packet arriving at Mint, heading to virtual network as part of S2S traffic
In this case, an encrypted ESP packet arrives on the physical interface (wlp58s0). Mint’s XFRM subsystem decrypts it and re-injects the inner packet back into the routing stack with dst=192.168.122.10. Since the destination (192.168.122.10) is on a different interface (libvirt bridge - virbr0), Mint’s kernel must forward the packet from the physical network to the libvirt bridge (virbr0). To enable this forwarding, ip_forward should be set to 1. Also, for traffic to be forwarded to DC1, the libvirt forwarding rule needs to be modified too.
Note - The actual transit from Azure into this virtual network will fail at the moment. We will break down exactly why it fails and how to fix it in Scenario 5.
From this point onwards, it is an unencrypted packet and there is no XFRM check at libvirt bridge. The kernel forwards the packet out through the libvirt bridge (virbr0) towards the destination.
Scenario 4 - DC1 [192.168.122.10] → Mint [192.168.1.106]
DC1 prepares a packet and consults its route table. From the screenshot, DC1’s default gateway is 192.168.122.1 for all traffic outside 192.168.122.0/24 which is Mint’s libvirt bridge (virbr0).
DC1 is completely tunnel-unaware. It simply forwards all non-local traffic to the libvirt bridge (virbr0) gateway 192.168.122.1 and Mint decides what happens next. Just like the previous scenario, there are two subflows
Subflow A - DC1 [192.168.122.10]→ Mint itself [192.168.1.106]
The packet arrives at Mint’s libvirt bridge (virbr0). Destination 192.168.1.106 is Mint itself so the kernel delivers it to the local stack. No XFRM check, encryption, or tunnel is required for this flow. The packet arrives as plain IP/ICMP with no ESP header, so the inbound XFRM hook finds nothing to process and passes it through.
Subflow B - DC1 [192.168.122.10] → Azure VM [10.66.5.4] via Mint [192.168.1.106]
The packet arrives at Mint’s libvirt bridge (virbr0). Since 10.66.5.4 is not a local destination, Mint’s kernel processes the packet for forwarding. Mint’s kernel checks the (src, dst, protocol) tuple against the installed XFRM out policies and finds that destination 10.66.5.4 falls within 10.66.0.0/16. The routing decision for the original packet has already been made, so the matching out policy causes the packet to be encrypted and encapsulated in ESP with outer src=192.168.1.106 and dst=20.219.67.227. Routing is then performed again on this outer packet, whose destination matches the default route via the Wi-Fi router 192.168.1.1.
Upon receiving the packet, the Wi-Fi router NATs src to its public IP 61.69.136.49 and forwards to Azure VPN Gateway. DC1 has no knowledge of any of this. It sent a packet to its default gateway and Mint’s XFRM policy made the tunnelling decision transparently.
Scenario 5 - Azure VM [10.66.5.4] → DC1 [192.168.122.10] -
The scenario that failed in Part 5
Step 1 - Packet creation and forwarding to VPN Gateway
Azure VM initiates a ping. An ICMP packet is created, and the Azure fabric consults the effective routes for the VM’s NIC. The gateway-injected route for 192.168.122.0/24 is a more specific match than the default 0.0.0.0/0 → Internet system route, so the next hop is resolved as VirtualNetworkGateway via longest prefix match (refer to effective routes below)
The routing mechanism is different in Azure. There are three types of routes - System, UDR and custom routes. System routes come in two forms: default routes created automatically when a virtual network is created, and optional routes added when specific capabilities are enabled.
In our case, adding the VPN Gateway caused optional system routes for the on-premises prefixes (192.168.1.0/24 and 192.168.122.0/24) to be injected into the route table of every subnet in the VNet. These appear in the effective routes table with source Virtual network gateway, distinguishing them from the routes created at VNet creation time which show source Default. Sample route table for a VM in my WorkloadSubnet
Source | State | Address Prefixes | Next Hop Type | NextHop IP
Default | Active | 10.66.0.0/16 | Virtual network
Virtual network gateway | Active | 192.168.122.0/24 | Virtual network gateway | 10.66.0.5
Virtual network gateway | Active | 192.168.1.0/24 | Virtual network gateway | 10.66.0.5
Default | Active | 0.0.0.0/0 | Internet
Look at the above table and you will see that for a packet with address prefix 192.168.122.0/24 the next hop is Virtual Network Gateway. Thus the packet is forwarded accordingly.
Step 2 - Encryption and forwarding
Upon receiving the packet, VPN Gateway encrypts and adds an outer layer to the packet with destination IP as the public IP of the on-premises network and source IP as its own public IP.
Step 3 - Packet arrives on-premises
The packet reaches the on-premises Wi-Fi router, where public-to-private IP NAT occurs. The Wi-Fi router checks its conntrack table and sees an entry to translate on-premises public IP to on-premises VPN Gateway IP - Mint [192.168.1.106]. The packet is forwarded to Mint.
Step 4 - Packet arrives at Mint
Once the packet reaches Mint, the kernel identifies the Security Association using the SPI in the ESP header and decrypts the packet. The inner packet is then re-injected into the networking stack and, because the destination lies on the KVM virtual network, the kernel attempts to forward it out through the libvirt bridge (virbr0).
This final leg will break because the default rules within the LIBVIRT_FWI chain block unsolicited traffic between the Azure virtual network range and libvirt bridge (virbr0) network range.
There is a subtle point that needs to be understood. Since DC1 lives within a virtualized network inside a host that is part of the Wi-Fi LAN, delivering the packet requires crossing an internal routing and firewall boundary.
Fix:
sudo iptables -I LIBVIRT_FWI 1 \
-s 10.66.0.0/16 \
-d 192.168.122.0/24 \
-o virbr0 \
-j ACCEPT
The above command inserts a rule into the LIBVIRT_FWI chain stating if a packet from 10.66.0.0/16 is heading to 192.168.122.0/24 and is being forwarded out of libvirt bridge (virbr0), accept it - and evaluate this before the existing REJECT rule gets a chance to fire.
Note:
- -I is critical in the above command - iptables evaluates rules top to bottom and stops at the first match. LIBVIRT_FWI already has a REJECT rule. If you appended (-A), your ACCEPT would sit after the REJECT and never fire. Inserting at position 1 puts your ACCEPT rule before the REJECT, so matching traffic is accepted before it ever reaches the REJECT.
- The iptables rule added above is non-persistent. libvirt rewrites the LIBVIRT_FWI chain on every restart, which will remove the manually inserted rule. Dealing with permanent persistence is not addressed in this lab setup.
- IMPORTANT - this fix only applies once ip_forward is enabled — without it the kernel drops the packet before iptables evaluates.
Step 5 - Packet reaches libvirt bridge (virbr0)
Once the above fix is in place, the packet is allowed to be forwarded to the on-premises virtual network.
At this stage the packet is plain IP traffic again and the kernel forwards it out through libvirt bridge (virbr0) towards the destination. Since the destination is within the on-premises virtual network, the packet is forwarded successfully.
Scenario 6 - DC1 [192.168.122.10] → Azure VM [10.66.5.4]
This is a combination of the flows already described in scenario 4 (Subflow B) and Scenario 1. DC1 forwards to Mint’s default gateway, Mint’s XFRM out policy matches, encrypts, and the packet travels the same path as Scenario 1.
So, that’s the entire flow with a reasonable amount of detail.
A couple of points worth calling out
- The ip_forward and LIBVIRT_FWI steps in this lab are specific to this topology — a KVM virtual network sitting behind a host on a home Wi-Fi LAN. A dedicated VPN appliance at the network edge would handle forwarding natively.
- Even if the tunnel has been up for days, a new VM pinging another VM on the other side is still a new connection. The tunnel being up doesn’t mean all traffic within it is pre-authorized end to end.
Summary
We have analyzed the flow of packets between Azure and the on-premises libvirt virtual machines, covering how packets are encrypted and decrypted, XFRM policy evaluation, forwarding rules, and iptables.
What we have at the moment is Layer 3 connectivity. To make our setup production-ready and suitable for actual workloads, we will need a mechanism that is easier to work with than IP addresses and that abstracts us from Layer 3 changes — DNS.
Up next.
Appendix
What is XFRM and how is it triggered
XFRM is the Linux kernel’s IPsec framework. It manages two distinct things: Security Associations (the actual encryption keys and algorithm parameters negotiated during IKE) and Security Policies (rules that determine which traffic gets encrypted and how). When StrongSwan establishes the tunnel, it installs both into the kernel via the XFRM API. From that point, the kernel handles all encryption and decryption autonomously - StrongSwan is not involved per packet.
XFRM policies operate in three directions, each applying at a different point in the kernel’s packet processing path:
- out - applies after the routing decision, before the packet leaves the interface. The kernel evaluates the (src, dst, protocol) tuple against installed out policies. If a match is found, the packet is handed to XFRM for encryption and ESP encapsulation before it goes on the wire. Traffic from
192.168.1.0/24or192.168.122.0/24destined for10.66.0.0/16hits an out policy - the kernel encrypts it and wraps it in ESP before it leaves via the Wi-Fi interface.- in - applies to packets arriving at the host and destined for the host itself. When an ESP packet arrives, the kernel identifies it as encrypted traffic, looks up the matching Security Association via the SPI in the ESP header, decrypts the payload, and re-injects the inner packet. The in policy then confirms the decrypted packet is authorised. If the inner packet’s destination is
192.168.1.106(the StrongSwan host itself), the in policy matches and the packet is consumed locally.- fwd - applies to packets arriving at the host but destined for another host - i.e., the machine is acting as a router. After decryption and re-injection, if the inner packet’s destination is
192.168.122.10(DC1 on the libvirt network), the kernel recognizes that it must forward the packet onward to libvirt bridge (virbr0). The fwd policy matches here. Without a fwd policy, the kernel drops the decrypted packet even though decryption succeeded - it has no instruction to forward it.
References
https://docs.strongswan.org/docs/5.9/features/natTraversal.html https://man7.org/linux/man-pages/man8/ip-xfrm.8.html https://docs.strongswan.org/docs/latest/howtos/ipsecProtocol.html https://libvirt.org/docs.html https://www.redhat.com/en/blog/iptables