In discussions with a friend recently a question came up about the best place in netfilter / legacy iptables to filter out spurious traffic to the local addresses on a router. To be specific the use case is a Linux device acting as a router, with certain services running locally on it we want to use iptables to prevent customers connecting to (BGP, SSH, SNMP or whatever).

Question

My friend maintained it was always preferable to have the rules to drop such traffic in the the prerouting chain of the raw table, ensuring it happens prior to the routing lookup. Especially in the case of a router with a full routing table of ~1 million ipv4 destinations this is a very expensive operation.

It made perfect sense to me that it was more efficient to drop packets before doing the routing lookup, however I wondered - in the standard case where the vast majority of packets coming in get forwarded through the router - does it make sense to evaluate all of them against the DROP rule, given the vast majority will pass on to the routing lookup anyway?

The question ultimately comes down to is it more costly to do a route table lookup for the small number of packets we eventually drop, or do a rule evaluation for the large number of packets we end up forwarding? Consider the two scenarios:

Rule in RAW/prerouting:

• All packets get evaluated on ingress by the DROP rule
• Dropped packets have not gone through routing lookup

Rule in FILTER/input

• All packets go through the route lookup stage
• Only those for the local system get evaluated by the DROP rule

Test Setup

I set up 3 VMs as shown below. They were run on Linux with qemu, with the connections between them made with virtio devices bound to a separate hypervisor bridge device for each link:

The middle VM had over 900,000 routes added to a selection of approx. 50 next-hops, all of which were configured on the receiver node. This was to ensure there was a very large routing table the system needed to look up as packets went through it. I set the next-hops to a bunch of about 50 IPs I configured on the "traffic dest" node so there were lots of next-hops groups too, for example:

root@iptrules:~# ip route show | head -6
default via 192.168.122.1 dev enp1s0 
1.0.0.0/24 via 203.0.113.8 dev enp8s0 
1.0.4.0/22 via 203.0.113.22 dev enp8s0 
1.0.5.0/24 via 203.0.113.5 dev enp8s0 
1.0.16.0/24 via 203.0.113.16 dev enp8s0 
1.0.32.0/24 via 203.0.113.1 dev enp8s0 

root@iptrules:~# ip route show | wc -l 
925525

root@iptrules:~# ip neigh show dev enp8s0 | head -4
203.0.113.32 lladdr 52:54:00:5f:02:5f REACHABLE
203.0.113.28 lladdr 52:54:00:5f:02:5f STALE
203.0.113.10 lladdr 52:54:00:5f:02:5f REACHABLE
203.0.113.6 lladdr 52:54:00:5f:02:5f REACHABLE

Test 1: Raw vs Filter table with packets only sent to the router IPs

This comparison does a like-for-like test on using raw vs. filter table. All the packets we send will get dropped, so the question is if it is more efficient to drop them in RAW, before routing lookup, than in FILTER, afterwards. It quite obviously should be much more efficient to drop in RAW.

In both tests approximately 280k 128-byte UDP packets were sent per second for 10 minutes. iperf was used to generate the UDP streams on the traffic source VM, to the IP configured on enp8s0 of the router VM. iperf command was as follows:

iperf -l 128 -i 10 -t 600 -c 203.0.113.1 -u -b 300M

The iptables rules were as follows, this shows the state after the test including packet counters:

    root@iptrules:~# iptables -L -v --line -n -t raw
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1     165M   26G CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK
    2     165M   26G DROP       udp  --  *      *       198.51.100.1         0.0.0.0/0            udp dpt:5001 ADDRTYPE match dst-type LOCAL

    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1      753  402K CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK

I then repeated the test with the iptables rules changed to drop what I wanted in the filter/input chain, the counters look similar in terms of total packets and drops, which is good:

    root@iptrules:~# iptables -L -v --line -n -t raw 
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1     165M   26G CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK

    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1      446  205K CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK

    root@iptrules:~# iptables -L -v --line -n -t filter 
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1     165M   26G DROP       udp  --  *      *       198.51.100.1         0.0.0.0/0            udp dpt:5001

    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         

    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination   

Looking at CPU usage during both tests we see the following:

As expected dropping in RAW, before the route lookup, used less CPU, hovered around 10% average as opposed to 12% average when the routing looking was also done. Which, removing 1.5% background cpu, is about 15% more CPU used when using the INPUT table.

Test 2: Raw vs Filter table with most packets sent to the 'Traffic Dest' VM

This test trys to get some insight on the original hypothesis. In this case I generated 4 iperf streams of traffic towards some of the IPs on the 'traffic dest' machine. These streams will be forwarded through the router. In parallel with those I also sent a lower number of packets to the router IP (similar to test 1 but lower rate). The idea was to test the relative impact of doing the rule evaluation on all packets (when rule is in raw), versus skipping that but having to do a routing lookup on everything, including what we're going to drop.

iptables was set up the same way, counters shown here for test with rule in raw:

    root@iptrules:~# iptables -L -v --line -n -t raw
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1     155M   84G CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK
    2    58604 9142K DROP       udp  --  *      *       198.51.100.1         0.0.0.0/0            udp dpt:5001 ADDRTYPE match dst-type LOCAL

    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1      386  185K CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK

We had similar counters when rule was instead put in the filter/input chain:

    root@iptrules:~# iptables -L -v --line -n -t raw 
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1     155M   83G CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK

    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1      412  187K CT         all  --  *      *       0.0.0.0/0            0.0.0.0/0            NOTRACK

    root@iptrules:~# iptables -L -v --line -n -t filter 
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         
    1    58603 9142K DROP       udp  --  *      *       198.51.100.1         0.0.0.0/0            udp dpt:5001

    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination         

    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    num   pkts bytes target     prot opt in     out     source               destination     

Looking at the CPU usage we can see the following:

There isn't much in it here, but it does look like the CPU usage was slightly lower with the drop rule placed in the filter/input chain. Despite the system having to do a routing lookup on the 58603 packets it dropped, not having to evaluate the DROP rule on the 155 million packets forwarded made the overall CPU lower.

Conclusion

I think it's safe to say that the mixture of traffic is definitely a factor in where it is best to deploy filtering rules for a given scenario, which was my original contention. This test was not very scientific and used fairly basic and rough measurements, but on the whole that seems to be borne out.

Obviously as the ratio of packets being dropped changes, with the ultimate scenario that everything is dropped like in test 1, being able to skip the route lookup for them brings greater gains. It's also worth noting that just because a normal scenario has a much higher proportion of forwarded packets to those destined for the router CPU, in certain scenarios, like a denial-of-service attack, those norms could change radically. It may well be critical in such a scenario to drop early to have any chance of preserving the system cpu. My goal wasn't to determine the best thing to do to cover every situation, but rather to get a small sense of the relative performance of one of these options versus the other.