The symptom

Single-node cluster at home. Cilium as the CNI, doing ingress and L2 load balancing. The goal was mundane: https://api.llama.internal/ should hit a llamacpp pod through Cilium’s ingress.

It didn’t. From any machine on my LAN:

$ curl -k --max-time 15 https://api.llama.internal/
curl: (28) Connection timed out after 15004 milliseconds

The connection never even got to TLS. But here’s the part that sent me down a three-hour rabbit hole: from the node itself, it worked fine.

# on the node
$ curl -k https://api.llama.internal/health
{"status":"ok"}

Ingress “up” from the node, dead from everywhere else. That asymmetry is the whole story, and it took a while to understand why.

Following the packet

The VIP (192.168.1.14) answered ARP with the node’s MAC, so L2 was fine. A tcpdump on the node’s uplink showed the SYNs arriving, and no SYN-ACK, ever:

IP 192.168.1.158.50015 > 192.168.1.14.443: Flags [S] ...
IP 192.168.1.158.50015 > 192.168.1.14.443: Flags [S] ...   # retransmit, forever

Cilium’s ingress runs an Envoy proxy on the host, bound to 127.0.0.1:17564. North-south traffic to the ingress VIP is supposed to be handed to Envoy via the kernel’s TPROXY facility: eBPF tags the packet with a proxy mark, an iptables TPROXY rule in mangle PREROUTING redirects it to the loopback proxy, and a policy-routing rule delivers it locally.

Checking the TPROXY rule counter, it was incrementing. The packets were being matched and redirected. But Envoy’s admin socket told the real story. Its downstream connection counter moved for localhost curls and stayed flat for LAN curls:

listener.127.0.0.1_17564.downstream_cx_total: 9   # before 5 LAN curls
listener.127.0.0.1_17564.downstream_cx_total: 9   # after  5 LAN curls  (!)
listener.127.0.0.1_17564.downstream_cx_total: 12  # after  3 localhost curls

So TPROXY fired, but the packet never reached Envoy’s socket. It was dying between the TPROXY redirect and local delivery.

nft monitor trace pinned the exact spot. The SYN sailed through raw → mangle → nat PREROUTING, the TPROXY target set the mark to 0x200 and accepted it… and then the trace simply stopped. No filter INPUT, no FORWARD. The packet vanished at the input-routing decision, a FIB step, not a netfilter hook. That’s why nothing logged it.

Not Cilium’s fault

At this point I did the single most useful thing in the whole investigation: I reproduced it without Cilium at all. A plain Python listener with IP_TRANSPARENT on 127.0.0.1:19999, and one hand-written TPROXY rule pointing a spare port at it:

iptables -t mangle -I ... -p tcp --dport 9999 \
  -j TPROXY --on-port 19999 --on-ip 127.0.0.1 --tproxy-mark 0x200/0xffffffff

Same failure. TPROXY counter climbed, listener never saw a connection, curl timed out. This wasn’t Cilium, wasn’t Envoy. It was the host’s kernel datapath. That reframed everything and let me iterate fast.

The usual suspects, all innocent

TPROXY-to-loopback problems have two famous causes, and neither was it:

  • rp_filter (strict reverse-path filtering), the classic cause of martian source drops in Cilium. I set it to 0 on every interface. Still failed.
  • route_localnet, needed when delivering to 127.0.0.0/8. Set it to 1. Still failed.

I also flipped Cilium’s bpf.hostLegacyRouting back and forth (BPF host routing vs. the legacy stack) on the theory that the datapath mode mattered. It didn’t. The drop was identical either way.

The smoking gun

Enable martian logging and the kernel finally admits what it’s doing:

$ sysctl net.ipv4.conf.all.log_martians=1
$ dmesg
IPv4: martian source 192.168.1.102 from 192.168.1.158, on dev enp18s0

A martian source, even with rp_filter=0. That’s the clue that cracks it, because rp_filter is not the only source-validation the kernel does.

Here’s the mechanism. TPROXY rewrites the packet’s mark to 0x200. Cilium’s policy-routing rule steers 0x200 traffic to a dedicated table:

ip rule:  9: from all fwmark 0x200/0xf00 lookup 2004
table 2004: local default dev lo

local default dev lo means deliver everything locally via loopback. That’s the standard TPROXY pattern. But when the kernel validates the packet’s source address under that same mark, the lookup resolves through table 2004 too, so the source 192.168.1.158 comes back as RTN_LOCAL.

And the kernel has a rule: a packet whose source address resolves to a local route is a martian unless net.ipv4.conf.<dev>.accept_local = 1. From the kernel ip-sysctl docs:

accept_local — Accept packets with local source addresses. […] default FALSE

It was 0. So every external SYN, once TPROXY assigned it the 0x200 mark, had a source that “looked local” and got dropped as a martian.

The fix

# /etc/sysctl.d/99-cilium-tproxy-accept-local.conf
net.ipv4.conf.all.accept_local = 1
net.ipv4.conf.default.accept_local = 1

$ curl -k https://api.llama.internal/health
{"status":"ok"}     # HTTP 200, ~20ms

One setting. The full chain came alive immediately: LAN client → VIP → Envoy (TLS terminates) → llamacpp pod. The martian logs stopped. Verified it was the sole necessary change by resetting rp_filter and route_localnet back to their defaults; ingress kept working.

Why it fooled me for so long

  • The node worked. Cilium’s socket-level load balancing rewrites a local connect() straight to 127.0.0.1:17564, so on-node traffic never touches the TPROXY/input-routing path. “Works from the node” was a false signal of health.
  • The drop is invisible. It’s a FIB martian drop: no Hubble event, no cilium monitor line, no rp_filter counter, no InNoRoutes. Without log_martians, there is nothing to see.
  • rp_filter=0 didn’t fix it. Every guide points you at rp_filter for martian drops. This one is accept_local, a different and much less discussed knob, triggered by the source resolving local through the proxy’s own policy-routing table.

Is this documented anywhere?

cilium#46260 independently arrived at the same root cause: TPROXY’d L7-proxy packets dropped as martian source in ip_route_input_slow, with the same diagnosis that Cilium sets accept_local=1 only on its own devices, not on the native L2-announced ingress interface. The issue is still open as of this writing.

The building blocks each exist in isolation: the kernel accept_local definition, the TPROXY + policy-routing pattern, and the well-known rp_filter-loose fix for a different Cilium martian case (cilium#13130). There are also several “ingress works from the node but not outside” reports (cilium#42275, cilium#25021) that never reach this diagnosis.

If you hit “Cilium ingress times out from the LAN but works from the node,” this is worth checking:

sysctl net.ipv4.conf.all.log_martians=1
dmesg -w        # curl from a LAN client; watch for "martian source"
# if you see it:
sysctl net.ipv4.conf.all.accept_local=1

Environment

Cilium v1.19.6 (tunnel/VXLAN, kube-proxy replacement, external Envoy DaemonSet on hostNetwork), kernel 6.19.13, Debian 13, single node. The reverse-path validation resolving the source through the marked policy table is the kind of behavior that can vary across kernels, so your mileage, and whether you ever see this at all, may depend on the kernel you’re running.