Fixing MetalLB “context deadline exceeded” Webhook Timeout: VXLAN Checksum Offload Bugs on RHEL-Based Kubernetes Clusters

The Problem: MetalLB Webhook Validation Timeout

We built a 3-node k3s cluster using k3sup, explicitly disabling both the built-in servicelb and traefik:

curl -sfL https://get.k3s.io | sh -s - --disable=servicelb --disable=traefik

Our plan was to install MetalLB separately to handle LoadBalancer services. We deployed MetalLB using the official YAML manifest:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml

All pods came up healthy:

kubectl get pods -n metallb-system
NAME                                      READY   STATUS    RESTARTS   AGE
metallb-controller-7b6b8f9b5d-abc12       1/1     Running   0          2m
metallb-speaker-6d8f9b5c7d-def34          1/1     Running   0          2m
webhook-server-5f6b8c9d7e-ghi56           1/1     Running   0          2m

Then we wrote our IPAddressPool manifest:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: my-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250

And applied it:

kubectl apply -f IPAddressPool.yaml

The response:

Error from server (InternalError): error when creating "IPAddressPool.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": context deadline exceeded

The webhook service was present and the ValidatingWebhookConfiguration was registered. The kube-apiserver was attempting to call MetalLB’s validation webhook but timing out after 10 seconds.

We confirmed the webhook service endpoint existed:

kubectl -n metallb-system get svc webhook-service
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
webhook-service  ClusterIP   10.43.123.456   <none>        443/TCP    5m

We even created a debug pod in the same namespace and tested connectivity:

kubectl run test-pod --rm -it --image=curlimages/curl -- sh
curl -k https://webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s

The webhook responded (with a 400 due to wrong method/body), proving the service itself was reachable. The issue wasn’t that the webhook was down—it was that the apiserver couldn’t reach it.

Initial Troubleshooting: The Dead Ends

We went through the usual checklist first.

SELinux and firewalld. We set SELinux to permissive and stopped firewalld:

sudo setenforce 0
sudo systemctl stop firewalld
sudo systemctl disable firewalld

No change. The context deadline exceeded persisted.

Node reboots. We rebooted all three nodes. Sometimes this resolves transient network issues on other distributions. On AlmaLinux, it changed nothing.

GitHub issues. We scrolled through dozens of MetalLB issues (#2486, #2566, #2599). Most suggested checking webhook availability, service endpoints, or network policies. None of these addressed our specific failure mode.

iPod network policies. We checked for any NetworkPolicies that might be blocking apiserver-to-webhook traffic. None were present.

At this point, we had a working webhook endpoint that responded to curl from a pod, but the apiserver couldn’t reach it. The 10-second timeout was being hit consistently. Something was wrong at the network layer between the apiserver and the webhook pod.

The Turning Point: Checksum Offload

A colleague, Daniel, pointed us in a different direction. He suggested the issue might be related to the flannel.1 VXLAN interface’s checksum offload. His recommendation:

sudo ethtool -K flannel.1 tx-checksum-ip-generic off

We ran this on all three nodes:

for node in node1 node2 node3; do
  ssh $node "sudo ethtool -K flannel.1 tx-checksum-ip-generic off"
done

Then we reapplied the IPAddressPool:

kubectl apply -f IPAddressPool.yaml

It worked immediately. The IPAddressPool was created without errors.

We were stunned. The entire 10-second timeout issue, the hours of debugging, the SELinux tweaks, the firewall toggles—all of it came down to a single ethtool command on a virtual network interface.

Why This Happens: The Kernel Bug

The tx-checksum-ip-generic feature, when enabled, offloads IP checksum calculation to the network interface hardware. For VXLAN interfaces like flannel.1, this means the NIC hardware attempts to compute checksums for VXLAN-encapsulated UDP packets.

On RHEL-based distributions (AlmaLinux, CentOS, RHEL, Rocky Linux), certain kernel versions and NIC driver combinations have a well-documented bug: the hardware computes incorrect checksums for VXLAN-encapsulated packets. When the kube-apiserver sends a webhook request to the MetalLB webhook service, the request traverses the VXLAN tunnel via flannel.1. The packet arrives at the target node with an invalid UDP checksum. The receiving kernel’s network stack sees the bad checksum and drops the packet.

The apiserver waits for a response. None comes. After 10 seconds, the context deadline expires, and we get context deadline exceeded.

This isn’t a MetalLB bug. It’s a kernel/networking bug that manifests when MetalLB’s webhook validation tries to cross the VXLAN tunnel. The webhook pod itself is healthy and responding—we proved that with the curl test—but the apiserver’s request never makes it through because the VXLAN-encapsulated packet is dropped at the receiving node.

The Persistence Problem

We rebooted one node to test persistence. The flannel.1 interface is dynamically created by k3s on startup. After reboot:

sudo ethtool -k flannel.1 | grep tx-checksum-ip-generic
tx-checksum-ip-generic: on

The setting was lost. The interface was recreated with default checksum offload enabled.

We tried using nmcli to make the change persistent:

nmcli con mod flannel.1 ethtool.tx-checksum-ip-generic off

This failed because flannel.1 is a virtual interface, not managed by NetworkManager.

We considered /etc/rc.local, but the boot order is unreliable—the interface might not exist when rc.local runs.

Cross-CNI Validation: Cilium and RKE2

We wanted to confirm whether this was specific to Flannel or a broader RHEL issue.

Cilium. We switched to Cilium as the CNI:

kubectl delete -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.15/install/kubernetes/quick-install.yaml

The same context deadline exceeded appeared when applying the IPAddressPool. But this time we knew what to look for. Cilium creates a VXLAN interface named cilium_vxlan:

sudo ethtool -K cilium_vxlan tx-checksum-ip-generic off

The IPAddressPool applied successfully. The bug wasn’t CNI-specific—it was the RHEL kernel’s VXLAN checksum offload implementation.

RKE2. We deployed RKE2 (which uses Canal—Calico + Flannel—by default). The same issue appeared. RKE2’s Helm chart does have a ChecksumOffloadBroken=true configuration option for Calico, but it’s not enabled by default, and it only applies to Calico’s VXLAN interface, not Flannel’s. We still had to run ethtool -K flannel.1 tx-checksum-ip-generic off manually on each node.

Production-Ready Solution: Systemd with Auto-Detection

Since the interface name varies by CNI (flannel.1, cilium_vxlan, vxlan.calico, or cali+ prefixes), hardcoding the interface name in a systemd service is brittle. We built a systemd service with an auto-detection script.

Create /usr/local/bin/fix-vxlan-checksum.sh:

#!/bin/bash
# Auto-detect VXLAN interfaces used by CNI and disable TX checksum offload
# RHEL family VXLAN checksum offload bug workaround

set -e

# List of known CNI VXLAN interface patterns
INTERFACE_PATTERNS="flannel.1 cilium_vxlan vxlan.calico"

for pattern in $INTERFACE_PATTERNS; do
    if ip link show "$pattern" >/dev/null 2>&1; then
        CURRENT=$(ethtool -k "$pattern" 2>/dev/null | grep "tx-checksum-ip-generic" | awk '{print $2}')
        if [ "$CURRENT" != "off" ] && [ "$CURRENT" != "off [fixed]" ]; then
            echo "Disabling tx-checksum-ip-generic on $pattern (was: $CURRENT)"
            ethtool -K "$pattern" tx-checksum-ip-generic off
        else
            echo "$pattern: tx-checksum-ip-generic already off (current: $CURRENT)"
        fi
    fi
done

# Fallback: find any vxlan-type interface not in the known list
for iface in $(ip -o link show type vxlan | awk -F': ' '{print $2}'); do
    if [[ ! " $INTERFACE_PATTERNS " =~ " $iface " ]]; then
        CURRENT=$(ethtool -k "$iface" 2>/dev/null | grep "tx-checksum-ip-generic" | awk '{print $2}')
        if [ "$CURRENT" != "off" ] && [ "$CURRENT" != "off [fixed]" ]; then
            echo "Disabling tx-checksum-ip-generic on $iface (was: $CURRENT)"
            ethtool -K "$iface" tx-checksum-ip-generic off
        fi
    fi
done

Make it executable:

sudo chmod +x /usr/local/bin/fix-vxlan-checksum.sh

Create /etc/systemd/system/fix-vxlan-checksum.service:

[Unit]
Description=Disable TX checksum offload for CNI VXLAN interfaces (RHEL workaround)
Documentation=https://github.com/metallb/metallb/issues/2486
After=network.target k3s.service
Wants=k3s.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/fix-vxlan-checksum.sh
RemainAfterExit=yes
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now fix-vxlan-checksum.service

Verify the service status:

sudo systemctl status fix-vxlan-checksum.service
sudo journalctl -u fix-vxlan-checksum.service

The service runs after k3s.service starts, ensuring the CNI interface exists before we attempt to modify it. The auto-detection script handles Flannel, Cilium, Calico VXLAN, and any other VXLAN-type interfaces that might appear.

Verification

After applying the fix, verify the checksum offload state on each node:

for iface in flannel.1 cilium_vxlan vxlan.calico; do
  if ip link show "$iface" >/dev/null 2>&1; then
    echo -n "$iface: "
    ethtool -k "$iface" | grep tx-checksum-ip-generic
  fi
done

Expected output:

flannel.1: tx-checksum-ip-generic: off

Then test the MetalLB IPAddressPool creation:

kubectl apply -f IPAddressPool.yaml

The apply should succeed within seconds—no timeout, no context deadline exceeded.

---

Summary

Component	What We Learned
Root Cause	RHEL-family kernels have a bug in VXLAN checksum offload, causing packets to be dropped due to invalid UDP checksums
Why Webhooks Fail	kube-apiserver → webhook requests traverse VXLAN; dropped packets cause 10s timeout
Why Other Distros Work	Ubuntu and Debian kernels handle VXLAN checksum offload correctly
Temporary Fix	ethtool -K <interface> tx-checksum-ip-generic off
Permanent Fix	Systemd service with auto-detection, running after k3s.service
CNI Agnostic	Affects Flannel (flannel.1), Cilium (cilium_vxlan), Calico (vxlan.calico)
RKE2 Note	ChecksumOffloadBroken=true exists but not default for Flannel/Canal

If you’re deploying Kubernetes on AlmaLinux, Rocky Linux, RHEL, or CentOS with VXLAN-based CNI, and you encounter MetalLB webhook timeouts, skip the SELinux and firewall debugging. Go straight to the VXLAN interface and disable TX checksum offload. The 10-second timeout is the symptom; the invalid checksum is the cause.

References

Official Documentation

1. MetalLB – IPAddressPool Configuration
   MetalLB official documentation on IPAddressPool CRD and validation webhooks
   https://metallb.io/configuration/
2. MetalLB – Custom Resources API Reference
   Full API documentation for MetalLB CRDs including IPAddressPool v1beta1
   https://metallb.io/apis/
3. K3s – Networking Services (Disabling ServiceLB and Traefik)
   Official K3s documentation on --disable=servicelb and --disable=traefik flags
   https://docs.k3s.io/networking/networking-services

Last verified: 20 July 2026 (Docker Engine 29.6.1, Linux kernel 7.0.9)