How Kubernetes Networking Works: From a Service IP to Your Pod

A frontend pod calls backend.default.svc.cluster.local. It does not know which backend pod will handle the request, where that pod runs, or whether a rollout replaced it a few seconds ago.

A Kubernetes Service gives the frontend one stable name and virtual address, while the pods behind it come and go, possibly on different nodes.

Suppose the cluster has this layout. The addresses are examples; your cluster's ranges may differ.

The frontend opens a TCP connection to backend:8080. For normal ClusterIP traffic, the conceptual path is:

Where selection and translation happen depends on the active Service datapath. An iptables cluster, an nftables cluster, and a kube-proxy-free eBPF cluster each use different kernel hooks and rule formats.

Each Pod has its own network identity. Containers in the same Pod share a network namespace, IP address, port space, and localhost. If one container listens on 127.0.0.1:8080, another container in that Pod can reach it, but a different Pod cannot. The Pod documentation describes this shared networking model.

A Pod IP is useful for direct communication and debugging, but it is not a stable address for an application.

Suppose backend-2 is deleted during a Deployment rollout. Kubernetes does not move that Pod to another node. It creates a replacement Pod, which is a new object with a new UID and may receive a different IP address. A container restart inside an existing Pod can keep the Pod IP, but treat a replaced Pod as a new network endpoint. The Pod lifecycle documentation covers the difference.

If the frontend called 10.244.2.31 directly, it would fail once backend-2 disappeared. It calls the backend Service instead, and Kubernetes keeps that address stable while it updates the list of current backends.

Inside the default namespace, the frontend can usually use the short name:

The Pod's DNS search path expands that name to something like:

For a normal ClusterIP Service, CoreDNS answers with the Service's virtual IP, not with any one backend Pod IP:

The exception is a headless Service. It has no ClusterIP, so DNS can return the endpoint Pod addresses instead.

A successful lookup proves only that the name resolved. The Service may still have no ready endpoints, the application may not be listening on the expected port, or a NetworkPolicy may block the traffic.

CoreDNS only answers the DNS query; the application request never passes through it. The frontend then sends its TCP packet to 10.96.42.17:8080.

A Service usually has a selector. Suppose backend selects Pods with:

Kubernetes watches matching Pods and maintains EndpointSlices for the Service. An EndpointSlice holds backend addresses, ports, node information, zones, and conditions such as readiness. The EndpointSlice API is how Kubernetes represents a Service's current backends.

Conceptually, the EndpointSlices for this example might contain:

A Pod can be running and still get no normal Service traffic. If its readiness probe fails, Kubernetes can leave the container running but remove its address from the usable endpoints, so new connections stop reaching a Pod that is starting, draining, or failing its readiness check.

That accounts for a common failure: the Service exists and DNS works, but calls time out because there are no ready endpoints. A selector typo, mismatched Pod labels, failing readiness probes, or a wrong targetPort can each cause it.

The frontend has now sent a packet to:

The ClusterIP is a virtual address. Unlike a Pod IP, you will not find it attached to a normal network interface. A node-level Service datapath captures traffic sent to that address and redirects it to a backend.

In the common iptables kube-proxy mode, kube-proxy watches Services and EndpointSlices and programs Linux netfilter rules on each node. Those rules match the Service IP and port, choose an endpoint, and rewrite the packet's destination with destination NAT (DNAT). The virtual IP and Service proxy reference documents this behavior.

Suppose a new TCP connection from the frontend selects backend-2.

After DNAT, the packet is addressed to the Pod IP. In this example, with no source NAT on the path:

Other cluster or CNI settings can add source NAT, so a packet capture will not always show the original Pod source address.

In this mode, kube-proxy normally does not handle packets in user space. It watches the Kubernetes API and programs kernel rules, and the Linux kernel processes the matching traffic.

Connection tracking (conntrack) records the NAT mapping for each connection, so reply packets from backend-2 are translated correctly on the way back. Conntrack failures can break connections, but check DNS, endpoints, ports, policy, and Pod connectivity first.

Once the destination is a Pod IP, the CNI plugin handles the path to that Pod.

If backend-1 on worker-a is selected, delivery stays on the same node:

A common Linux setup connects each Pod's network namespace to the node through a veth pair. One end appears as eth0 inside the Pod, and the host-side end connects to a bridge, a route, or some other CNI-managed datapath. Names such as cni0 show up in some setups, but Kubernetes does not require them.

If backend-2 is selected, the packet has to cross from worker-a to worker-b.

Kubernetes requires Pods to be able to reach each other across nodes, but it leaves the routing design and CNI implementation to the network plugin. See the cluster networking model.

With native routing, the underlying network must know how to reach Pod addresses or Pod CIDRs, so check node routes, cloud routes, and firewall rules. With an overlay, the CNI wraps Pod traffic in another packet addressed between nodes, so also check that encapsulated traffic is allowed and that MTU and tunnel settings are right.

A Service does not promise that each HTTP request reaches a different Pod.

Backend selection generally happens when the client opens a new connection or flow. If the frontend opens one TCP connection to backend and HTTP keep-alive holds it open, ten HTTP requests can all land on backend-2.

Connection pools do the same thing. A frontend with four long-lived connections might send most of its traffic to one or two Pods even though all three endpoints are healthy.

In iptables mode, Kubernetes documents random endpoint selection unless session affinity applies, so expect neither strict round-robin nor per-request balancing. What you observe also depends on client connection reuse, session affinity, proxy mode, and CNI settings.

When one backend gets more requests than the others, check connection behavior before assuming Service load balancing is broken.

The iptables example shows destination translation clearly, but it is only one implementation.

Kube-proxy can also run in nftables mode on supported Linux systems. It has supported IPVS mode too, though that mode's status depends on the version. According to the Kubernetes documentation, IPVS was deprecated in v1.35, is disabled by default in v1.40, and is planned for removal in v1.43. Check the current proxy mode documentation for the version you run.

A Cilium cluster can run without kube-proxy and use eBPF maps for Service load balancing and translation. In that setup there may be no kube-proxy iptables rules at all, so searching for them tells you nothing. Cilium calls this kube-proxy-free operation.

Kubernetes API checks work the same in every mode, but rule inspection does not, so identify the active proxy mode and CNI before you run node-level diagnostics.

Start in the frontend Pod by checking its resolver settings and name resolution.

Many production application images do not include nslookup. If yours doesn't, use a diagnostic Pod with DNS tools; a missing binary is not a DNS failure.

Next, inspect the Service and its endpoints.

Compare the Service selector with the Pod labels, and the ready endpoint addresses with the Pod IPs and their nodes. Make sure the Service targetPort matches the port the backend actually listens on.

Then test the Service, and test a current ready endpoint directly. Use a frontend image that includes wget, or run a separate diagnostic Pod.

Take <ready-pod-ip> and <target-port> from a ready EndpointSlice, not from an old example or a previous rollout.

Comparing the two results narrows the problem:

If you have node access, look at the actual routes instead of guessing how the CNI works:

For a remote endpoint, check whether the node has a route or CNI-managed path to the remote Pod network. The output varies by CNI, cloud provider, and routing mode.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.