How Internet Routing Recovers From Network Failures

A router can have a valid route while users still cannot reach the service.

The route may point to an unreachable next hop. The backup link may be congested or have a smaller MTU. The forwarding plane may still contain stale state. Or packets may reach the destination while DNS, TLS, a load balancer, or the application fails.

That is the central constraint in Internet failover: BGP can change the path, but it cannot make every dependency on that path healthy.

Remember this framework:

If any link in that chain fails, the incident continues even if a BGP command shows a new route.

Consider this illustrative topology for the documentation prefix 203.0.113.0/24:

Here, each AS is an autonomous system: a network, or group of networks, managed by one organization and following a common routing policy. The number after AS is its ASN, or autonomous system number. You can think of an ASN as a network's identifier on the Internet - similar to a name tag that routing protocols use to distinguish one independently managed network from another.

Under normal conditions, packets follow this path:

If the direct connection from AS64500 to AS64501 fails, AS64500 may instead send traffic through AS64502:

AS65010 does not select AS64502 in this example. It continues to send traffic to AS64500. AS64500 makes the local decision to use its alternate path and then advertises the resulting reachability to AS65010.

This distinction matters when debugging. The router that appears in a traceroute is not necessarily the router that selected the route. Each autonomous system makes its own decision based on the routes it receives, its policy, and the reachability of the next hop.

BGP operates in the control plane. It exchanges reachability information between autonomous systems and selects routes according to policy. The forwarding plane uses the resulting local entries to send packets, usually with longest-prefix matching.

BGP handles routing between autonomous systems, or ASes. It distributes routes for prefixes along with attributes describing how to reach them. Within an AS, an IGP such as OSPF or IS-IS commonly provides the route to a selected BGP next hop.

In practical terms:

RFC 4271 describes separate stores for routes received from peers, routes selected locally, and routes selected for advertisement. A route received from a neighbor is not automatically the route used for forwarding or advertised to another neighbor. RFC 4271

That distinction matters during an outage. A BGP session can remain established while a next hop is unreachable through the local network. A selected BGP path also does not prove that the destination service is working.

At AS64500, two routes to the example prefix might be available:

The AS_PATH records the ASes an announcement has traversed. It also helps prevent loops: an AS normally rejects a route containing its own AS number. RFC 4271

AS64500 may advertise its selected route to AS65010. Before the failure, AS65010 sees a route through AS64500 with this path:

It is tempting to assume BGP always chooses the shortest AS_PATH. It does not. BGP is primarily a policy system. An operator can prefer a customer route, a private interconnect, lower-cost transit, or an operationally safer path even when it has a longer AS path.

A simplified selection process is:

The exact order varies by implementation and configuration. RFC 4271 describes the protocol decision process, while vendors document their own best-path behavior. Cisco's BGP best-path documentation is one example.

This creates a common failure mode: the backup exists, but policy makes it ineligible or less preferred than another path. Another is that the BGP route is selected but its next hop cannot be resolved through a connected route or IGP. Both cases require checking more than the BGP table.

RPKI origin validation can classify an announcement as Valid, Invalid, or NotFound based on whether the origin AS is authorized for the prefix. It does not validate the entire AS_PATH, prove that every transit network is working, or prove that the service is healthy. RFC 6811

In this example, AS64500 prefers its direct route through AS64501. Its local policy gives that route higher preference than the backup through AS64502, so new traffic uses:

Now simulate a failure of the BGP session or forwarding path between AS64500 and AS64501. Detection can come from several sources:

BFD is a fast failure-detection mechanism. It does not choose a BGP route itself, but a BFD-down event can cause BGP to tear down a session sooner than waiting for the Hold Timer. RFC 5880 RFC 5882

The recovery sequence is then:

After convergence, new packets follow:

AS65010 does not directly select AS64502 in this topology. AS64500 selects the backup because it owns the backup connection. AS65010 continues sending traffic to AS64500.

BGP UPDATE messages can announce replacement reachability or withdraw previously advertised reachability. If AS64500 has no usable alternate, it withdraws the prefix from AS65010. If the backup is usable, AS64500 can advertise the replacement path instead, which AS65010 sees as:

KEEPALIVEs maintain sessions, and Hold Timer expiry ends a session when expected messages stop arriving. RFC 4271

Packets already in flight can be lost while forwarding entries change. A route withdrawal is not an instant global deletion, and each router applies its own policy on its own schedule. Return traffic may also use another path, so a forward traceroute does not prove end-to-end symmetry.

A backup route can restore reachability while still causing user-visible trouble. The alternate path may have more latency, less capacity, a lower MTU, or a congested interconnect. It can also briefly blackhole traffic if BGP selects it before the next hop is usable in the forwarding system.

A link failure and a policy failure need different responses. With a link failure, a route disappears and routers may select another path. With a route leak, a network advertises a route beyond its intended scope and can attract traffic that should not enter that network. RFC 7908 classifies several route-leak types, including routes learned from a peer and improperly advertised to a provider. RFC 7908

BGP can also settle on an unexpected stable path. RFC 4264 describes cases with multiple stable outcomes. After a physical link returns, policy and update ordering can leave traffic on the backup route. Restoring connectivity does not guarantee that forwarding returns to its earlier state. RFC 4264

Graceful Restart is another trade-off. During some control-plane restarts, a peer can retain routes as stale while the restarting router reconnects and completes route exchange. This can preserve forwarding during a software restart. If the forwarding plane failed too, stale routes can instead send traffic into a blackhole. RFC 4724

Traffic engineering cannot create capacity. In an August 2025 incident report, Cloudflare said that withdrawing prefixes to move traffic away from congested AWS links shifted traffic onto other links that also saturated. The result included packet loss and latency until manual actions changed the traffic pattern. BGP can redistribute offered load, but remaining paths must still carry it. Cloudflare’s incident report

Start with the system closest to the user or service. On Linux, inspect the route the kernel would use:

Check the selected interface, source address, gateway, policy-routing rules, and neighbor resolution. ip route get reports the kernel's decision; it does not send a packet. Linux ip-route manual

Then inspect the path:

Asterisks do not prove that the path is broken. ICMP filtering, rate limits, MPLS, load balancing, and asymmetric return traffic can all make traceroute incomplete. Linux traceroute manual

Test the service separately:

A successful route does not prove that DNS, TLS, firewall policy, load-balancer behavior, or the application is healthy.

For networks under your control, compare these states:

For external context, compare multiple BGP collectors. RIPE NCC's BGPlay and BGP Updates interfaces show collector-observed announcements, withdrawals, AS paths, and timestamps. They do not expose every router's private RIB or forwarding state. RIPE BGPlay API RIPE BGP Updates API

The practical test is not "does the backup route exist?" It is:

Redundant routing is only real redundancy when those conditions are tested during both failure and restoration. BGP supplies route choices. Production recovery depends on validating the entire chain from detection to user-visible service.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.