Every Tuesday - Deep dives, architecture lessons, and real engineering stories.

Every Saturday - The best DevOps, SRE, Cloud, AI, tools, tutorials, and projects from the week.

Sponsored by

How 2M+ Professionals Stay Ahead on AI

AI is moving fast and most people are falling behind. 

The Rundown AI keeps you ahead of the curve. 

It's a free AI newsletter that keeps you up-to-date on the latest AI news, and teaches you how to apply it in just 5 minutes a day.

Plus, complete the quiz after signing up and they’ll recommend the best AI tools, guides, and courses — tailored to your needs.

📬 Missed This Week’s Uptime Sync?

This week’s best DevOps, SRE, Cloud, AI infrastructure, database, and production engineering reads:

  • The AI outage nobody is explaining

  • What the Buildkite incident taught us about failures

  • AI crawlers consuming 20% of kernel dot org traffic

  • Why six identical $20 VMs can perform very differently

  • Why updating a vector database is not as simple as it sounds

  • Nvidia’s reported $12.9B Hugging Face acquisition

Plus: disaster recovery with rsync, DynamoDB recovery, Unix domain sockets, low-cost AWS static hosting, GCP cost optimisation, and Docker migration lessons.

Do not keep editing or rolling back a DNS record just because some users still see the old service. The record may already be correct. Repeated edits can make diagnosis harder because different resolvers may cache different versions at different times.

The first question is simple: does authoritative DNS return the intended record, or are users getting an older answer from somewhere else?

A request usually follows this path:

application → local DNS cache/stub → recursive resolver → authoritative DNS → returned IP → load balancer/routing → service

Each layer can fail differently. The authoritative nameserver owns the DNS record. A recursive resolver can answer from its cache or query authoritative servers and cache the result, as described in the BIND resolver documentation.

After DNS returns an IP address, DNS is no longer the only thing involved. A CDN, load balancer, route, backend pool, or deployment can still send traffic to the wrong application.

For example, api.example.com moves from 198.51.100.10 to 198.51.100.20. The authoritative server returns .20, but a corporate resolver may still return .10 because it cached the old answer before the change. That is stale resolution, not necessarily a bad record.

TTL Explains Delayed Refreshes

TTL means Time To Live. It tells resolvers how long they may normally cache an answer.

If a record had a TTL of 300 seconds, a resolver that cached the old IP just before your change may keep returning it for about five more minutes. Another resolver may have cached it much earlier and refresh sooner. Users can therefore receive different answers during the same period.

This is why “DNS propagation takes 24-48 hours” is not a DNS rule. The delay depends on existing caches, resolver settings, local caches, and sometimes DNS policy. Longer TTLs reduce lookup traffic, but they also make record changes slower to reach resolvers that already cached an answer, as explained in Cloudflare’s TTL documentation.

The TTL shown by dig is usually the remaining cache time at that resolver. It is not always the TTL configured in the zone.

api.example.com. 143 IN A 198.51.100.10

This means that resolver may serve the old address for roughly 143 more seconds. It does not prove the zone record was configured with a 143-second TTL.

Compare Public and Local Resolvers

Collect evidence before flushing caches or editing records again. Compare two public resolvers with the resolver used by the affected host.

name=api.example.com
for dns in 8.8.8.8 1.1.1.1 127.0.0.53; do
  echo "=== $dns ==="
  dig @$dns "$name" A +noall +answer +authority
  dig @$dns "$name" AAAA +noall +answer +authority
done

Google documents 8.8.8.8 as a Public DNS resolver in its usage documentation. Cloudflare documents its DNS query service and response format in its DNS-over-HTTPS documentation.

On many Linux systems, 127.0.0.53 is a local systemd-resolved stub. It is not universal, so check the host configuration first. The systemd-resolved documentation describes this local stub behavior.

For every result, record the timestamp, resolver IP, record type, returned IP or CNAME, and remaining TTL. Then query an authoritative server directly:

dig @authoritative-ns.example.com api.example.com A +noall +answer +authority

If authoritative DNS returns the old IP, inspect the zone record, CNAME target, delegation, and consistency across authoritative nameservers.

If authoritative DNS returns the new IP but one recursive resolver returns the old IP with a positive TTL, cached positive data is the likely explanation. Also check AAAA records. A client that prefers IPv6 may reach an old destination if only the A record was updated.

When Old DNS Answers Aren’t Cached

Sometimes users do not receive an old IP. They receive a cached “not found” response.

Suppose new-api.example.com did not exist when a resolver first queried it. The resolver can cache NXDOMAIN, which means the name does not exist. It may continue returning that result after you create the record.

NXDOMAIN is different from NOERROR with no A or AAAA answer. NOERROR can mean the hostname exists but does not have the requested record type. For example, it may have an A record but no AAAA record.

Inspect the authority section when checking a negative result:

dig @1.1.1.1 new-api.example.com A +noall +answer +authority

RFC 2308 defines DNS negative caching. The SOA record in the authority section is relevant to the negative cache lifetime. Lowering the TTL of a newly created record does not remove an NXDOMAIN response that was already cached.

There is also a stale-answer exception. A resolver may temporarily serve expired data when it cannot refresh from authoritative DNS. RFC 8767 defines this serve-stale behavior. An unexpected answer with a low or unusual TTL needs investigation; do not assume the resolver ignored TTL without checking authority reachability.

When Correct DNS Still Looks Broken

Split-horizon DNS means different users intentionally receive different answers for the same hostname. Internal users may get a private IP, while internet users get a public IP.

This can be caused by corporate resolvers, VPN DNS, conditional forwarding, subnet-based DNS policies, or DNS views. Microsoft documents this as split-brain DNS, where policies can return answers based on the client network in its split-brain DNS guide.

Check for local and environment-specific overrides too:

  • /etc/hosts

  • Container DNS settings

  • Service mesh DNS

  • Search-domain expansion

  • VPN state

  • Corporate resolver configuration

DNS can be correct while the application path is still wrong. The new IP may point to a load balancer with an old backend target. A CDN may still use an old origin, or a deployment may have left the old service in a target pool.

Test the new IP while keeping the hostname for HTTP Host handling and TLS SNI:

curl -v --resolve api.example.com:443:NEW_IP https://api.example.com/

If this returns the old application on NEW_IP, DNS is not the main problem. Check load balancer targets, CDN origin settings, routing rules, and deployment configuration.

A Short Checklist

Use this order during an incident:

  1. Query authoritative DNS. If it is wrong, check the zone record, CNAME chain, delegation, and authoritative nameservers.

  2. Compare public, corporate, and local resolvers. Check A, AAAA, CNAMEs, returned addresses, and remaining TTLs.

  3. An old IP with a positive TTL usually points to cached positive data. Wait for expiry or test through the affected resolver.

  4. NXDOMAIN or NOERROR with no requested record and an SOA in authority points to negative caching.

  5. Different internal and external answers require checking split-horizon policy, VPN DNS, and conditional forwarding.

  6. A correct new IP that returns the old application means the issue is likely routing, CDN, load balancer, or service configuration.

  7. Flush local caches only after collecting evidence. Flushing one laptop or server does not clear corporate, ISP, or public recursive caches.

On systems using systemd-resolved, resolvectl flush-caches clears that local resolver cache. Its cache behavior is documented in the resolved configuration documentation.

Join 1,000+ engineers becoming better DevOps & SRE professionals.

Every week, I share:

  • How I'd approach problems differently (real projects, real mistakes)

  • Career moves that actually work (not LinkedIn motivational posts)

  • Technical deep-dives that change how you think about infrastructure

No fluff. No roadmaps. Just what works when you're building real systems.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.

Reply

Avatar

or to participate