Every Tuesday - Deep dives, architecture lessons, and real engineering stories.

Every Saturday - The best DevOps, SRE, Cloud, AI, tools, tutorials, and projects from the week.

📬 In Case You Missed This Week’s Uptime Sync

Every week, I curate the best DevOps, SRE, Cloud, Kubernetes, database, AI infrastructure, and production engineering reads so you don’t have to hunt for them yourself.

This week’s edition featured:

  • How OpenAI models escaped a cyber-eval sandbox and reached Hugging Face production

  • How a telemetry pipeline survives millions of events per second by rejecting work strategically

  • How Meta modernized its Ads service using an open-source kernel scheduler

  • How ClickHouse scales PgBouncer for managed Postgres

  • What happens when SQLite and DuckDB hit very different performance limits on the same tiny machine

  • Tutorials on local Kubernetes platforms, self-hosted email, MySQL audit logs, parallel AI agents, internal developer platforms, and SQL COUNT(DISTINCT ...)

  • Projects like Terraform Review Agent, Datadog SAST, PGSync, Ballast, TiDB, and Herdr

❓ Interview Question of the Week

Your application works fine locally, but inside Kubernetes it cannot reach another service by name.

How would you debug it?

Don’t just say “check DNS.”

Think through what you’d verify, and in what order.

I’ll share how I’d answer this in an interview next week.

A workload in a private subnet needs to call an external API. Should it use a NAT gateway, a VPC endpoint, AWS PrivateLink, or no new network component at all?

The answer is not in the service name. It is in the packet path.

AWS networking becomes simpler when you see VPCs, subnets, route tables, security groups, NAT gateways, and endpoints as stages in one process. A request resolves a name, selects a route, passes security controls, and then uses a gateway or endpoint to reach its destination.

Start With the Packet Path

Start with a simple request:

Application in a private subnet -> external HTTPS API

Before the request reaches its destination, AWS networking must answer several questions:

  1. What IP address does DNS return?

  2. Which route-table entry best matches that address?

  3. Do security controls allow the connection and its return traffic?

  4. Does the selected route lead to a NAT gateway, internet gateway, endpoint, peering connection, or another target?

This order matters. A NAT gateway cannot help when DNS resolves to an unexpected address. A correct route does not bypass a security group. An open security group cannot create a missing route.

A Virtual Private Cloud (VPC) is a logically isolated virtual network in AWS. You assign one or more IP address ranges to it using CIDR notation, such as 10.0.0.0/16. Subnets divide that VPC address space into smaller ranges.

VPC: 10.0.0.0/16

Public subnet, AZ-a:   10.0.1.0/24
Private subnet, AZ-a:  10.0.11.0/24
Public subnet, AZ-b:   10.0.2.0/24
Private subnet, AZ-b:  10.0.12.0/24

Each subnet belongs to one Availability Zone and cannot span multiple AZs. If an application runs in more than one AZ, it needs subnet capacity, routes, and supporting components in each relevant AZ.

The most important beginner correction is this:

A subnet is not inherently public or private.

AWS describes a subnet as public when its route table includes a direct route to an internet gateway. A subnet is private when it does not have that direct internet gateway route. A private subnet can still initiate internet connections through a NAT gateway. This is a routing distinction, not a label permanently attached to the subnet. Check this AWS subnet configuration guidance.

Use one workload throughout this article: an EC2 instance, container, or pod runs in a private subnet. It makes an HTTPS request to an external API, reads data from Amazon S3, and calls a service exposed through AWS PrivateLink.

The workload may remain in the same subnet for all three requests. What changes is the DNS answer and the route selected for the destination.

“Private” therefore means “not directly reachable through an internet gateway route.” It does not mean “unable to communicate outside the subnet” or “unable to make outbound requests.”

Routes and Security Controls

Every subnet uses one route table. You can explicitly associate a route table with a subnet, or the subnet can use the VPC’s main route table. A route has a destination and a target.

Destination

Target

Meaning

10.0.0.0/16

local

Traffic for the VPC CIDR stays within the VPC

S3 prefix list

Gateway endpoint

S3 traffic uses a gateway endpoint

0.0.0.0/0

NAT gateway

Other IPv4 destinations use NAT

The local route is created for the VPC CIDR. It enables AWS to route traffic between resources in that VPC. It is not an authorization rule. A workload can still be blocked by security groups, network ACLs, host firewalls, an unavailable listener, or application authentication.

AWS route tables use longest-prefix matching. In simple terms, the most specific matching route wins.

For example, a private subnet can have both an S3 prefix-list route to a gateway endpoint and a broad 0.0.0.0/0 route to a NAT gateway. Traffic for S3 follows the more-specific endpoint route. Other public IPv4 traffic follows the default route to NAT.

This is why an S3 gateway endpoint can remove S3 traffic from the NAT path without requiring an application change.

A route table answers only one question:

Where should this packet go next?

It does not answer whether the packet is allowed. That is the job of security controls.

A security group applies to an Elastic Network Interface (ENI) or a resource that uses one. Security groups are stateful and support allow rules. When an outbound TCP connection is allowed, return traffic for that established connection is automatically allowed.

A network ACL, or NACL, applies at the subnet boundary. NACLs are stateless, support both allow and deny rules, and evaluate rules in numerical order. Because they are stateless, request and return traffic both need matching rules. AWS documents these differences in its guidance on VPC infrastructure security and network ACL behavior.

A common failure pattern is an incomplete NACL:

  • The workload sends an outbound HTTPS request.

  • The outbound rule permits it.

  • The destination returns a response to an ephemeral client port.

  • The inbound NACL rule does not allow that return port range.

  • The application times out.

The route may be correct, and the security group may be correct, but the stateless NACL still breaks the connection.

DNS is another early dependency. Applications usually request names rather than IP addresses. Inside a VPC, the Amazon-provided resolver is part of Route 53 Resolver. Private hosted zones can return private addresses for names that resolve differently on the public internet. Interface endpoint private DNS can return private endpoint ENI addresses for supported AWS service names. Go through VPC DNS settings and Amazon DNS concepts once.

When a request fails, begin by confirming the IP address DNS returned. That answer determines which route AWS evaluates.

Public Access, Private Outbound Access, and NAT Gateway Are Different Things

An internet gateway, or IGW, attaches to a VPC and provides a path for internet-routed traffic. A typical public subnet includes a route like this:

0.0.0.0/0 -> internet gateway

That route alone does not make an EC2 instance publicly reachable.

For a typical IPv4 workload to accept inbound public traffic, it generally needs an internet gateway route, public addressing, security group permission, suitable NACL rules, and an application listening on the intended port. AWS provides routing examples in its route table documentation.

This is why many production architectures use a public-facing load balancer as the deliberate entry point. Application services and databases can remain in private subnets, with only the load balancer accepting selected public traffic.

A public NAT gateway solves a different problem: allowing private IPv4 workloads to initiate outbound connections.

The path is:

  1. A private workload sends traffic to a public destination.

  2. Its subnet route table sends default traffic to the NAT gateway.

  3. The NAT gateway translates the private source address and port to its public address.

  4. The NAT gateway sends traffic through the internet gateway.

  5. The response returns to the NAT gateway.

  6. The NAT gateway matches the response to the existing translation and forwards it to the initiating workload.

AWS has documented this behavior in its NAT gateway overview.

NAT does not make a private instance publicly reachable. It supports connections initiated from the private side. It is not an inbound reverse proxy, load balancer, or firewall replacement.

NAT gateways also do not have security groups. Security control remains the responsibility of workload security groups and subnet NACLs- NAT gateway basics.

A NAT gateway is created in one AZ. A private subnet in another AZ can route to it, but doing so creates an AZ dependency and can introduce cross-AZ traffic. As an engineering recommendation, deploy a NAT gateway per AZ and route each private subnet to its local NAT gateway when the workload’s resilience requirements justify the added fixed and data-processing costs. This is a design choice, not an AWS requirement.

AWS also documents operational behavior that can explain incidents:

  • NAT gateways have a 350-second idle connection timeout.

  • NAT gateways do not support IP fragmentation.

  • NAT gateways have no security groups.

  • High connection volumes to a destination can exhaust available ports.

  • AWS documents 55,000 concurrent connections per destination per source IPv4 address, with expansion options using additional addresses.

If many short-lived outbound connections begin failing, inspect NAT metrics and connection behavior before assuming the remote service is unavailable.

A NAT gateway provides broad outbound connectivity. VPC endpoints provide private paths to specific supported services.

The simplest type is a gateway endpoint. Gateway endpoints are available for Amazon S3 and DynamoDB. They add routes to selected route tables, commonly using AWS-managed prefix lists.

S3 prefix list -> gateway endpoint

When a workload accesses S3, that specific route can win over the default NAT route. The traffic reaches S3 without traversing a NAT gateway or internet gateway. AWS explains this model in its gateway endpoint documentation.

Gateway endpoints are useful but limited in scope. They are not a generic tool for every AWS API or private service.

An interface endpoint works differently. AWS creates endpoint ENIs in the subnets you select. Those ENIs receive private IP addresses. The workload connects to those private addresses, and security groups control traffic to the endpoint ENIs.

When private DNS is enabled, a standard AWS service hostname can resolve to those endpoint IPs. This often allows an application to keep using the normal AWS service name while its traffic uses private connectivity. AWS documents endpoint ENIs, security groups, subnet selection, and private DNS in Create an interface endpoint.

AWS PrivateLink is the service model behind interface endpoints. Its central idea is narrow service exposure.

With PrivateLink, a consumer does not receive general routed access to a provider VPC. Instead, the consumer creates interface endpoint ENIs that connect to a specific service published by the provider.

Provider VPC
Application -> Network Load Balancer -> Endpoint service

Consumer VPC
Application -> Interface endpoint ENIs -> Provider service

A provider can publish an endpoint service behind a Network Load Balancer or Gateway Load Balancer. The consumer creates an interface endpoint in its own VPC. Provider permissions, acceptance settings, endpoint security groups, DNS configuration, and AZ coverage remain separate concerns. A configured endpoint service does not guarantee that every consumer-side dependency is correct - AWS PrivateLink endpoint service documentation.

PrivateLink is not VPC peering.

VPC peering provides direct private connectivity between VPC networks through routes to peer CIDR ranges. PrivateLink exposes a specific service without broadly exposing the provider network.

Use the smallest connectivity pattern that meets the requirement:

  • Gateway endpoint for S3 or DynamoDB.

  • Interface endpoint for supported AWS services or narrowly exposed private services.

  • NAT gateway for general internet egress that endpoints cannot replace.

  • VPC peering or Transit Gateway when broader network-to-network connectivity is genuinely required.

How to Trace a Failed Request?

Return to the private workload calling an external HTTPS API.

Application
  -> VPC DNS resolver
  -> instance ENI
  -> subnet route table
  -> NAT gateway
  -> internet gateway
  -> external service

DNS returns a public IP address. The packet leaves the workload ENI and reaches the subnet route table. If no more-specific route exists, 0.0.0.0/0 selects the NAT gateway.

The workload security group must allow outbound HTTPS. The NACL must allow the outbound request and the return path. NAT translates the source address and forwards traffic through the IGW. The response returns through NAT, which reverses the translation.

S3 uses a different path:

Application
  -> DNS
  -> subnet route table
  -> S3 gateway endpoint
  -> Amazon S3

The S3 prefix-list route is more specific than the default NAT route, so S3 traffic uses the gateway endpoint.

An interface endpoint uses another path:

Application
  -> private DNS answer
  -> interface endpoint ENI
  -> AWS service or PrivateLink provider service

Here, private DNS can return endpoint ENI addresses in the VPC. Endpoint security groups and endpoint policies can still deny access.

Debug failed requests in layers:

  1. DNS: Use dig or nslookup from the workload environment. Confirm the returned address. Check VPC DNS support, DNS hostnames, private hosted zones, and endpoint private DNS.

  2. Routes: Identify the workload subnet and inspect its effective route table. Look for missing NAT or IGW routes, unexpected endpoint routes, incorrect peer routes, or a NAT dependency in another AZ.

  3. Permissions: Check workload security groups, both NACL directions, endpoint security groups, endpoint policies, destination firewalls, and listener ports.

  4. Static path: Use VPC Reachability Analyzer to analyze expected connectivity and identify a blocking configuration component. It is static configuration analysis, so it does not replace an application test.

  5. Observed traffic: Enable VPC Flow Logs at VPC, subnet, or ENI scope. Flow Logs contain traffic metadata, not packet payloads, and are not real-time packet capture.

  6. NAT health: For NAT-backed traffic, inspect CloudWatch metrics including ErrorPortAllocation, PacketsDropCount, connection counts, and byte or packet counters. AWS publishes NAT gateway metrics at one-minute intervals. See NAT gateway metrics.

  7. Application behavior: Use curl -v, nc, TLS diagnostics, and application logs. Do not treat a failed ICMP ping as proof that an interface endpoint is broken; AWS documents that interface endpoints do not respond to ping.

Follow the packet path first, and AWS networking becomes easier to explain, test, and operate.

Join 1,000+ engineers becoming better DevOps & SRE professionals.

Every week, I share:

  • How I'd approach problems differently (real projects, real mistakes)

  • Career moves that actually work (not LinkedIn motivational posts)

  • Technical deep-dives that change how you think about infrastructure

No fluff. No roadmaps. Just what works when you're building real systems.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.

Reply

Avatar

or to participate