Every Tuesday - Deep dives, architecture lessons, and real engineering stories.
Every Saturday - The best DevOps, SRE, Cloud, AI, tools, tutorials, and projects from the week.
📬 Missed This Week’s Uptime Sync?
This week’s best DevOps, SRE, Cloud, Kubernetes, database, AI infrastructure, and production engineering reads:
What happened in the eMudhra intermediate CA compromise
Inside the compromised LiteLLM v1.107.0 release
Why Anthropic built its own source control system
How DigitalOcean built microVMs that pause and resume in milliseconds
How LinkedIn traces user journeys with OpenTelemetry
What went wrong in Control D’s October DNS outage
Plus: the risks of ON DELETE CASCADE, AI agents in CI/CD, Karpenter on EKS with Terraform, Azure cost cuts, migrating from AWS to GKE, giving AI agents useful context, Blackbox Exporter probes and alerts, and throwaway apps with Dokku.
Most posts about AI careers focus on building: agents, LLM fundamentals, launching an AI app. Those skills matter. But someone also has to keep those apps running, and I think DevOps and SRE engineers should look at that side more closely.
Who runs the infrastructure behind these apps? Someone deploys the models. Someone works out why inference is slow when every Pod looks healthy. Someone manages GPU capacity, scaling, monitoring, networking, failures, and the bill at the end of the month.
That's the work I find interesting. I work in SRE, and as AI systems grow, I think AI infrastructure is a strong specialization for engineers who already understand production systems.
Keep your DevOps skills. The field doesn't guarantee a high salary, and you won't master it by learning a few new tools. It's a direction worth exploring.
What is an AI infrastructure engineer?
Start with a normal backend service. You deploy it behind a load balancer, set up autoscaling, collect metrics, watch latency, and debug failures. Along the way you use Linux, containers, Kubernetes, cloud services, networking, and observability tools.
Now replace the REST API with an LLM. Many of the responsibilities stay the same, but the workload has different constraints:
The model may need a GPU with enough memory to hold it.
Request costs vary widely. A long prompt or a long answer uses far more compute than a short one.
You may need to manage request queues, batch requests together, tune the serving software, and decide whether an extra GPU lowers latency enough to justify its price.
AI infrastructure is the work of building and running the systems that train, deploy, serve, and support these workloads.
The job comes in several shapes. Some engineers build platforms for training models. Others focus on inference (serving models to users), GPU clusters, distributed storage, fast networking, or developer platforms. Many use managed cloud services instead of owning GPUs.
This differs from ML engineering, which centers on model development and quality, and from MLOps, which covers data pipelines, experiments, registries, and model lifecycle. The titles overlap a lot, so read the responsibilities in a job posting rather than relying on the title.
Why DevOps and SRE engineers have a head start
You don't start from zero. If you've debugged Linux, run Kubernetes, chased down latency, configured networks, built deployment pipelines, or handled incidents, you already know many of the problems AI platforms have.
What you know | Where it applies in AI infrastructure |
|---|---|
Linux and containers | GPU drivers, runtimes, node images, process debugging |
Kubernetes | GPU scheduling, deployments, node pools, rollouts |
Networking | Model-serving APIs, large data transfers, distributed workloads |
Observability | Inference latency, request queues, GPU utilization, capacity |
SRE fundamentals | Reliability, incident response, capacity planning, cost trade-offs |
You still have to learn new material. GPU memory, CUDA compatibility, model serving, batching, KV caches, and distributed inference don't become familiar just because you know Kubernetes. The mindset does carry over.
Take GPUs on Kubernetes. Kubernetes can schedule GPUs through vendor device plugins, which expose resources like nvidia.com/gpu. If you've set resource requests and limits before, that part looks familiar. The harder problems come after: installing compatible drivers, keeping accelerators healthy, and getting good utilization out of them. The Kubernetes documentation explains the scheduling side clearly.
Your job is to learn what changes when the workload runs on accelerators.
Data is part of the job
Training and serving both depend on data moving reliably and quickly. That makes data pipelines an SRE concern too.
Slow data loading leaves GPUs idle. A training job waiting on storage or preprocessing still pays for the GPU. Watch data throughput and GPU utilization together. Low utilization on healthy GPUs often points to the input pipeline.
Vector databases in RAG. Retrieval-augmented generation (RAG) stores document embeddings in a vector index and pulls relevant chunks into the prompt. The index is a stateful service. It needs backups, latency monitoring, and a plan for rebuilding it when the embedding model changes.
Freshness and lineage. If a pipeline that feeds a retrieval index or training set fails silently, the system keeps answering from stale data. Track when each data source last updated, and alert on pipeline failures the same way you alert on service failures.
A production example
Say you're on call for an AI chatbot. The model Pods are running. CPU looks fine, error rates are low, and users still say the bot takes 20 seconds to start replying.
For a typical web service, you'd check database queries, slow dependencies, thread pools, or overloaded instances. Those checks still make sense here, but inference adds new bottlenecks.
Requests may wait in a queue because the GPU can't keep up. Long prompts may use up memory. The KV cache, which stores attention state for active requests, may be full. The model may simply be too big for the hardware and the traffic.
To debug these, you need a few terms:
Time to first token (TTFT): how long the user waits before the first word appears. This is often the number users feel most.
Queue time: how long a request waits before the GPU starts working on it.
Inter-token latency: the gap between words once the response starts streaming.
GPU memory usage: how much of the GPU's memory the model and cache are using.
Serving systems such as vLLM export metrics for these. The fix might be batching, admission limits (rejecting or delaying requests when the system is full), a smaller model, better capacity planning, or more hardware. Adding replicas blindly can raise your bill without fixing the bottleneck, because the limit may be memory or queueing rather than the number of Pods.
This is still production engineering. It just comes with a new set of systems and failure modes.
Are companies hiring for this?
There's a reason to take it seriously, though I wouldn't read every AI hiring headline as proof of a boom.
Cisco's 2025 AI Workforce Consortium report named AI Infrastructure Engineer among the fastest-growing ICT role labels across G7 countries, with 124% year-over-year growth in demand for that label. That's a directional signal. It doesn't tell you how many openings exist near you or what they pay. Read the report.
Companies are building serious infrastructure too. Modal has written about GPU health and automated recovery across its fleet. CoreWeave and the major cloud providers are investing in the capacity and software needed to run AI workloads at scale. That work spans infrastructure operations, platform engineering, inference serving, cluster reliability, storage, networking, security, and cost management.
Keep one caveat in mind: not every company running an AI app will hire someone to run GPUs. A startup that calls a hosted model API needs strong backend and SRE skills, but probably not its own GPU scheduler. Cloud and inference providers handle much of that. Some specialist roles expect distributed systems depth, GPU experience, or an ML background.
So search past the exact title. Look for Inference Engineer, ML Platform Engineer, AI Platform Engineer, GPU Infrastructure Engineer, SRE (AI Platform), or plain Platform Engineer. What matters is what the team operates.
Is it better than traditional DevOps?
You don't have to pick one. Companies adding AI features still need networking, databases, deployment pipelines, security, observability, and people who can debug production. AI workloads need all of that too.
The risk is chasing one trendy tool and neglecting the fundamentals. Frameworks change, model architectures change, managed services take over some tasks, and GPU investment cycles can shift hiring quickly.
Ask yourself what work you enjoy. If you like production debugging, performance tuning, distributed systems, capacity planning, and messy reliability problems, AI infrastructure may suit you. If your fundamentals are shaky, fix them first. You'll need them either way.
My bet is on becoming an infrastructure engineer who understands AI workloads, not an engineer who only knows AI tools.
Build a small inference service and find its first bottleneck
Run a small model you're allowed to use with llama.cpp on your machine.
Expose an API endpoint and send it short and long prompts.
Compare latency at low and higher concurrency.
Add basic metrics and a dashboard, or at least a results table.
Add a concurrency limit and watch what changes.
Write a README that explains the bottleneck, your measurements, and the trade-offs.
Publish the code and findings on GitHub. State your hardware and test conditions. If you ran on CPU, say so.
You'll have something concrete to talk about in interviews, and you'll learn whether you enjoy this kind of work before you spend more time or money on it.
Should you move into AI infrastructure?
My answer: explore it, but don't rebrand yourself yet.
If you have a couple of years in DevOps, cloud, or SRE, you may already have a head start. Learn how AI workloads behave, try serving a model, learn the GPU stack, and read real job descriptions closely.
If you're just starting out, learn Linux, networking, Kubernetes, and debugging first. Those skills give you more options.
Don't assume AI infrastructure is an automatic route to a higher salary. The good roles are competitive, and specialist teams often expect deep systems experience.
Still, this is a career opportunity our side of engineering should pay more attention to. Plenty of people are learning to build AI applications. Fewer are looking at the engineers who keep those applications fast, reliable, and affordable once real users arrive.
👋 Find me on Twitter | Linkedin | Connect 1:1
Thank you for supporting this newsletter.
Y’all are the best.