Every Tuesday - Deep dives, architecture lessons, and real engineering stories.

Every Saturday - The best DevOps, SRE, Cloud, AI, tools, tutorials, and projects from the week.

📬 Missed This Week’s Uptime Sync?

This week’s best DevOps, SRE, Cloud, Kubernetes, database, AI infrastructure, and production engineering reads:

  • How to grow beyond senior engineer

  • Monoliths vs microservices: the trade-offs that matter

  • How Cloudflare saved 100 TB of RAM with math and Rust

  • How the internet moves from packets to pixels

  • Deployment checklists that actually prevent incidents

  • Why Karpenter may not be reducing your cloud bill

Plus: hardened Linux sandboxes for AI agents, systemd Dynamic Users, Terraform observability, DeepSec, and tools for Kubernetes, databases, object storage, upgrades, and Go tracing.

How 2M+ Professionals Stay Ahead on AI

What’s the secret to staying ahead of the curve in the world of AI? Information. 

Luckily, you can join 2,000,000+ early adopters reading The Rundown AI — the free newsletter that makes you smarter on AI with just a 5-minute read per day.

A Linux server shows 180 MB of free memory.

Someone sees the number, assumes the machine is running out of RAM, drops caches, watches free jump, and considers the problem fixed.

The dashboard looks healthier.

The workload may now be slower because Linux has to rebuild the cache from storage.

And the original question was never answered: was the machine actually under memory pressure?

Consider this:

The number that looks scary is free: 297 MB.

The number I would start with is available: about 19.1 GB.

What I follow:

Start with available, not free. Then look for data that reclaiming memory is actually hurting the workload.

Low free memory is a description.

A memory problem starts when the system cannot satisfy demand cheaply enough within the allocation boundary that matters.

Think in terms of reclaimability

The useful mental model is not used vs unused.

It is how expensive this memory would be to reclaim.

Completely free pages are easy. Linux can allocate them immediately.

Clean file cache is usually reclaimable too, but reclaiming it has a cost. If the application needs that data again, Linux has to fetch it from storage.

Dirty file-backed pages are harder. They cannot simply be discarded because they have to be written back first.

Anonymous memory, such as process heaps and stacks, is harder again. There is no file backing it that Linux can reread later. Reclaim may require swap, and under severe pressure the system may eventually invoke the OOM killer.

This is a diagnostic mental model, not the kernel's literal reclaim sequence.

Real allocation behaviour also depends on things like zone watermarks, writeback, active references, fragmentation, NUMA locality, huge pages, pinned pages, and memory cgroups.

That distinction matters because a machine can appear to have plenty of memory and still fail a particular allocation.

And containers introduce another boundary entirely.

A workload can hit its cgroup limit while the host still has gigabytes available.

Start with available, not free

MemAvailable, shown as available by free, estimates how much memory can be given to new applications without swapping.

It considers free memory, reclaimable page cache and slab, while keeping memory the kernel expects to need for normal operation.

It is not a guarantee.

But it answers a much more useful question than free does:

Can this host absorb more work without disruptive memory pressure?

That is usually what the engineer staring at free -m wanted to know in the first place.

How I actually read free -m

Run:

free -m

The -m flag only changes the display unit. Exact formulas can vary by procps-ng and distribution version, so the local man page remains authoritative for a specific machine.

For normal host triage, I would read the output roughly in this order.

available

Start here.

Compare it with what the workload may need next, not with some generic rule like "keep 20% RAM free."

A database that may grow by several gigabytes needs a different margin from a stateless service that can be rescheduled elsewhere.

Capacity is workload-specific.

free

This is memory currently doing nothing.

Low free is normal on an active Linux system. Keeping RAM empty does not make the machine faster.

If Linux can use otherwise idle memory to avoid storage I/O, it usually should.

used

Do not treat this as the sum of every process's RSS.

On current procps-ng implementations, it is generally calculated as:

used = total - available

So if process RSS does not add up to used, that is not proof of a leak.

Shared mappings, page cache, tmpfs, and kernel memory are some of the reasons the comparison breaks down.

buff/cache

This is useful context, but it is not a pile of disposable RAM.

Some cached pages are clean and easy to reclaim.

Some are dirty and need writeback.

Some kernel caches are reclaimable.

Some memory is not.

That is why saying "we have 6 GB in cache, so Linux can instantly free 6 GB" is bad capacity math.

shared

This largely reflects Shmem, including tmpfs and shared-memory use.

A large number here may point toward /dev/shm, tmpfs mounts, or an application using shared-memory segments.

That is a different investigation from page cache growth.

Swap

Nonzero swap usage does not automatically mean the host is unhealthy.

Pages may remain swapped out long after a temporary pressure event.

What I care about more is active movement.

A static swap number tells you much less than sustained swap-in or swap-out while request latency is climbing.

A screenshot is weak evidence.

A trend is much stronger.

Cached memory is not wasted memory

This becomes obvious after someone drops it.

Imagine a service repeatedly reading the same files.

Linux keeps recently accessed file contents in page cache. Later reads can come from RAM instead of storage.

That memory may show up as cached or used, but it is not wasted. It is serving the workload.

Writes are different.

Dirty pages contain data that still needs to reach storage. Linux cannot reclaim them as cheaply as clean cache because writeback has to happen first.

This is also why MemAvailable is more useful than simply adding free + cache.

Not every cached byte is equally reclaimable.

And not every reclaimed byte is free from performance cost.

I would not diagnose a memory leak from a rising buff/cache number.

I would first ask what is actually growing:

  • anonymous process memory

  • tmpfs or shared memory

  • dirty pages waiting on storage

  • reclaimable kernel caches

  • unreclaimable kernel memory

Those have different owners and very different failure modes.

Dropping caches usually fixes the wrong problem

The familiar command is:

echo 3 > /proc/sys/vm/drop_caches

It requires sufficient host privilege and may be blocked in containers or managed environments.

More importantly, it is not routine memory maintenance.

Dropping caches makes free rise and buff/cache fall.

That feels like progress because the graph moves in the direction you expected.

But what really changed is cache locality.

If the workload needs the same data again, Linux has to read it from storage and rebuild those caches. Repeated cache dropping can therefore increase both I/O and CPU work.

It also does not fix the problems people often hope it fixes:

  • A growing heap still grows.

  • Dirty pages still need writeback.

  • A cgroup limit is still a cgroup limit.

  • A leaking process still leaks.

  • Fragmentation and pinned memory do not disappear because file cache was removed.

There are legitimate reasons to drop caches.

A controlled benchmark may need a cold-cache run. A diagnostic experiment may intentionally isolate cache state.

Those are experiments with a clear purpose.

I would not put cache dropping on a cron job. It makes the screenshot look reassuring while leaving the capacity problem untouched.

First check if you actually have memory pressure

If available is declining, the next question is not immediately "what is leaking?"

The next question is:

Is the workload paying for memory pressure yet?

Look for rising latency, memory PSI, sustained swapping, writeback stalls, allocation failures, or OOM activity.

Start with a time series:

vmstat 1

On common procps implementations, si and so show swap-in and swap-out rates.

One nonzero sample means very little.

Sustained movement while request latency or job duration gets worse is much more interesting.

Then look at what is consuming the memory you expected to be reclaimable:

grep -E 'MemAvailable|MemFree|Cached|SReclaimable|SUnreclaim|Shmem|Dirty|AnonPages|Slab' /proc/meminfo

Do not collect those numbers just to create a bigger dashboard.

Use them to choose the next investigation.

If AnonPages keeps growing, look at application memory, allocators, and process behaviour.

If Shmem is large, inspect tmpfs and shared-memory users.

If Dirty is high, start thinking about writeback and storage.

If SUnreclaim is climbing, that points toward kernel memory, not automatically an application heap leak.

That branch is far more useful than asking whether "memory usage" is high.

There is no single owner of used RAM.

A healthy node can still have an OOM-killed pod

This is where host-level thinking breaks down.

A Kubernetes pod can be OOM-killed while the node still reports substantial MemAvailable.

The node had room.

The pod's cgroup did not.

For a container memory incident, inspect the workload's own usage, limit, event counters, and memory pressure before trusting node-wide free.

With cgroup v2, files such as:

memory.current
memory.stat
memory.events

are often more relevant than the host summary.

The exact metrics and semantics differ under cgroup v1 and managed platforms, so establish what environment you are actually running before applying a copied debugging checklist.

This is also why available should never be treated as a promise.

It is a host-level estimate.

It cannot override a container limit or guarantee that every allocation will succeed.

What I check in free -m

Do not clear caches just because free is low.

Do not restart a process just because it looks large.

Instead:

  1. Start with available for host-level headroom.

  2. Look at the trend, not one sample.

  3. Check whether the workload is actually experiencing memory pressure.

  4. Identify what kind of memory is growing.

  5. If the workload is containerized, inspect its cgroup before trusting host-level numbers.

  6. Preserve useful cache unless you are deliberately running a cold-cache experiment.

Join 1,000+ engineers becoming better DevOps & SRE professionals.

Every week, I share:

  • How I'd approach problems differently (real projects, real mistakes)

  • Career moves that actually work (not LinkedIn motivational posts)

  • Technical deep-dives that change how you think about infrastructure

No fluff. No roadmaps. Just what works when you're building real systems.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.

Reply

Avatar

or to participate