Linux Load Average, I/O Wait and Steal Time Explained

In partnership with

Both machines report 100% cpu. One completes the same job faster than the other. How do we figure out what each machine is starved for.

We take the same job and run it on two machines. They both say they have 2 vCPU available. We use the same code at the same commit and run the same commands.

Machine one completes in less than a minute. Machine two completes in around half as long again.

We check top on both machines and notice the cpu is sitting around 100%. Both of these machines were working as hard as they could. One of them was able to complete more work though.

If you have ever seen a high cpu or load average of 4 on a box and didnt know what to do next. Heres why. They are both reporting true values. Its just not the whole picture.

By the end of this article you will know how to identify if a linux machine is starved for cpu, starved for disk io or starved for the underlying host. You will also know how many vCPU's would fix your problem if you added more.

A vCPU on most cloud providers and ci machines will be a thread on a shared host machine. With most x86 chips today they are dual threaded (hyperthreading). So a box with 2 vCPU's is most likely one physical core that is split into two threads. Those two threads will share resources on the core.

There are some arm chips that will allocate 1 core per vCPU. So when you see 2 vCPU on a machine it may not give you the same amount of power.

This is where our two build boxes can start to vary. They could be running on a newer chip with more clock speed and larger cache allowing each cpu cycle to do more work. Or they could be sharing their host with other workloads. Maybe they are writing to local disk and the other is using network attached storage. The amount of vCPU's you have only tells you how many threads can execute at the same time. Not how powerful each one is.

CPU usage is measured by time. There are multiple counters the kernel tracks for each cpu and classifies time into one of these categories. Programs like top will display how much of the last sample you spent in each category.

So when you see 100% cpu that just means the cpu wasnt idle. It doesnt say how much work was done in those seconds. 100% on a slower cpu and 100% on a faster cpu are the same.

These are the categories that will show up on your %Cpu(s) line in top. They will also show up in vmstat.

There are a couple of these that will trip people up.

IO Wait is idle time. If you have 40% io wait your cpu is not working on disk. Your cpu is doing nothing and the kernel is telling you that a task is waiting on io. If you have another task that is keeping your cpu busy at the same time your io wait will not show up. So you can have a low io wait and still have disk problems. Well cover disk later.

Steal is when another virtual machine used your cpu. This will only show up on virtual machines. On bare metal it will always be 0. If your steal is at 25% then your virtual cpu wanted to run but was only allowed to run 75% of the time. Some machine types will throttle you once you use all of your cpu credits. These may also show up as steal.

Any given task (process or thread) will be in one of these states. Its either executing on a cpu or its ready to execute but is waiting in the runqueue. Or it could be in an un-interruptable sleep. These will show up as D in ps or top. Usually because its waiting on a disk or network drive to return data. It will not respond to signals until its io has completed. Even a kill -9 wont work. If its sleeping on something like a timer it will not be considered.

A task will move between these states like so

When you type uptime you will see three numbers

These are the number of tasks that are running or ready to run over the past 1, 5 and 15 minutes. They are averaged with an exponential decay so more recent seconds will have more weight. You can see a spike show up in the 1 minute average and eventually fall off in the 15 minute average. In this example we have 3.91, 2.40 and 1.12 so our load has been increasing over the past few minutes.

Load average means nothing by itself. You need to know how many cpus you have

A load average of 3.91 on 2 cpus means on average we had 4 tasks that wanted a cpu or were waiting on io. If we had 16 cpus a load average of 3.91 would be very low.

One thing that makes linux different from other unix based systems is that load average includes tasks that are in a D state. So you can have a load average of 8 on a machine with 2 cpus and see your cpu is only 70% busy. In this example you are not starved for cpu. You have 6 tasks that are waiting on your disk. A high load average just means something is waiting in a queue. You still need to figure out what queue.

top, vmstat and uptime are all in the procps package which most systems should have installed. mpstat, iostat and pidstat are in the sysstat package (apt install sysstat). Depending on your version of these programs the columns may change a bit. Make sure to look at the header of your program to understand what you are looking at.

vmstat 1 is a great place to start. Below is our fast machine while it is compiling.

Ignore the first line of data when you are watching vmstat live. It shows you stats since boot. r is how many tasks are runnable. Meaning they are running or waiting for a cpu. b is how many are in D state. The last 5 columns are the cpu stats we talked about earlier. We have 4-5 tasks running our compiler and they are all userspace time. No wait on disk and no steal.

mpstat -P ALL 1 will give you the same stats but broken down by cpu. This is helpful if you see one cpu pegged and another that is not.

iostat -x 1 will give you disk stats. Below is a snippet of the output.

Await is how many milliseconds it took for an io to complete. Including how long it spent waiting in the queue. aqu-sz is the amount of requests that were queued up. %util is the percentage of time that disk was being used. On a mechanical disk 100% would mean its maxed out. SSD's and nvme drives can handle multiple requests at the same time so 100% just means its busy. Look at the await times and queue size to determine if your disk is overloaded.

pidstat 1 will show you which process is taking up the time.

On newer versions of sysstat you will see a wait column. This means your process is waiting on a cpu to become available. This is a good indicator that you are starved for cpu. pidstat -d 1 will show you which processes are doing io.

/proc/pressure/ (kernel 4.20+) will show you how much time your processes are being stalled on a given resource.

If you see some avg10=48.20 on cpu that means your process was waiting on a cpu for 48% of the last 10 seconds. Full means that all processes were waiting on io.

Going back to our build that was slower on one machine than the other. There are a few things that could be happening.

Starved for CPU

Your vmstat will show r higher than your cpu count. Us will be high and your io wait and steal will be 0. Your pidstat will show processes with a high wait column. /proc/pressure/cpu will also be high. Adding more cpu's or getting a faster cpu will solve this. As long as your workload can run in parallel.

Starved for Disk

Your load average will be high. But your cpu usage will be lower. You will see your b column increase in vmstat along with your wa. When you check your disk with iostat -x you will see the await times and queue size increase. This is what would happen if your build was writing many small files to a slow disk. To find what tasks are waiting on your disk run

ps -eo allows you to print certain columns from ps. In this case we want to see the state of the process, the pid and the command. The awk will filter for any process in a D state. Adding cpu's will not help you here. Your tasks are not waiting on cpu. You would want to add a faster disk. Reduce the amount of io your process is doing or have enough memory to hold your data in the page cache.

Starved for Host

If your mpstat shows a high steal column you are starved for your underlying host. Maybe your workload is ready to run but your host is allowing other processes to use your cpu. Changing your workload will not fix this. You would need a different host or machine type.

Well if we saw our steal column was around 25% then our cpu is only running 75% of the time. If our steal column was 0 and us was 100 then we have a slower cpu on one machine than the other. Just run vmstat 1 on both machines and you will see the difference.

Check your load average against the number of cpu's you have. If its higher you have something waiting in a queue.

Check vmstat or mpstat to see if you have high userspace/system time, io wait or steal. See if you have more tasks ready to run than cpu's you have.

Check what process is causing the problem with pidstat. Or use ps to find tasks that are in a D state.

Fix the problem. If your userspace is high you may need more cpu's. If your system time is high maybe you are making too many system calls or switching context too often. Io wait and D state tasks point to disk problems. Steal means your host.

Adding vCPU's will help if your workload can run in parallel. And you are already queueing up. Something like a compile that can run many jobs at once.

It will not help if your workload is single threaded. You will see one cpu at 100% and the rest idle in mpstat -P ALL. You would need a faster cpu or change your code to be parallel. Adding cpu's also will not help if you are io bound. Your tasks are in a D state and are not waiting on cpu.

Another scenario is if you are waiting on locks in your application. Adding more cpu's will just give you more processes that are waiting on a lock. You will see your system time go up and possibly your cs (context switches) column in vmstat.

Find a vm or laptop you can play with. Do not do this on a production server. Open two terminal windows. Run vmstat 1 in the first window.

In the second window overload your cpu with

This will start (number of cpu's + 2) processes that print y to stdout. We redirect that to /dev/null so its just using cpu. The & will put each process in the background. You should see your r column increase past your cpu count. And your us column shoot up. After a minute check your uptime and load average against your number of cpu's. Kill all of your processes with

Now overload your disk with

This will write 2gb of zeros to a file. Of 1mb blocks. By using oflag=direct we bypass the page cache and force the io to hit the disk. Note this may not work if you are on a ramdisk like tmpfs. Try to do this in a normal directory. You should see your wa and b column increase. Also check iostat -x 1 in a third window to see your disk stats. Your load average will go up but your cpu should be low.

See how both of these scenarios had high load averages. But they had very different issues. Next time you see a high cpu check what your processes are waiting on.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.