The Uptime Engineer
👋 Hi, I am Yoshik Karnawat
Don’t scroll past, this week’s 180 second read might just save you hours at work.
Every time you run cat file.txt, open a database connection, or accept an HTTP request, you're using file descriptors.
They're the invisible integers connecting your process to every I/O resource on the system - files, sockets, pipes, devices, everything.
But here's what most engineers don't understand:
File descriptors aren't just numbers. They're handles to kernel-managed resources that enforce isolation, enable concurrency, and make Linux's "everything is a file" philosophy actually work.
This is the foundation underneath every I/O operation and understanding it explains why epoll works, why processes can't corrupt each other's data, and why closing a socket from one thread doesn't break another.
Layer 1: File Descriptors
A file descriptor is a non-negative integer that uniquely identifies an open I/O resource within a process.
When you call open(), socket(), or pipe(), the kernel returns a file descriptor, usually the smallest unused integer starting from 3.
Every process has three file descriptors by default:
FD | Name | Meaning |
|---|---|---|
0 | stdin | Standard input (keyboard) |
1 | stdout | Standard output (terminal) |
2 | stderr | Standard error (terminal) |
Any FD ≥ 3 represents something you explicitly opened: a file, socket, pipe, or device.
Why integers?
File descriptors are integers because they're indexes into a per-process table maintained by the kernel.
Your process doesn't have direct access to the file. It only has the handle. The kernel does the actual I/O on your behalf.
This separation is critical for:
Security: Processes can't corrupt each other's files
Isolation: Each process has its own FD table
Abstraction: Same interface for files, sockets, pipes, devices
Layer 2: The Kernel Tables
When you open a file, the kernel doesn't just return a number and forget about it.
Three kernel tables work together to manage I/O:
1. Per-Process File Descriptor Table
Each process maintains its own FD table.
Process A FD Table:
FD 0 = points to stdin
FD 1 = points to stdout
FD 2 = points to stderr
FD 3 = points to /var/log/app.log
FD 4 = points to socket connectionKey property: FD numbers are process-local.
Process A's FD 3 and Process B's FD 3 can point to completely different files.
2. System-Wide Open File Table
The kernel maintains a global table of all open files.
Each entry contains:
File offset (current read/write position)
Access mode (read, write, read-write)
Status flags (O_NONBLOCK, O_APPEND, etc.)
Reference count (how many FDs point to this entry)
Why a separate table?
Multiple file descriptors can share the same open file entry.
When a process calls fork(), parent and child share the same file table entry meaning they share the same file offset.
3. Inode Table
The inode table describes the actual underlying files.
This contains:
File type (regular file, directory, device, socket)
Permissions
Size
Location on disk (data blocks)
Multiple open file entries can point to the same inode.
Example: Two processes open /var/log/app.log independently they get separate file table entries (independent offsets), but both point to the same inode.
The Three-Table Lookup
Here's what happens when you call read(fd, buffer, size):
Kernel looks up fd in the process's FD table = Gets pointer to open file table entry
Kernel reads open file table entry = Gets current file offset and access mode
Kernel follows pointer to inode table = Gets actual file location on disk
Kernel reads data from disk into buffer = Updates file offset in open file table
Returns number of bytes read to userspace
This three-layer design enables:
Process isolation: Each process has its own FD namespace
Resource sharing: Multiple FDs can share file state
Efficient inheritance:
fork()shares file tables without copying
Layer 3: System Calls
Your application code runs in userspace, a restricted environment with no direct hardware access.
The kernel runs in kernel space with full hardware privileges.
System calls are the controlled bridge between them.
How a system call works:
When you call read(fd, buffer, size):
Step 1: Userspace prepares the call
Your code makes the function call.
Step 2: CPU switches to kernel mode
This is called a context switch. Registers are saved. CPU privilege level changes. Execution jumps to kernel code.
Step 3: Kernel validates the request
Is fd valid for this process? Does the process have read permission? Is the buffer address valid?
Step 4: Kernel performs the I/O
Reads from disk, socket, pipe, or device.
Step 5: Kernel copies data to userspace buffer
Data moves from kernel memory = your process's memory.
Step 6: CPU switches back to userspace
Registers restored. Execution returns to your code. Return value (bytes read or -1 for error) is set.
Why this boundary exists:
Security: Processes can't directly access hardware or other processes' memory
Stability: A buggy process can't crash the kernel
Abstraction: Same syscall interface works for files, sockets, pipes, devices
Common System Calls for I/O
Creating file descriptors:
open() for files, socket() for network connections, pipe() for inter-process communication. Each returns a file descriptor.
Reading and writing:
read(fd, buffer, size) and write(fd, buffer, size) work on any file descriptor - files, sockets, pipes, devices.
Manipulating file descriptors:
dup2() duplicates FDs (used for redirection), close() frees resources, fcntl() sets modes like non-blocking I/O.
Multiplexing (monitoring multiple FDs):
select(), poll(), and epoll_wait() let one thread monitor thousands of file descriptors simultaneously.
This is what makes high-concurrency servers possible.
Everything Is a File: How Linux Unifies I/O
In Linux, everything is a file descriptor:
Resource | File Descriptor? |
|---|---|
Regular files | Yes |
Directories | Yes |
Sockets (network) | Yes |
Pipes (IPC) | Yes |
Devices (/dev/null) | Yes |
Terminals (/dev/tty) | Yes |
Why this matters:
The same read() and write() calls work on all of them.
This unification is what makes I/O multiplexing (select, poll, epoll) possible - you can monitor files, sockets, and pipes using the exact same mechanism.
File Descriptor Limits
Every process has limits on how many file descriptors it can open.
Check current limits: ulimit -n shows soft limit (typically 1024), ulimit -Hn shows hard limit (typically 4096 or more).
Check system-wide limit: cat /proc/sys/fs/file-max shows total FDs for entire system.
Why this matters in production:
If your server handles 10,000 concurrent connections, each connection needs at least one file descriptor (the socket).
If your ulimit -n is 1024, you'll hit the limit at 1,024 connections.
Fix:
Temporarily: ulimit -n 65536
Permanently: Edit /etc/security/limits.conf and add:
* soft nofile 65536
* hard nofile 65536This is why high-concurrency servers (Nginx, Redis) document FD limits in their setup guides.
To conclude:
File descriptors (the handle your process sees)
Kernel tables (how Linux tracks resources)
System calls (the userspace-kernel bridge).
Thanks for reading!