📬 In Case You Missed This Week’s Uptime Sync
Every week, I curate the best DevOps, SRE, Cloud, Linux, database, and infrastructure reads so you don’t have to hunt for them yourself.
This week’s edition featured:
Why most rewrites solve an engineer’s problem, not a business problem
What it took to get Kubernetes running directly in a browser
Why the real operating cost usually starts after the demo
How to design DB partitions you don’t have to babysit
Getting started with Claude loops
Self-hosting MinIO as an S3-compatible object store
Building a minimal ZFS NAS without Synology, QNAP, or TrueNAS
Blue/green deployments on Kubernetes with Argo Rollouts
Projects like Ponytrail, DNSGlobe, Freno, Mcpsnoop, and SeekDB
🧠 Think Like an SRE
You are on-call.
A service is throwing more errors than usual, but only for one region.
Dashboards show CPU is normal. Memory is normal. Pods are healthy. No recent deployment from your team.
Someone says:
“Nothing changed on our side.”
What do you check next?
My first checks would be:
Recent config changes.
DNS resolution from that region.
Dependency latency.
Load balancer health.
TLS/certificate issues.
Cloud provider or network events.
Logs
Traffic pattern changes.
Never trust your assumptions, try to gather as much data as possible.
This one is a bit deeper and longer to read. Ensure you bookmark this article to revisit it later!
A Lambda function cannot upload a file to S3. A deployment fails with AccessDenied. Someone accidentally gives a developer admin access. An audit finds roles with permissions they never needed.
Until then, IAM feels like theoretical.
But every request made to AWS passes through IAM.
Starting an EC2 instance, reading an S3 object, updating DynamoDB, accessing Secrets Manager or deploying through CI/CD all depend on one decision:
Should AWS allow this request or deny it?
Once you understand how AWS makes that decision, IAM becomes much easier.

Table of Contents
What IAM Actually Does
IAM stands for Identity and Access Management.
It answers three basic questions:
Who is making the request?
What action are they trying to perform?
Which resource are they trying to access?
For example:
Who: A Lambda execution role
Action:
s3:PutObjectResource: A specific S3 bucket
AWS checks all applicable policies and then allows or denies the request.
IAM mainly handles authentication and authorization.
Authentication confirms who you are.
This can happen through:
A password
An access key
Temporary STS credentials
Federation through an identity provider
Authorization decides what you are allowed to do after AWS knows your identity.
Successfully logging in does not mean you have permission to perform actions.
An IAM user can log in successfully and still receive AccessDenied for every AWS API request.
The Two IAM Rules You Must Remember
You do not need to memorize every IAM feature on day one.
Start with these two rules.
Rule 1: Everything is denied by default
AWS starts with an implicit deny.
Unless a policy explicitly allows an action, AWS blocks it.
For example, creating a role does not automatically give it access to S3, EC2 or DynamoDB. You must explicitly allow those actions.
Rule 2: An explicit deny always wins
Suppose one policy allows S3 access, but another policy explicitly denies it.
The final result is deny.
It does not matter how many policies allow the action. One matching explicit deny is enough to block the request.
A useful troubleshooting method is:
Look for an explicit deny.
Check whether the action is explicitly allowed.
Confirm that no boundary, SCP or session policy limits the permission.
If nothing allows the action, AWS denies it by default.
The Main IAM Identities
IAM has four important identity types.
IAM Users
An IAM user is a long-term identity inside one AWS account.
It can have:
A console password
Access keys
Permissions attached directly or through groups
IAM users were commonly used for developers and applications in older AWS setups.
Today, AWS roles and temporary credentials are preferred.
Long-term IAM user credentials can be copied, leaked, forgotten or accidentally committed to Git.
IAM Groups
A group is simply a collection of IAM users.
For example, you may create groups such as:
Developers
ReadOnlyUsers
DatabaseAdmins
Instead of attaching the same policy to 20 users, you attach it once to the group.
Groups cannot log in, make API calls or be assumed. They only make user permission management easier.
IAM Roles
Roles are the preferred way to provide AWS access.
A role does not normally have permanent credentials. It provides temporary credentials when someone or something assumes it.
Roles are commonly used by:
EC2 instances
Lambda functions
ECS tasks
EKS workloads
GitHub Actions
Developers using IAM Identity Center
Applications accessing another AWS account
Temporary credentials expire automatically, which makes them safer than permanent access keys.
Root User
The root user is created when the AWS account is created.
It has full control over the account and can perform some actions that normal IAM identities cannot.
You should:
Enable MFA on the root user
Avoid creating root access keys
Store the credentials safely
Use root only for account-level emergencies or tasks that specifically require it
Do not use the root user for daily AWS work.
IAM Policies
IAM permissions are written as JSON policy documents.
A basic policy contains:
Effect: Allow or DenyAction: The AWS API actionResource: The resource the action applies toCondition: Optional rules that must be true
Example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-app-bucket/*"
}
]
}This policy allows reading objects from one specific S3 bucket.
It does not allow:
Uploading objects
Deleting objects
Reading other buckets
Managing the bucket configuration
That is what least privilege should look like.
Important Policy Types
There are several policy types in AWS, but juniors should first understand these five.
1. Identity-Based Policies
These policies are attached to users, groups or roles.
They define what that identity can do.
For example, a Lambda role may have permission to:
Read from one S3 bucket
Write to one DynamoDB table
Send logs to CloudWatch
Identity policies can be managed or inline.
Managed policies are reusable and can be attached to multiple identities.
Inline policies exist directly inside one user, group or role.
Managed policies are generally easier to reuse, audit and maintain.
2. Resource-Based Policies
These policies are attached directly to resources.
Common examples include:
S3 bucket policies
KMS key policies
SQS queue policies
SNS topic policies
IAM role trust policies
A resource policy usually answers:
Which principal can access this resource?
For example, an S3 bucket policy can allow a role from another AWS account to upload objects.
Resource policies are especially important for cross-account access.
3. Trust Policies
A trust policy is attached to an IAM role.
It controls who is allowed to assume that role.
A role can have permission to access S3, but nobody can use that role unless its trust policy allows them to assume it.
This is a common source of confusion.
For role assumption to work:
The caller must have permission to call
sts:AssumeRoleThe target role must trust the caller
Both sides must be configured correctly.
4. Permissions Boundaries
A permissions boundary sets the maximum permissions a user or role can receive.
It does not grant permission by itself.
Suppose a role policy allows S3 and EC2, but its permissions boundary only allows S3.
The role can use S3, but not EC2.
Think of a permissions boundary as a ceiling.
The role still needs an identity policy that allows the action. The boundary only limits how far those permissions can go.
Boundaries are useful when teams are allowed to create IAM roles but should not be able to grant unlimited access.
5. Service Control Policies
Service Control Policies, or SCPs, are applied through AWS Organizations.
They set permission limits for entire AWS accounts or organizational units.
For example, an SCP may:
Block resources outside approved AWS regions
Prevent CloudTrail from being disabled
Block public S3 bucket changes
Prevent member accounts from modifying security services
SCPs do not grant permissions.
A role may have AdministratorAccess, but an SCP can still block specific actions.
This is why a permission problem may not be visible when you only inspect the IAM role.
Long-Term and Temporary Credentials
AWS credentials generally fall into two categories.
Long-Term Credentials
IAM user access keys are long-term credentials.
They include:
Access key ID
Secret access key
These credentials remain valid until they are manually disabled or deleted.
That creates several risks:
They may be committed to Git
They may remain on an old laptop
They may be copied into a Docker image
They may not be rotated
Nobody may know which application still uses them
Access key IDs beginning with AKIA are usually long-term IAM user credentials.
Temporary Credentials
Temporary credentials are normally issued by AWS STS.
They contain:
Access key ID
Secret access key
Session token
Expiration time
Access key IDs beginning with ASIA are normally temporary credentials.
Temporary credentials are safer because they automatically expire.
Use temporary credentials wherever possible.
Common patterns include:
EC2 instance profiles
Lambda execution roles
ECS task roles
EKS Pod Identity or IRSA
GitHub Actions with OIDC
IAM Identity Center for employees
Cross-account role assumption
You should not place AWS access keys inside application code, Git repositories, Dockerfiles or Kubernetes manifests.
In most cases, the correct solution is to attach or assume a role.
How AWS Evaluates a Request
Imagine a Lambda function tries to read an object from S3.
AWS roughly follows this process.
Step 1: Authenticate the caller
AWS verifies the Lambda role's temporary credentials.
Step 2: Build the request context
AWS collects information such as:
Principal ARN
Requested action
Resource ARN
AWS account
Source IP
Region
Resource tags
MFA status
Session details
Step 3: Collect applicable policies
AWS checks relevant policies, including:
Identity policies
Resource policies
Permissions boundaries
SCPs
Session policies
KMS key policies when encryption is involved
Step 4: Look for an explicit deny
Any matching explicit deny blocks the request.
Step 5: Look for an allow
At least one applicable policy must allow the request.
The permission must also be inside the limits set by boundaries, SCPs and session policies.
Step 6: Deny if nothing allows it
When AWS finds no valid allow, the request is denied.
That is the basic IAM evaluation model.
Same-Account Access
Most workloads access resources inside the same AWS account.
The normal pattern is to attach permissions to the workload's role.
For example, suppose a Lambda function needs to:
Read files from one S3 bucket
Write records to one DynamoDB table
Send logs to CloudWatch
Its role should receive only those permissions.
Avoid broad policies such as:
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}This may quickly fix an AccessDenied error, but it gives the workload access to almost everything in the account.
A production policy should use:
Specific actions
Specific resources
Conditions where useful
Start small and add permissions based on actual requirements.
Cross-Account Access
Cross-account access involves two AWS accounts.
The most common method is role assumption.
Suppose Account A needs access to resources in Account B.
Account B creates a role with:
A trust policy allowing Account A to assume it
Permission policies defining what the role can do
The identity in Account A also needs permission to call sts:AssumeRole on that role.
Another method is a resource-based policy.
For example, an S3 bucket in Account B can directly allow a role from Account A to upload objects.
Cross-account problems usually happen because only one side was configured.
Always check:
Does the source identity have permission?
Does the destination trust the source?
Does the resource policy allow the source?
Does an SCP or explicit deny block the action?
Does KMS also allow access when encryption is enabled?
IAM for Human Access
Creating IAM users for every developer does not scale well.
Modern AWS environments normally use federation or IAM Identity Center.
Users authenticate through an identity provider such as:
Microsoft Entra ID
Okta
Google Workspace
IAM Identity Center
After authentication, they assume a role and receive temporary credentials.
This provides several benefits:
No permanent AWS access keys
Central employee access management
Easier onboarding and offboarding
Different roles for different accounts
Better auditing
Temporary sessions that expire
When someone leaves the company, disabling their identity provider account removes their AWS access.
You do not need to manually search every AWS account for IAM users and access keys.
IAM for CI/CD
CI/CD systems also need AWS access.
The unsafe method is storing an IAM user's access key as a CI/CD secret.
A better method is OIDC federation.
Platforms such as GitHub Actions can use OIDC to request temporary AWS credentials.
The flow is:
The CI/CD job proves its identity to AWS.
AWS checks the role's trust policy.
AWS provides temporary credentials.
The pipeline deploys using those credentials.
The credentials expire.
There is no permanent AWS secret to rotate or leak.
You can also restrict the role so that only a specific repository, branch or environment can assume it.
Least Privilege in Real Projects
Least privilege means giving an identity only the access it requires.
Instead of:
"Action": "s3:*",
"Resource": "*"Prefer:
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::my-app-bucket/uploads/*"Do not try to create the perfect policy immediately.
A practical process is:
Start with the smallest permissions you believe the workload needs.
Run the workload.
Read the
AccessDeniederror.Confirm which API action failed.
Add the specific permission.
Test again.
Remove permissions that are no longer used.
AWS tools such as IAM Access Analyzer, CloudTrail and service last-accessed information can help identify required and unused permissions.
How to Debug AccessDenied Errors
When an AWS request fails with AccessDenied, do not immediately attach AdministratorAccess.
Check the problem in this order.
1. Confirm the real identity
Find out which user or role is making the request.
From the CLI, you can run:
aws sts get-caller-identityDo not assume you are using the role you expected.
Your local terminal may be using an old profile, expired session or different AWS account.
2. Find the exact API action
The application may say it cannot access S3, but the failed action may actually be:
s3:GetObjects3:ListBuckets3:PutObjectkms:Decrypt
These actions require different permissions and sometimes different resource ARNs.
3. Check the resource ARN
The policy may allow the right action on the wrong resource.
For S3, the bucket ARN and object ARN are different.
Bucket:
arn:aws:s3:::my-bucketObjects inside the bucket:
arn:aws:s3:::my-bucket/*s3:ListBucket normally uses the bucket ARN.
s3:GetObject normally uses the object ARN.
4. Check identity policies
Confirm that the role or user has an applicable allow statement.
5. Check resource policies
Inspect the S3 bucket policy, KMS key policy, SQS policy or other resource policy.
6. Look for explicit denies
Check:
Identity policies
Resource policies
Permissions boundaries
SCPs
Session policies
Conditions based on IP, region, tags or MFA
The failed request may involve more than one AWS service.
An S3 object encrypted with KMS may require:
s3:GetObjectkms:Decrypt
Allowing S3 alone may not be enough.
8. Use CloudTrail
CloudTrail shows:
Which identity made the request
Which API action was called
Which resource was involved
When the request happened
Whether AWS allowed or denied it
CloudTrail is often the fastest way to stop guessing.
IAM Security Rules Worth Following
For most DevOps teams, these rules provide a strong starting point:
Enable MFA on the root account.
Avoid root access for daily work.
Use roles instead of IAM users where possible.
Prefer temporary credentials.
Never hardcode AWS access keys.
Use IAM Identity Center for employee access.
Use OIDC for CI/CD systems.
Avoid
"Action": "*"and"Resource": "*".Review old and unused permissions.
Use permissions boundaries when delegating IAM management.
Use SCPs for organization-wide guardrails.
Check KMS permissions when encrypted resources fail.
Use CloudTrail instead of guessing during access issues.
To Summarize
IAM becomes easier when you stop thinking of it as a large collection of JSON policies.
For every failed request, ask:
Who is making the request?
Which API action are they calling?
Which resource are they accessing?
Which policy allows it?
Is any policy explicitly denying it?
Is a boundary, SCP or session policy limiting it?
The final rules are simple:
No allow means deny.
Explicit deny always wins.
Roles are better than permanent access keys.
Permissions should be as narrow as practical.
Always debug the real identity, action and resource.
You do not need to memorize all of IAM. You need to understand how AWS reaches the final allow or deny decision.
Join 1,000+ engineers becoming better DevOps & SRE professionals.
Every week, I share:
How I'd approach problems differently (real projects, real mistakes)
Career moves that actually work (not LinkedIn motivational posts)
Technical deep-dives that change how you think about infrastructure
No fluff. No roadmaps. Just what works when you're building real systems.

👋 Find me on Twitter | Linkedin | Connect 1:1
Thank you for supporting this newsletter.
Y’all are the best.
