📬 In Case You Missed This Week’s Uptime Sync

Every week, I curate the best DevOps, SRE, Cloud, Linux, database, and infrastructure reads so you don’t have to hunt for them yourself.

This week’s edition featured:

  • Why most rewrites solve an engineer’s problem, not a business problem

  • What it took to get Kubernetes running directly in a browser

  • Why the real operating cost usually starts after the demo

  • How to design DB partitions you don’t have to babysit

  • Getting started with Claude loops

  • Self-hosting MinIO as an S3-compatible object store

  • Building a minimal ZFS NAS without Synology, QNAP, or TrueNAS

  • Blue/green deployments on Kubernetes with Argo Rollouts

  • Projects like Ponytrail, DNSGlobe, Freno, Mcpsnoop, and SeekDB

🧠 Think Like an SRE

You are on-call.

A service is throwing more errors than usual, but only for one region.

Dashboards show CPU is normal. Memory is normal. Pods are healthy. No recent deployment from your team.

Someone says:

“Nothing changed on our side.”

What do you check next?

My first checks would be:

Recent config changes.
DNS resolution from that region.
Dependency latency.
Load balancer health.
TLS/certificate issues.
Cloud provider or network events.
Logs
Traffic pattern changes.

Never trust your assumptions, try to gather as much data as possible.

This one is a bit deeper and longer to read. Ensure you bookmark this article to revisit it later!

A Lambda function cannot upload a file to S3. A deployment fails with AccessDenied. Someone accidentally gives a developer admin access. An audit finds roles with permissions they never needed.

Until then, IAM feels like theoretical.

But every request made to AWS passes through IAM.

Starting an EC2 instance, reading an S3 object, updating DynamoDB, accessing Secrets Manager or deploying through CI/CD all depend on one decision:

Should AWS allow this request or deny it?

Once you understand how AWS makes that decision, IAM becomes much easier.

Table of Contents

What IAM Actually Does

IAM stands for Identity and Access Management.

It answers three basic questions:

  1. Who is making the request?

  2. What action are they trying to perform?

  3. Which resource are they trying to access?

For example:

  • Who: A Lambda execution role

  • Action: s3:PutObject

  • Resource: A specific S3 bucket

AWS checks all applicable policies and then allows or denies the request.

IAM mainly handles authentication and authorization.

Authentication confirms who you are.

This can happen through:

  • A password

  • An access key

  • Temporary STS credentials

  • Federation through an identity provider

Authorization decides what you are allowed to do after AWS knows your identity.

Successfully logging in does not mean you have permission to perform actions.

An IAM user can log in successfully and still receive AccessDenied for every AWS API request.

The Two IAM Rules You Must Remember

You do not need to memorize every IAM feature on day one.

Start with these two rules.

Rule 1: Everything is denied by default

AWS starts with an implicit deny.

Unless a policy explicitly allows an action, AWS blocks it.

For example, creating a role does not automatically give it access to S3, EC2 or DynamoDB. You must explicitly allow those actions.

Rule 2: An explicit deny always wins

Suppose one policy allows S3 access, but another policy explicitly denies it.

The final result is deny.

It does not matter how many policies allow the action. One matching explicit deny is enough to block the request.

A useful troubleshooting method is:

  1. Look for an explicit deny.

  2. Check whether the action is explicitly allowed.

  3. Confirm that no boundary, SCP or session policy limits the permission.

  4. If nothing allows the action, AWS denies it by default.

The Main IAM Identities

IAM has four important identity types.

IAM Users

An IAM user is a long-term identity inside one AWS account.

It can have:

  • A console password

  • Access keys

  • Permissions attached directly or through groups

IAM users were commonly used for developers and applications in older AWS setups.

Today, AWS roles and temporary credentials are preferred.

Long-term IAM user credentials can be copied, leaked, forgotten or accidentally committed to Git.

IAM Groups

A group is simply a collection of IAM users.

For example, you may create groups such as:

  • Developers

  • ReadOnlyUsers

  • DatabaseAdmins

Instead of attaching the same policy to 20 users, you attach it once to the group.

Groups cannot log in, make API calls or be assumed. They only make user permission management easier.

IAM Roles

Roles are the preferred way to provide AWS access.

A role does not normally have permanent credentials. It provides temporary credentials when someone or something assumes it.

Roles are commonly used by:

  • EC2 instances

  • Lambda functions

  • ECS tasks

  • EKS workloads

  • GitHub Actions

  • Developers using IAM Identity Center

  • Applications accessing another AWS account

Temporary credentials expire automatically, which makes them safer than permanent access keys.

Root User

The root user is created when the AWS account is created.

It has full control over the account and can perform some actions that normal IAM identities cannot.

You should:

  • Enable MFA on the root user

  • Avoid creating root access keys

  • Store the credentials safely

  • Use root only for account-level emergencies or tasks that specifically require it

Do not use the root user for daily AWS work.

IAM Policies

IAM permissions are written as JSON policy documents.

A basic policy contains:

  • Effect: Allow or Deny

  • Action: The AWS API action

  • Resource: The resource the action applies to

  • Condition: Optional rules that must be true

Example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-app-bucket/*"
    }
  ]
}

This policy allows reading objects from one specific S3 bucket.

It does not allow:

  • Uploading objects

  • Deleting objects

  • Reading other buckets

  • Managing the bucket configuration

That is what least privilege should look like.

Important Policy Types

There are several policy types in AWS, but juniors should first understand these five.

1. Identity-Based Policies

These policies are attached to users, groups or roles.

They define what that identity can do.

For example, a Lambda role may have permission to:

  • Read from one S3 bucket

  • Write to one DynamoDB table

  • Send logs to CloudWatch

Identity policies can be managed or inline.

Managed policies are reusable and can be attached to multiple identities.

Inline policies exist directly inside one user, group or role.

Managed policies are generally easier to reuse, audit and maintain.

2. Resource-Based Policies

These policies are attached directly to resources.

Common examples include:

  • S3 bucket policies

  • KMS key policies

  • SQS queue policies

  • SNS topic policies

  • IAM role trust policies

A resource policy usually answers:

Which principal can access this resource?

For example, an S3 bucket policy can allow a role from another AWS account to upload objects.

Resource policies are especially important for cross-account access.

3. Trust Policies

A trust policy is attached to an IAM role.

It controls who is allowed to assume that role.

A role can have permission to access S3, but nobody can use that role unless its trust policy allows them to assume it.

This is a common source of confusion.

For role assumption to work:

  • The caller must have permission to call sts:AssumeRole

  • The target role must trust the caller

Both sides must be configured correctly.

4. Permissions Boundaries

A permissions boundary sets the maximum permissions a user or role can receive.

It does not grant permission by itself.

Suppose a role policy allows S3 and EC2, but its permissions boundary only allows S3.

The role can use S3, but not EC2.

Think of a permissions boundary as a ceiling.

The role still needs an identity policy that allows the action. The boundary only limits how far those permissions can go.

Boundaries are useful when teams are allowed to create IAM roles but should not be able to grant unlimited access.

5. Service Control Policies

Service Control Policies, or SCPs, are applied through AWS Organizations.

They set permission limits for entire AWS accounts or organizational units.

For example, an SCP may:

  • Block resources outside approved AWS regions

  • Prevent CloudTrail from being disabled

  • Block public S3 bucket changes

  • Prevent member accounts from modifying security services

SCPs do not grant permissions.

A role may have AdministratorAccess, but an SCP can still block specific actions.

This is why a permission problem may not be visible when you only inspect the IAM role.

Long-Term and Temporary Credentials

AWS credentials generally fall into two categories.

Long-Term Credentials

IAM user access keys are long-term credentials.

They include:

  • Access key ID

  • Secret access key

These credentials remain valid until they are manually disabled or deleted.

That creates several risks:

  • They may be committed to Git

  • They may remain on an old laptop

  • They may be copied into a Docker image

  • They may not be rotated

  • Nobody may know which application still uses them

Access key IDs beginning with AKIA are usually long-term IAM user credentials.

Temporary Credentials

Temporary credentials are normally issued by AWS STS.

They contain:

  • Access key ID

  • Secret access key

  • Session token

  • Expiration time

Access key IDs beginning with ASIA are normally temporary credentials.

Temporary credentials are safer because they automatically expire.

Use temporary credentials wherever possible.

Common patterns include:

  • EC2 instance profiles

  • Lambda execution roles

  • ECS task roles

  • EKS Pod Identity or IRSA

  • GitHub Actions with OIDC

  • IAM Identity Center for employees

  • Cross-account role assumption

You should not place AWS access keys inside application code, Git repositories, Dockerfiles or Kubernetes manifests.

In most cases, the correct solution is to attach or assume a role.

How AWS Evaluates a Request

Imagine a Lambda function tries to read an object from S3.

AWS roughly follows this process.

Step 1: Authenticate the caller

AWS verifies the Lambda role's temporary credentials.

Step 2: Build the request context

AWS collects information such as:

  • Principal ARN

  • Requested action

  • Resource ARN

  • AWS account

  • Source IP

  • Region

  • Resource tags

  • MFA status

  • Session details

Step 3: Collect applicable policies

AWS checks relevant policies, including:

  • Identity policies

  • Resource policies

  • Permissions boundaries

  • SCPs

  • Session policies

  • KMS key policies when encryption is involved

Step 4: Look for an explicit deny

Any matching explicit deny blocks the request.

Step 5: Look for an allow

At least one applicable policy must allow the request.

The permission must also be inside the limits set by boundaries, SCPs and session policies.

Step 6: Deny if nothing allows it

When AWS finds no valid allow, the request is denied.

That is the basic IAM evaluation model.

Same-Account Access

Most workloads access resources inside the same AWS account.

The normal pattern is to attach permissions to the workload's role.

For example, suppose a Lambda function needs to:

  • Read files from one S3 bucket

  • Write records to one DynamoDB table

  • Send logs to CloudWatch

Its role should receive only those permissions.

Avoid broad policies such as:

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

This may quickly fix an AccessDenied error, but it gives the workload access to almost everything in the account.

A production policy should use:

  • Specific actions

  • Specific resources

  • Conditions where useful

Start small and add permissions based on actual requirements.

Cross-Account Access

Cross-account access involves two AWS accounts.

The most common method is role assumption.

Suppose Account A needs access to resources in Account B.

Account B creates a role with:

  • A trust policy allowing Account A to assume it

  • Permission policies defining what the role can do

The identity in Account A also needs permission to call sts:AssumeRole on that role.

Another method is a resource-based policy.

For example, an S3 bucket in Account B can directly allow a role from Account A to upload objects.

Cross-account problems usually happen because only one side was configured.

Always check:

  • Does the source identity have permission?

  • Does the destination trust the source?

  • Does the resource policy allow the source?

  • Does an SCP or explicit deny block the action?

  • Does KMS also allow access when encryption is enabled?

IAM for Human Access

Creating IAM users for every developer does not scale well.

Modern AWS environments normally use federation or IAM Identity Center.

Users authenticate through an identity provider such as:

  • Microsoft Entra ID

  • Okta

  • Google Workspace

  • IAM Identity Center

After authentication, they assume a role and receive temporary credentials.

This provides several benefits:

  • No permanent AWS access keys

  • Central employee access management

  • Easier onboarding and offboarding

  • Different roles for different accounts

  • Better auditing

  • Temporary sessions that expire

When someone leaves the company, disabling their identity provider account removes their AWS access.

You do not need to manually search every AWS account for IAM users and access keys.

IAM for CI/CD

CI/CD systems also need AWS access.

The unsafe method is storing an IAM user's access key as a CI/CD secret.

A better method is OIDC federation.

Platforms such as GitHub Actions can use OIDC to request temporary AWS credentials.

The flow is:

  1. The CI/CD job proves its identity to AWS.

  2. AWS checks the role's trust policy.

  3. AWS provides temporary credentials.

  4. The pipeline deploys using those credentials.

  5. The credentials expire.

There is no permanent AWS secret to rotate or leak.

You can also restrict the role so that only a specific repository, branch or environment can assume it.

Least Privilege in Real Projects

Least privilege means giving an identity only the access it requires.

Instead of:

"Action": "s3:*",
"Resource": "*"

Prefer:

"Action": [
  "s3:GetObject",
  "s3:PutObject"
],
"Resource": "arn:aws:s3:::my-app-bucket/uploads/*"

Do not try to create the perfect policy immediately.

A practical process is:

  1. Start with the smallest permissions you believe the workload needs.

  2. Run the workload.

  3. Read the AccessDenied error.

  4. Confirm which API action failed.

  5. Add the specific permission.

  6. Test again.

  7. Remove permissions that are no longer used.

AWS tools such as IAM Access Analyzer, CloudTrail and service last-accessed information can help identify required and unused permissions.

How to Debug AccessDenied Errors

When an AWS request fails with AccessDenied, do not immediately attach AdministratorAccess.

Check the problem in this order.

1. Confirm the real identity

Find out which user or role is making the request.

From the CLI, you can run:

aws sts get-caller-identity

Do not assume you are using the role you expected.

Your local terminal may be using an old profile, expired session or different AWS account.

2. Find the exact API action

The application may say it cannot access S3, but the failed action may actually be:

  • s3:GetObject

  • s3:ListBucket

  • s3:PutObject

  • kms:Decrypt

These actions require different permissions and sometimes different resource ARNs.

3. Check the resource ARN

The policy may allow the right action on the wrong resource.

For S3, the bucket ARN and object ARN are different.

Bucket:

arn:aws:s3:::my-bucket

Objects inside the bucket:

arn:aws:s3:::my-bucket/*

s3:ListBucket normally uses the bucket ARN.

s3:GetObject normally uses the object ARN.

4. Check identity policies

Confirm that the role or user has an applicable allow statement.

5. Check resource policies

Inspect the S3 bucket policy, KMS key policy, SQS policy or other resource policy.

6. Look for explicit denies

Check:

  • Identity policies

  • Resource policies

  • Permissions boundaries

  • SCPs

  • Session policies

  • Conditions based on IP, region, tags or MFA

The failed request may involve more than one AWS service.

An S3 object encrypted with KMS may require:

  • s3:GetObject

  • kms:Decrypt

Allowing S3 alone may not be enough.

8. Use CloudTrail

CloudTrail shows:

  • Which identity made the request

  • Which API action was called

  • Which resource was involved

  • When the request happened

  • Whether AWS allowed or denied it

CloudTrail is often the fastest way to stop guessing.

IAM Security Rules Worth Following

For most DevOps teams, these rules provide a strong starting point:

  • Enable MFA on the root account.

  • Avoid root access for daily work.

  • Use roles instead of IAM users where possible.

  • Prefer temporary credentials.

  • Never hardcode AWS access keys.

  • Use IAM Identity Center for employee access.

  • Use OIDC for CI/CD systems.

  • Avoid "Action": "*" and "Resource": "*".

  • Review old and unused permissions.

  • Use permissions boundaries when delegating IAM management.

  • Use SCPs for organization-wide guardrails.

  • Check KMS permissions when encrypted resources fail.

  • Use CloudTrail instead of guessing during access issues.

To Summarize

IAM becomes easier when you stop thinking of it as a large collection of JSON policies.

For every failed request, ask:

Who is making the request?

Which API action are they calling?

Which resource are they accessing?

Which policy allows it?

Is any policy explicitly denying it?

Is a boundary, SCP or session policy limiting it?

The final rules are simple:

  • No allow means deny.

  • Explicit deny always wins.

  • Roles are better than permanent access keys.

  • Permissions should be as narrow as practical.

  • Always debug the real identity, action and resource.

You do not need to memorize all of IAM. You need to understand how AWS reaches the final allow or deny decision.

Join 1,000+ engineers becoming better DevOps & SRE professionals.

Every week, I share:

  • How I'd approach problems differently (real projects, real mistakes)

  • Career moves that actually work (not LinkedIn motivational posts)

  • Technical deep-dives that change how you think about infrastructure

No fluff. No roadmaps. Just what works when you're building real systems.

👋 Find me on Twitter | Linkedin | Connect 1:1

Thank you for supporting this newsletter.
Y’all are the best.

Keep Reading