---
name: aws-iam-debugging
description: Use when hitting AWS AccessDenied, authorization failures, IRSA/EKS pod permission errors, SSO session issues, cross-account AssumeRole failures, or MalformedPolicyDocument errors involving AWSReservedSSO_* principals in multi-account/Organizations environments.
---

# AWS IAM Debugging

## Overview

IAM failures have predictable root causes. Identify the caller, simulate or inspect the policy, check SCPs if multi-account. S3 requires BOTH IAM and bucket policy to allow — either can block independently.

## Error Reference

| Error | Likely cause |
|-------|-------------|
| `is not authorized to perform: X on resource: Y` | Missing IAM policy statement |
| `MalformedPolicyDocument: Invalid principal` | Using `AWSReservedSSO_*` role as principal (not allowed) |
| `Access Denied` (S3) | Bucket policy + IAM both must allow; SCP may be blocking |
| `AccessDenied` (STS AssumeRole) | Trust policy missing caller ARN, or SCP blocks |
| `InvalidClientTokenId` | Wrong region, expired credentials, wrong profile |
| `TokenRefreshRequired` | SSO session expired — run `aws sso login` |
| `Unable to locate credentials` | No credentials configured — check `~/.aws/credentials` or env vars |

## Diagnostic Flow

**Step 1: Who is calling?**
```bash
aws sts get-caller-identity
# Arn field tells you exactly what entity is making the call
```

**Step 2: Simulate the permission**
```bash
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<account>:role/<role> \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::<bucket>/*

aws iam list-attached-role-policies --role-name <role>
aws iam list-role-policies --role-name <role>        # inline policies
aws iam get-role-policy --role-name <role> --policy-name <policy>
```

**Step 3: Check SCPs (multi-account)**
```bash
aws organizations list-policies-for-target \
  --target-id <account-id> --filter SERVICE_CONTROL_POLICY
aws organizations describe-policy --policy-id <policy-id>
```

## AWSReservedSSO_* Principal Gotcha

`AWSReservedSSO_*` roles **cannot** be used as IAM principals in trust policies.

```hcl
# WRONG:
principals {
  type        = "AWS"
  identifiers = ["arn:aws:iam::123456789:role/AWSReservedSSO_Admin_abc"]
}

# CORRECT — allow via condition:
principals {
  type        = "AWS"
  identifiers = ["arn:aws:iam::123456789:root"]
}
condition {
  test     = "StringLike"
  variable = "aws:PrincipalArn"
  values   = ["arn:aws:iam::123456789:assumed-role/AWSReservedSSO_Admin_*/*"]
}
```

Alternatives: `aws:PrincipalOrgID` (if all callers are in the org), or `aws:PrincipalTag`.

## IRSA (EKS IAM Roles for Service Accounts)

```bash
# Check ServiceAccount annotation
kubectl get sa <name> -n <namespace> -o yaml | grep eks.amazonaws.com

# Verify OIDC provider is registered
aws iam list-open-id-connect-providers

# Inspect role trust policy condition (must match exactly)
aws iam get-role --role-name <role> \
  | jq '.Role.AssumeRolePolicyDocument.Statement[].Condition'
# Required: "oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>:sub":
#            "system:serviceaccount:<namespace>:<sa-name>"

# Test from inside the pod
kubectl exec -n <ns> <pod> -- aws sts get-caller-identity
```

Common mistakes: namespace/SA name typo in trust policy; OIDC provider not registered.

## S3 Access Denied

```bash
aws s3api get-bucket-policy --bucket <bucket>
aws s3api get-bucket-acl --bucket <bucket>
aws s3api get-public-access-block --bucket <bucket>
aws s3 ls s3://<bucket> --debug 2>&1 | grep "Final credentials"
```

## Cross-Account AssumeRole

```bash
# Try manually
aws sts assume-role \
  --role-arn arn:aws:iam::<target-account>:role/<role> \
  --role-session-name test-session

# If AccessDenied, check:
# 1. Trust policy of target role allows caller's ARN
# 2. Caller has sts:AssumeRole in their own account
# 3. No SCP blocks sts:AssumeRole in either account

aws iam get-role --role-name <role> | jq '.Role.AssumeRolePolicyDocument'
```

## SSO / Identity Center Sessions

```bash
aws sso login --profile <profile>
aws configure list-profiles
aws sts get-caller-identity --profile <profile>

# Clear stale tokens
rm ~/.aws/sso/cache/*.json && aws sso login --profile <profile>
```

## CloudTrail — Find What Was Denied

```bash
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "2024-01-01T00:00:00Z" --max-results 10

# Filter by error code
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<username> \
  | jq '.Events[] | select(.CloudTrailEvent | fromjson | .errorCode != null)'
```