autojanet/skills/aws-iam-debugging/SKILL.md

name	description
aws-iam-debugging	Use when hitting AWS AccessDenied, authorization failures, IRSA/EKS pod permission errors, SSO session issues, cross-account AssumeRole failures, or MalformedPolicyDocument errors involving AWSReservedSSO_* principals in multi-account/Organizations environments.

AWS IAM Debugging

Overview

IAM failures have predictable root causes. Identify the caller, simulate or inspect the policy, check SCPs if multi-account. S3 requires BOTH IAM and bucket policy to allow — either can block independently.

Error Reference

Error	Likely cause
is not authorized to perform: X on resource: Y	Missing IAM policy statement
MalformedPolicyDocument: Invalid principal	Using AWSReservedSSO_* role as principal (not allowed)
Access Denied (S3)	Bucket policy + IAM both must allow; SCP may be blocking
AccessDenied (STS AssumeRole)	Trust policy missing caller ARN, or SCP blocks
InvalidClientTokenId	Wrong region, expired credentials, wrong profile
TokenRefreshRequired	SSO session expired — run aws sso login
Unable to locate credentials	No credentials configured — check ~/.aws/credentials or env vars

Diagnostic Flow

Step 1: Who is calling?

aws sts get-caller-identity
# Arn field tells you exactly what entity is making the call

Step 2: Simulate the permission

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<account>:role/<role> \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::<bucket>/*

aws iam list-attached-role-policies --role-name <role>
aws iam list-role-policies --role-name <role>        # inline policies
aws iam get-role-policy --role-name <role> --policy-name <policy>

Step 3: Check SCPs (multi-account)

aws organizations list-policies-for-target \
  --target-id <account-id> --filter SERVICE_CONTROL_POLICY
aws organizations describe-policy --policy-id <policy-id>

AWSReservedSSO_* Principal Gotcha

AWSReservedSSO_* roles cannot be used as IAM principals in trust policies.

# WRONG:
principals {
  type        = "AWS"
  identifiers = ["arn:aws:iam::123456789:role/AWSReservedSSO_Admin_abc"]
}

# CORRECT — allow via condition:
principals {
  type        = "AWS"
  identifiers = ["arn:aws:iam::123456789:root"]
}
condition {
  test     = "StringLike"
  variable = "aws:PrincipalArn"
  values   = ["arn:aws:iam::123456789:assumed-role/AWSReservedSSO_Admin_*/*"]
}

Alternatives: aws:PrincipalOrgID (if all callers are in the org), or aws:PrincipalTag.

IRSA (EKS IAM Roles for Service Accounts)

# Check ServiceAccount annotation
kubectl get sa <name> -n <namespace> -o yaml | grep eks.amazonaws.com

# Verify OIDC provider is registered
aws iam list-open-id-connect-providers

# Inspect role trust policy condition (must match exactly)
aws iam get-role --role-name <role> \
  | jq '.Role.AssumeRolePolicyDocument.Statement[].Condition'
# Required: "oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>:sub":
#            "system:serviceaccount:<namespace>:<sa-name>"

# Test from inside the pod
kubectl exec -n <ns> <pod> -- aws sts get-caller-identity

Common mistakes: namespace/SA name typo in trust policy; OIDC provider not registered.

S3 Access Denied

aws s3api get-bucket-policy --bucket <bucket>
aws s3api get-bucket-acl --bucket <bucket>
aws s3api get-public-access-block --bucket <bucket>
aws s3 ls s3://<bucket> --debug 2>&1 | grep "Final credentials"

Cross-Account AssumeRole

# Try manually
aws sts assume-role \
  --role-arn arn:aws:iam::<target-account>:role/<role> \
  --role-session-name test-session

# If AccessDenied, check:
# 1. Trust policy of target role allows caller's ARN
# 2. Caller has sts:AssumeRole in their own account
# 3. No SCP blocks sts:AssumeRole in either account

aws iam get-role --role-name <role> | jq '.Role.AssumeRolePolicyDocument'

SSO / Identity Center Sessions

aws sso login --profile <profile>
aws configure list-profiles
aws sts get-caller-identity --profile <profile>

# Clear stale tokens
rm ~/.aws/sso/cache/*.json && aws sso login --profile <profile>

CloudTrail — Find What Was Denied

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "2024-01-01T00:00:00Z" --max-results 10

# Filter by error code
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=<username> \
  | jq '.Events[] | select(.CloudTrailEvent | fromjson | .errorCode != null)'