cc74ad0bd0
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
- .woodpecker.yaml: image paths -> library/autojanet-{agent,dispatcher}
- .woodpecker.yaml: secret names RS_HARBOR_USER / RS_HARBOR_PASS (global)
- container/Dockerfile: restore COPY skills/, skills/ populated from opencode config
- skills/: 84 opencode skills bundled into image
- k8s/manifests: update image refs to library/
144 lines
4.5 KiB
Markdown
144 lines
4.5 KiB
Markdown
---
|
|
name: aws-iam-debugging
|
|
description: Use when hitting AWS AccessDenied, authorization failures, IRSA/EKS pod permission errors, SSO session issues, cross-account AssumeRole failures, or MalformedPolicyDocument errors involving AWSReservedSSO_* principals in multi-account/Organizations environments.
|
|
---
|
|
|
|
# AWS IAM Debugging
|
|
|
|
## Overview
|
|
|
|
IAM failures have predictable root causes. Identify the caller, simulate or inspect the policy, check SCPs if multi-account. S3 requires BOTH IAM and bucket policy to allow — either can block independently.
|
|
|
|
## Error Reference
|
|
|
|
| Error | Likely cause |
|
|
|-------|-------------|
|
|
| `is not authorized to perform: X on resource: Y` | Missing IAM policy statement |
|
|
| `MalformedPolicyDocument: Invalid principal` | Using `AWSReservedSSO_*` role as principal (not allowed) |
|
|
| `Access Denied` (S3) | Bucket policy + IAM both must allow; SCP may be blocking |
|
|
| `AccessDenied` (STS AssumeRole) | Trust policy missing caller ARN, or SCP blocks |
|
|
| `InvalidClientTokenId` | Wrong region, expired credentials, wrong profile |
|
|
| `TokenRefreshRequired` | SSO session expired — run `aws sso login` |
|
|
| `Unable to locate credentials` | No credentials configured — check `~/.aws/credentials` or env vars |
|
|
|
|
## Diagnostic Flow
|
|
|
|
**Step 1: Who is calling?**
|
|
```bash
|
|
aws sts get-caller-identity
|
|
# Arn field tells you exactly what entity is making the call
|
|
```
|
|
|
|
**Step 2: Simulate the permission**
|
|
```bash
|
|
aws iam simulate-principal-policy \
|
|
--policy-source-arn arn:aws:iam::<account>:role/<role> \
|
|
--action-names s3:GetObject \
|
|
--resource-arns arn:aws:s3:::<bucket>/*
|
|
|
|
aws iam list-attached-role-policies --role-name <role>
|
|
aws iam list-role-policies --role-name <role> # inline policies
|
|
aws iam get-role-policy --role-name <role> --policy-name <policy>
|
|
```
|
|
|
|
**Step 3: Check SCPs (multi-account)**
|
|
```bash
|
|
aws organizations list-policies-for-target \
|
|
--target-id <account-id> --filter SERVICE_CONTROL_POLICY
|
|
aws organizations describe-policy --policy-id <policy-id>
|
|
```
|
|
|
|
## AWSReservedSSO_* Principal Gotcha
|
|
|
|
`AWSReservedSSO_*` roles **cannot** be used as IAM principals in trust policies.
|
|
|
|
```hcl
|
|
# WRONG:
|
|
principals {
|
|
type = "AWS"
|
|
identifiers = ["arn:aws:iam::123456789:role/AWSReservedSSO_Admin_abc"]
|
|
}
|
|
|
|
# CORRECT — allow via condition:
|
|
principals {
|
|
type = "AWS"
|
|
identifiers = ["arn:aws:iam::123456789:root"]
|
|
}
|
|
condition {
|
|
test = "StringLike"
|
|
variable = "aws:PrincipalArn"
|
|
values = ["arn:aws:iam::123456789:assumed-role/AWSReservedSSO_Admin_*/*"]
|
|
}
|
|
```
|
|
|
|
Alternatives: `aws:PrincipalOrgID` (if all callers are in the org), or `aws:PrincipalTag`.
|
|
|
|
## IRSA (EKS IAM Roles for Service Accounts)
|
|
|
|
```bash
|
|
# Check ServiceAccount annotation
|
|
kubectl get sa <name> -n <namespace> -o yaml | grep eks.amazonaws.com
|
|
|
|
# Verify OIDC provider is registered
|
|
aws iam list-open-id-connect-providers
|
|
|
|
# Inspect role trust policy condition (must match exactly)
|
|
aws iam get-role --role-name <role> \
|
|
| jq '.Role.AssumeRolePolicyDocument.Statement[].Condition'
|
|
# Required: "oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>:sub":
|
|
# "system:serviceaccount:<namespace>:<sa-name>"
|
|
|
|
# Test from inside the pod
|
|
kubectl exec -n <ns> <pod> -- aws sts get-caller-identity
|
|
```
|
|
|
|
Common mistakes: namespace/SA name typo in trust policy; OIDC provider not registered.
|
|
|
|
## S3 Access Denied
|
|
|
|
```bash
|
|
aws s3api get-bucket-policy --bucket <bucket>
|
|
aws s3api get-bucket-acl --bucket <bucket>
|
|
aws s3api get-public-access-block --bucket <bucket>
|
|
aws s3 ls s3://<bucket> --debug 2>&1 | grep "Final credentials"
|
|
```
|
|
|
|
## Cross-Account AssumeRole
|
|
|
|
```bash
|
|
# Try manually
|
|
aws sts assume-role \
|
|
--role-arn arn:aws:iam::<target-account>:role/<role> \
|
|
--role-session-name test-session
|
|
|
|
# If AccessDenied, check:
|
|
# 1. Trust policy of target role allows caller's ARN
|
|
# 2. Caller has sts:AssumeRole in their own account
|
|
# 3. No SCP blocks sts:AssumeRole in either account
|
|
|
|
aws iam get-role --role-name <role> | jq '.Role.AssumeRolePolicyDocument'
|
|
```
|
|
|
|
## SSO / Identity Center Sessions
|
|
|
|
```bash
|
|
aws sso login --profile <profile>
|
|
aws configure list-profiles
|
|
aws sts get-caller-identity --profile <profile>
|
|
|
|
# Clear stale tokens
|
|
rm ~/.aws/sso/cache/*.json && aws sso login --profile <profile>
|
|
```
|
|
|
|
## CloudTrail — Find What Was Denied
|
|
|
|
```bash
|
|
aws cloudtrail lookup-events \
|
|
--lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
|
|
--start-time "2024-01-01T00:00:00Z" --max-results 10
|
|
|
|
# Filter by error code
|
|
aws cloudtrail lookup-events \
|
|
--lookup-attributes AttributeKey=Username,AttributeValue=<username> \
|
|
| jq '.Events[] | select(.CloudTrailEvent | fromjson | .errorCode != null)'
|
|
```
|