For aws-ecommerce-platform, my flagship build, I wired up a GitHub Actions deploy pipeline that authenticates into AWS via OIDC — no long-lived access keys sitting in a repo secret. The IAM role it assumes is deliberately scoped down to exactly "redeploy application code": push to the four ECR repos, update the four ECS services, sync and invalidate the frontend. Nothing else. Changing the VPC, RDS, WAF, or DNS still requires a human running terraform apply locally with broader credentials.
That decision is documented as its own ADR in the repo, and I'd defend it again. But deciding to scope a role down is the easy part. Getting it to actually work — proving it end-to-end against real AWS, not just writing Terraform that looks right — turned into one of the more genuinely instructive debugging sessions of this whole project. This is that story.
The trust policy assumed the wrong subject format
The first wall: every AssumeRoleWithWebIdentity call came back Not authorized, even though the trust policy's condition on GitHub's OIDC subject claim looked correct — matching the "classic" repo:owner/repo:* format every tutorial uses. aws cloudtrail lookup-events for that API call showed what GitHub was actually sending: an ID-suffixed subject, repo:owner@<owner_id>/repo@<repo_id>, not the plain slug. A quick gh api /repos/<owner>/<repo>/actions/oidc/customization/sub confirmed that's genuinely the account's current default, even with no custom subject claim configured. Fixed by adding a configurable override for the trust policy's expected prefix, verified against that API call rather than assumed.
"No resource-level permissions" isn't limited to the one action everyone warns you about
It's fairly well known that ecs:RegisterTaskDefinition can't be scoped to a specific ARN — AWS always evaluates it against *, and a "scoped" policy statement for it is silently ignored. I already knew that going in. What I didn't know: the same is true of ecs:DeregisterTaskDefinition, and — this was the expensive one — ecs:DescribeTaskDefinition too.
Terraform re-reads a resource immediately after creating it, and that read kept failing with a generic couldn't find resource — which looks exactly like an eventual-consistency race. I chased that theory first: serialized the four services' task-definition creates with -parallelism=1 on the assumption they were racing each other. No change. I added the tag-related permissions Terraform's read also needed (ecs:TagResource, ecs:ListTagsForResource — genuinely missing, genuinely required, but not the actual blocker). Still no change. The failure reproduced identically, every time, regardless.
What actually cracked it was giving up on inference and reading raw evidence instead: setting TF_LOG=DEBUG for one run and finding the real AWS API response underneath Terraform's own error wrapper. It said, in plain text, AccessDeniedException: ... not authorized to perform: ecs:DescribeTaskDefinition on resource: * — the exact same "always evaluated against wildcard" behavior as Register and Deregister, just hidden behind a much more misleading Terraform-level message. This is the part of the job I actually enjoy: the fix was one line once I had the real error, but getting the real error took refusing to accept a plausible-sounding guess.
What was actually wrong, end to end
| Symptom | Real cause | Fix |
|---|---|---|
| AssumeRoleWithWebIdentity denied | Trust policy expected the wrong OIDC subject format | Configurable subject prefix, verified via the GitHub API |
| State lock never released | Missing s3:DeleteObject on the state bucket | Granted, scoped to the state bucket only |
| Targeted apply failing on unrelated services | -target still refreshes the entire state by default | Added -refresh=false |
| DeregisterTaskDefinition denied despite a scoped ARN | No resource-level IAM support for this action | Granted on * instead |
| Task-definition read failed right after creation | Missing tag permissions, and DescribeTaskDefinition also has no resource-level support | Added the tag permissions; moved Describe onto * |
| Retried deploy failing on an old image tag | ECR's immutable tags rejecting a re-push after a transient CI outage | Push step checks ECR first, skips already-pushed tags |
Why not just widen the role and move on
At one point, mid-debugging, the easy option was sitting right there — grant the role broad read access across the account and stop chasing individual permissions one at a time. I didn't, and I'd make the same call again. A deploy role that can only ever do what it's actually meant to do is worth more on a resume — and worth more in an incident — than one that "works" because it can touch everything. Every fix here is one specific, minimal grant, not a blanket exception, and the repo's commit history shows that trail permission by permission.
The payoff: a real, green, end-to-end GitHub Actions run — image builds, ECR pushes, ECS task-definition updates, frontend sync, CloudFront invalidation — done entirely through that scoped role, no standing AWS keys anywhere in GitHub. That run is linked from the project's README, and it's the actual evidence the design works, not just that the Terraform for it is syntactically valid.