Troubleshooting

The problems people hit early, and how to read them. If something here doesn’t match what you see, the full Run — logs, plan, and status — is always in the Console.

A Run is stuck, or came back “force-failed”

Every executing Run sends a liveness heartbeat while it works, independent of log output — so a quiet tofu apply never looks dead. If a Run stops beating (its pod crashed, the node was lost, or it never reported), a background stale sweep eventually force-fails it rather than leaving it hung forever.

• What “runner timed out or never reported” means: the control plane stopped hearing from the runner and closed the Run out. The work may have partly run.
• A late write is recorded, not hidden. If a force-failed runner was mid-apply and still manages to write state, that version lands and is flagged loudly — the state records what actually happened to your infrastructure. Read the newest state version before you retry.
• What to do: retry the Run (it starts a fresh Run over the same source), or trigger a new manual Change Run. A plan that failed this way is always safe to retry; after an apply, read the latest state first so a retry re-plans against the truth.

A Run that has genuinely exceeded its tier’s phase timeout is interrupted, not hung — size it up (see Reference) if your applies routinely approach the limit.

A push or PR didn’t trigger a Run

Work down this checklist — most misses are one of these:

1. Is the App installed on that repository? The Stack’s overview shows the connection. If it isn’t connected, Connect GitHub and install the App on the repo (see the Quickstart).
2. Does the Stack track that branch? A push only triggers a Change Run on a Stack whose tracked branch is the pushed branch. A pull request only plans on a Stack whose tracked branch is the PR’s base.
3. Do the changed files touch the Stack’s watch paths? Path filtering is prefix-based: a Stack triggers only when a changed file equals, or sits under, one of its watch paths — which default to the Stack’s own path. A push that changes only files outside those prefixes triggers nothing. Add extra watch paths for shared modules the Stack depends on.
4. Is it an event Stacks handles? Tag pushes and branch/tag deletions are ignored by design — see Reference.

A private-repo clone fails

Clones use a short-lived installation token minted per Run, so a private repo needs the App installed on it and bound to the Stack.

• The App wasn’t installed when the Stack was created. Creating a Stack before installing the App leaves the binding empty, and an anonymous clone of a private repo fails. Fix it by installing (or reconnecting) the App from the Stack’s overview; a manual Run or retry re-resolves the binding once automatically before it gives up, so a freshly-installed App self-heals on the next Run when the App now covers the repository.
• The token only ever has read access to contents, and it dies with the Run — there is nothing long-lived to rotate or leak.

An OIDC trust policy rejects the token

Run identity is the most error-prone thing to wire, and almost every failure is one of five. When AWS refuses the assume-role, you’ll see InvalidIdentityToken or AccessDenied from STS in the Run log.

Read the real token first. Mint one and decode its payload to see the exact claims your policy must match — don’t hand-write the subject from memory. The request URL returns {"value": "<jwt>", "expires_at": …}, so pull out value and decode its middle (.-separated) segment:

# inside a Run (or reproduce the request with the Run env vars)
curl -sS -H "Authorization: Bearer $BOSUN_ID_TOKEN_REQUEST_TOKEN" \
  "$BOSUN_ID_TOKEN_REQUEST_URL?audience=sts.amazonaws.com" \
  | jq -r .value | cut -d. -f2 | base64 -d 2>/dev/null

That prints the sub, aud, and the rest. Then check, in order:

1. The sub condition. The subject is org:<org>:stack:<stack>:run_phase:<plan|apply>, and <stack> is the Stack’s id, not its display name — pasting the human name is the single most common mistake. Match the decoded sub exactly.
2. The aud condition. The AWS path requests the audience sts.amazonaws.com; your IAM OIDC provider’s audience and the trust policy’s aud condition must both be that string. A different audience (OpenBao, GCP, Azure) is whatever that consumer requests — see Run identity.
3. Plan-role vs apply-role. A plan reads and refreshes your existing resources, so the run_phase:plan role needs read on the state’s resources — a role scoped to zero permissions fails at refresh, not only at write. The run_phase:apply role needs write. Granting the plan role too little is a frequent surprise.
4. The provider trust anchor. The IAM OIDC identity provider must be for https://api.stacks.lngbrdg.com (the issuer), with the JWKS reachable — AWS fetches it to verify the signature.
5. Clock skew. A Run identity token lives 10 minutes. A cloud account or Vault whose clock is far off can reject a valid token as expired or not-yet-valid; fix the skew.

A state lock is held (423 Locked)

The backend lock serializes every writer — a hosted Run and a laptop tofu apply can never write over each other.

• Reading the holder. A contended LOCK returns 423 with the current holder’s lock info as the body: who holds it, since when, and the operation. tofu prints this when it can’t acquire the lock.
• Waiting instead of failing. Pass -lock-timeout=<duration> (for example -lock-timeout=5m) to make a local tofu wait for the lock instead of failing immediately — useful when a hosted Run holds it briefly.
• Force-unlock. If a process crashed and left the lock, release it with the exact lock ID from the holder info — from the Stack’s Console page, or tofu force-unlock <lock-id>.

Warning: Force-unlock only when you are certain nothing is still writing. Force-unlocking a live writer lets two applies write the same state and corrupt it, and that cannot be undone by re-locking.

A provider or module won’t download

Hosted Runs execute in an egress-restricted sandbox: the public internet is reachable, but anything private is not. So:

• Public registries work — registry.opentofu.org, registry.terraform.io, GitHub-hosted modules, public provider mirrors.
• Private or internal registries do not. A provider or module served from an address inside your own network (an internal Artifactory, a VPC-only mirror, a registry behind your VPN) is unreachable from a hosted Run, and the download fails. This is deliberate — see Trust & isolation.
• The path forward is a self-hosted runner pool (on the roadmap): the same agent on your own machines, with your own network reach. Until then, use public sources, or a public mirror of the artifacts a hosted Run needs.