Troubleshooting: What Changed?
Audience: Operator responding to a production issue who needs to quickly determine what changed, when, and why.
The 5-minute investigation
When something breaks, run through these steps in order:
1. Check current drift
strata deploy drift -f deploy/deploy-prd.yaml
This runs a non-destructive terraform plan for each stage and classifies any changes
by severity (critical → info). Resources that changed since the last deploy are listed
with the attributes that differ.
No drift? The issue isn’t a configuration change — look at application logs, external dependencies, or upstream provider outages.
Drift found? Continue to step 2 to find out who/what caused it.
See Detecting Infrastructure Drift for full details on severity levels, thresholds, and CI integration.
2. Check execution history
strata log list
This shows recent build and deploy operations with timestamps, exit codes, and the deployment file used. Look for recent operations that correlate with the start of the issue.
3. Check git history for the deployment file
# What changed in the deployment config?
git log --oneline -10 deploy/deploy-prd.yaml
# What changed in the environment?
git log --oneline -10 envs/env-prd.yaml
# What changed in the workspace or resources?
git log --oneline -10 stack/
4. See the exact diff for a suspicious commit
git show <commit-hash> -- deploy/ envs/ stack/
5. Rebuild and compare
If you suspect the current build output doesn’t match what’s deployed:
# See what build run WOULD produce now
strata build plan --file deploy/deploy-prd.yaml
# Compare with what's currently in the build output directory
ls .strata/build/haven_deploy_prd/infrastructure/
Common scenarios
“A variable value changed unexpectedly”
Variables layer: workspace → environment → deployment. Check each file:
# Which file sets this variable?
grep -r "MY_VARIABLE" stack/ envs/ deploy/
The last file in the merge order wins. If envs/env-prd.yaml was recently edited, that’s likely the source.
“A new resource appeared / disappeared”
Resources are defined in workspace files. Check:
git log --oneline -5 stack/ws-platform.yaml
git diff HEAD~1 stack/ws-platform.yaml
“Secrets stopped resolving”
Secrets are fetched at build time from the configured store (Bitwarden, Key Vault, etc.).
# Check which secrets are defined
grep -A2 "store:" envs/env-prd.yaml
# Rebuild with verbose to see secret resolution
strata build run --file deploy/deploy-prd.yaml --verbose
Common causes: expired credentials, rotated secret IDs, vault access policy changes.
“Terraform apply failed but config looks right”
The build output is plain Terraform. Inspect it directly:
# Look at what strata generated
cat .strata/build/haven_deploy_prd/infrastructure/terraform.tfvars.json
# Run terraform plan manually for more detail
cd .strata/build/haven_deploy_prd/infrastructure/
terraform init
terraform plan
Timeline reconstruction
To build a full timeline of what happened:
# 1. When was the last successful deploy?
strata log list | head -5
# 2. What config commits happened since then?
git log --oneline --since="2025-01-15" -- deploy/ envs/ stack/ config/
# 3. What's the current state vs. desired state?
strata build plan --file deploy/deploy-prd.yaml
Prevention
Practice |
Why |
|---|---|
Run |
Catch drift before users notice |
Commit before deploying |
Git history = audit trail |
Use |
Review infrastructure changes like code |
Tag deployments in git |
|
Use separate secrets per environment |
Rotating one environment’s secrets doesn’t break another |