Going deeper on how software runs in production

Tue, 27 Oct 2026 00:00:00 GMT

For 8 years I’ve built backend systems in Java. The code is the part I know best.

But the hardest problems I’ve worked on often sat just past the code: how a service is deployed, how it’s secured, how we see what it’s doing, and how fast we can recover when something breaks.

So I’m going deeper on that side, about 10 hours a week:

AWS the way it’s run in production: multi-account, SSO, guard rails, budgets
Infrastructure as code, rebuilt from scratch on a timer
Kubernetes on my own hardware: an old laptop and two Raspberry Pi 5s
Observability, SLOs and restores I’ve actually timed

One rule: every step ends with evidence — what I built, how I checked it, what broke.

I write the interesting parts here, mostly the failures. The step-by-step versions live under Runbooks.

No human uses AWS root

Thu, 22 Oct 2026 00:00:00 GMT

Goal: an AWS setup where nobody signs in as root or as an IAM user — the foundation for every later AWS lab.

Built: an organization with a management account and two member accounts (dev, prod) in a workloads OU; IAM Identity Center with MFA on every sign-in; two service control policies; a $50 budget with three alerts in each account; three SSO profiles on my Mac.

Verified:

both member accounts report no root credentials present;
zero IAM users in all three accounts, no access keys on my machine;
an EC2 call to a region I don’t use is denied by a service control policy — I made that call on purpose to see it fail.

Broke and fixed:

The CLI returned exit code 0 and printed nothing. The output pager was swallowing it; --no-cli-pager proved it, AWS_PAGER="" fixed it. My checks now look for expected output, not just a zero exit code.
I found a forgotten third guard-rail policy from an earlier attempt, still attached. Exactly the drift Terraform will catch in Stage 2.

Decision: the region guard rail applies only to the workloads OU; the management account runs nothing but billing, Organizations and SSO.

Next: A1 — cross-account roles and least privilege.

Runbook: A0 — An AWS organization nobody logs into as root

Marek Reinhold | Blog

Going deeper on how software runs in production

No human uses AWS root