Posts

I woke up on March 25th to a Slack message from our security team: “ingress-nginx is EOL as of yesterday. Timeline for migration?” I had been ignoring this for months. The retirement was announced back in November 2025, but it felt distant. Now it was real. No more CVE patches. No more bug fixes. The clock was ticking. What Actually Happened On March 24, 2026, Kubernetes SIG Network and the Security Response Committee officially retired ingress-nginx. The project is done. Container images and Helm charts will stay available (they’re not deleting anything), but there will be no new releases. If a critical vulnerability drops tomorrow, you’re on your own. ...

I woke up this morning to a Slack message from our security lead: “axios got owned on npm.” I thought it was a joke. Axios has 60 million weekly downloads. It is one of those packages you just assume is safe because everyone uses it. It was not a joke. What Actually Happened Two malicious versions hit npm overnight: [email protected] and [email protected]. The attacker compromised a lead maintainer’s npm credentials, changed the account email to a ProtonMail address, and published manually via the npm CLI. No pull request. No CI run. No code review. Just a npm publish from a stolen account. ...

Every team wanted their own cluster. QA had one, staging had one, each developer wanted one for feature branches. We ended up with 12 EKS clusters, most of them sitting at 15% utilization, all of them costing real money. I kept hearing about vCluster from Loft Labs and finally gave it a shot three months ago. The pitch sounded too good: full Kubernetes clusters running inside a single host cluster, each with its own API server, its own resources, complete isolation. No extra nodes, no extra control planes to manage. ...

Last week I spent a full Saturday auditing every GitHub Actions workflow across our repos. Not because I wanted to, but because the Trivy supply chain attack made me realize how thin the ice was under my feet. If you missed it: someone managed to sneak a malicious commit into the actions/checkout action by exploiting GitHub’s fork commit reachability. They swapped a SHA pin in Trivy’s release workflow to point at an orphaned commit in a fork. The commit looked legit, the comment said # v6.0.2, the author was spoofed to look like a real maintainer. The actual payload downloaded Go files from a typosquatted domain and replaced Trivy’s source code during the build. ...

Last week I noticed that the Kubernetes project had quietly rewritten its image promoter, the tool that pushes official images to registry.k8s.io. The interesting part was not the rewrite itself. It was the fact that the new version now ships proper SLSA provenance attestations and cosign signatures across the mirrors. That was the moment I had to admit something slightly embarrassing: I had been signing my own images in CI for a while, but I was not actually enforcing verification anywhere in the cluster. The signatures existed, but nothing was checking them. So I finally sat down and fixed it. ...

I spent two years being the guy who provisions databases. Every Monday morning, same Slack message: “Hey, can I get a Postgres instance for the new service?” I’d open Terraform, copy a module block, change three variables, run the plan, wait for approval, apply. Twenty minutes of my life, gone. Multiply that by four teams and it adds up fast. Then I set up Crossplane with Compositions, and now developers do it themselves with a single YAML file. Here’s how I got there and what broke along the way. ...

Last week I got paged at 2 AM for a payment service that was dropping requests. My first instinct was the same as always: grab the cluster-admin kubeconfig from the shared wiki page and start poking around. I caught the bug in ten minutes, but the next morning our security team flagged my session in the audit logs. Fair enough. That cluster-admin kubeconfig had been “temporary” for about eight months. ...

Last month I got paged at 2 AM because the API server in a production cluster started timing out. Pods stopped scheduling, kubectl just hung, and the on-call Slack channel had already turned into chaos. About thirty minutes later, I traced it back to etcd. Again. etcd sits in the middle of every Kubernetes cluster, so when it starts having a bad day, the whole cluster feels it. The tricky part is that etcd failures rarely announce themselves clearly. You usually do not get a clean “etcd is broken” signal. You get fuzzy symptoms instead: slow API calls, delayed scheduling, weird timeouts. After dealing with enough of these incidents, I ended up with a playbook of checks that I run almost automatically now. Lately, a tool called etcd-diagnosis has made that process much easier. ...

Last month I finally pulled the trigger. After months of watching the OpenTofu project mature and HashiCorp’s licensing situation settle into something I wasn’t comfortable with for client work, I migrated 47 Terraform modules across three production environments to OpenTofu. It took about two weeks of actual work spread over a month, and most of it was smooth. Most. Why I Switched The BSL license change was the catalyst, but not the only reason. A few of my clients started asking uncomfortable questions about their Terraform Enterprise contracts. One of them got a letter from HashiCorp’s sales team that made the cost trajectory pretty clear. OpenTofu had reached a point where the risk of staying felt bigger than the risk of moving. ...

I have deleted an S3 bucket exactly once and regretted it immediately. Back in 2022, I tore down a staging environment, and within a few hours someone else had claimed the same bucket name. A CloudFormation stack in another account kept happily writing logs to a bucket I no longer controlled. Not my favorite Friday. AWS has finally shipped a real fix: account regional namespaces for S3 general purpose buckets. It took about seven years, which feels both absurd and very on-brand. ...