Security

I woke up this morning to a Slack message from our security lead: “axios got owned on npm.” I thought it was a joke. Axios has 60 million weekly downloads. It is one of those packages you just assume is safe because everyone uses it. It was not a joke. What Actually Happened Two malicious versions hit npm overnight: [email protected] and [email protected]. The attacker compromised a lead maintainer’s npm credentials, changed the account email to a ProtonMail address, and published manually via the npm CLI. No pull request. No CI run. No code review. Just a npm publish from a stolen account. ...

Last week I spent a full Saturday auditing every GitHub Actions workflow across our repos. Not because I wanted to, but because the Trivy supply chain attack made me realize how thin the ice was under my feet. If you missed it: someone managed to sneak a malicious commit into the actions/checkout action by exploiting GitHub’s fork commit reachability. They swapped a SHA pin in Trivy’s release workflow to point at an orphaned commit in a fork. The commit looked legit, the comment said # v6.0.2, the author was spoofed to look like a real maintainer. The actual payload downloaded Go files from a typosquatted domain and replaced Trivy’s source code during the build. ...

Last week I noticed that the Kubernetes project had quietly rewritten its image promoter, the tool that pushes official images to registry.k8s.io. The interesting part was not the rewrite itself. It was the fact that the new version now ships proper SLSA provenance attestations and cosign signatures across the mirrors. That was the moment I had to admit something slightly embarrassing: I had been signing my own images in CI for a while, but I was not actually enforcing verification anywhere in the cluster. The signatures existed, but nothing was checking them. So I finally sat down and fixed it. ...

Last week I got paged at 2 AM for a payment service that was dropping requests. My first instinct was the same as always: grab the cluster-admin kubeconfig from the shared wiki page and start poking around. I caught the bug in ten minutes, but the next morning our security team flagged my session in the audit logs. Fair enough. That cluster-admin kubeconfig had been “temporary” for about eight months. ...

I have deleted an S3 bucket exactly once and regretted it immediately. Back in 2022, I tore down a staging environment, and within a few hours someone else had claimed the same bucket name. A CloudFormation stack in another account kept happily writing logs to a bucket I no longer controlled. Not my favorite Friday. AWS has finally shipped a real fix: account regional namespaces for S3 general purpose buckets. It took about seven years, which feels both absurd and very on-brand. ...

I spent most of last week chasing image pull failures in a multi-tenant cluster. It turned out the problem was our private registry mirror. We were using it as a pull-through cache, but the credentials lived on the nodes. One team rotated their credentials and, a few minutes later, pods in three other namespaces started failing too. That was the moment it became obvious we had a shared-credentials problem. That sent me down the rabbit hole of CRI-O’s credential provider for registry mirrors. After setting it up, I do not really want to go back. ...

I got tired of managing separate logins for Grafana, ArgoCD, Harbor, and every other internal tool we run. Every new team member meant creating five accounts. Every offboarding meant hoping I remembered to revoke all of them. So I finally sat down and deployed Keycloak on our Kubernetes cluster. This is what actually happened, not the sanitized version. Why Keycloak I looked at Dex, Authelia, and Keycloak. Dex is lightweight but limited if you need more than OIDC proxying. Authelia is great for simple setups but felt thin for our use case. Keycloak is heavier, but it handles OIDC, SAML, user federation, and has a proper admin UI. For a team running 8+ internal services, the weight is justified. ...

I’ve been running Falco for runtime security on most of my clusters for the past two years. It did the job, but the kernel module approach always felt brittle. Every kernel upgrade felt like rolling dice. When Cilium Tetragon reached 1.3 stable and went full eBPF with no kernel module, I finally gave it a real try on a production cluster. This is what happened. Why I Switched from Falco Falco has been solid, no question. But I kept running into the same issues: ...

Last Tuesday, one of my Go services got 14 Dependabot PRs in a single day. All of them came from one CVE, and none of them affected the way our code actually runs in production. We still had to read the alerts, review the PRs, wait for CI, and merge changes. That was the moment I decided to stop using Dependabot for this workflow. What finally broke it for me The issue was CVE-2026-26958 in filippo.io/edwards25519. ...

Kyverno 1.17 landed yesterday, and the big news is that CEL policy types are now GA. If you’ve been running Kyverno with JMESPath-based ClusterPolicy resources, the clock is ticking. They’re officially deprecated and scheduled for removal in v1.20 (October 2026). I spent today migrating a production cluster with about 60 policies. Here is what actually happened. Why This Matters Kyverno has been using JMESPath expressions for years. They work, but they’re Kyverno-specific. CEL (Common Expression Language) is what Kubernetes itself uses for ValidatingAdmissionPolicy since 1.30. By switching to CEL, Kyverno aligns with upstream and gets significantly better evaluation performance. ...