Last week I got paged at 2 AM for a payment service that was dropping requests. My first instinct was the same as always: grab the cluster-admin kubeconfig from the shared wiki page and start poking around. I caught the bug in ten minutes, but the next morning our security team flagged my session in the audit logs. Fair enough. That cluster-admin kubeconfig had been “temporary” for about eight months. ...
Last month I got paged at 2 AM because the API server in a production cluster started timing out. Pods stopped scheduling, kubectl just hung, and the on-call Slack channel had already turned into chaos. About thirty minutes later, I traced it back to etcd. Again. etcd sits in the middle of every Kubernetes cluster, so when it starts having a bad day, the whole cluster feels it. The tricky part is that etcd failures rarely announce themselves clearly. You usually do not get a clean “etcd is broken” signal. You get fuzzy symptoms instead: slow API calls, delayed scheduling, weird timeouts. After dealing with enough of these incidents, I ended up with a playbook of checks that I run almost automatically now. Lately, a tool called etcd-diagnosis has made that process much easier. ...
Last month I finally pulled the trigger. After months of watching the OpenTofu project mature and HashiCorp’s licensing situation settle into something I wasn’t comfortable with for client work, I migrated 47 Terraform modules across three production environments to OpenTofu. It took about two weeks of actual work spread over a month, and most of it was smooth. Most. Why I Switched The BSL license change was the catalyst, but not the only reason. A few of my clients started asking uncomfortable questions about their Terraform Enterprise contracts. One of them got a letter from HashiCorp’s sales team that made the cost trajectory pretty clear. OpenTofu had reached a point where the risk of staying felt bigger than the risk of moving. ...
I have deleted an S3 bucket exactly once and regretted it immediately. Back in 2022, I tore down a staging environment, and within a few hours someone else had claimed the same bucket name. A CloudFormation stack in another account kept happily writing logs to a bucket I no longer controlled. Not my favorite Friday. AWS has finally shipped a real fix: account regional namespaces for S3 general purpose buckets. It took about seven years, which feels both absurd and very on-brand. ...
I spent most of last week chasing image pull failures in a multi-tenant cluster. It turned out the problem was our private registry mirror. We were using it as a pull-through cache, but the credentials lived on the nodes. One team rotated their credentials and, a few minutes later, pods in three other namespaces started failing too. That was the moment it became obvious we had a shared-credentials problem. That sent me down the rabbit hole of CRI-O’s credential provider for registry mirrors. After setting it up, I do not really want to go back. ...
I got tired of managing separate logins for Grafana, ArgoCD, Harbor, and every other internal tool we run. Every new team member meant creating five accounts. Every offboarding meant hoping I remembered to revoke all of them. So I finally sat down and deployed Keycloak on our Kubernetes cluster. This is what actually happened, not the sanitized version. Why Keycloak I looked at Dex, Authelia, and Keycloak. Dex is lightweight but limited if you need more than OIDC proxying. Authelia is great for simple setups but felt thin for our use case. Keycloak is heavier, but it handles OIDC, SAML, user federation, and has a proper admin UI. For a team running 8+ internal services, the weight is justified. ...
I kept postponing this migration for way too long. Every time Gateway API came up, I had the same answer: “yeah, I know, I should do it.” Then last week I finally stopped talking about it and migrated three production clusters from Ingress to Gateway API. After doing it end to end, I wish I had moved sooner. Why I Finally Did It The trigger was a multi-tenant cluster where two teams shared the same domain but needed different TLS behavior. ...
I’ve been running Falco for runtime security on most of my clusters for the past two years. It did the job, but the kernel module approach always felt brittle. Every kernel upgrade felt like rolling dice. When Cilium Tetragon reached 1.3 stable and went full eBPF with no kernel module, I finally gave it a real try on a production cluster. This is what happened. Why I Switched from Falco Falco has been solid, no question. But I kept running into the same issues: ...
Last week I inherited a cluster with around 40 microservices. Observability was close to nonexistent: basic Prometheus metrics, plus a few random log lines. The team wanted distributed tracing “by next sprint.” There was no realistic way to touch app code across a dozen repos in two weeks. So I chose OpenTelemetry Operator auto-instrumentation. This is what happened in practice. The Setup We run Kubernetes 1.31 on EKS. The goal was simple: get traces and metrics from every service into Grafana Tempo and Mimir without changing application code. ...
I’ve been running mixed clusters with ML training jobs and regular services for about two years. Scheduling has been the biggest headache. A distributed training run would get only some pods placed, GPUs would sit there doing nothing, and everyone would lose time. Kubernetes 1.35 came out last week, so I spent the weekend testing it on our staging cluster. A few of these changes are genuinely useful. Gang Scheduling Finally Exists The biggest addition is workload-aware scheduling with gang scheduling support. It’s still alpha, so I would not put it in production yet, but the model is exactly what we needed: a group of pods either gets scheduled together, or not at all. ...