I run a few Kubernetes clusters on bare metal with Cluster API and the BYOH (Bring Your Own Host) provider. Until now, every upgrade followed the same pattern: drain nodes, delete machines, rebuild everything, then wait. Reliable, yes. Fast, not even close, especially once you are past 40 nodes with little spare capacity.

Cluster API v1.12 shipped a few weeks ago, and the headline for me was in-place updates. Instead of always doing the immutable delete-and-recreate path, CAPI can now apply some changes directly on existing machines. I spent last week testing this on our staging cluster, and the result was better than I expected.

Where immutable rollouts hurt

I still like immutable infrastructure. The issue is the cost on bare metal. A “new machine” means PXE booting a physical server, running a full OS install, joining the cluster, and pulling images all over again. On a good day, that is 15-20 minutes per node.

That is a lot of churn when the change is small, like rotating kubelet credentials or updating a cloud-init user. We worked around that with side-channel Ansible changes, but that drifts away from declarative cluster management.

How in-place updates work in v1.12

The feature is built around update extensions. You implement Runtime SDK hooks that tell CAPI, “I can handle this change on the current machine.”

The flow is simple:

  1. You change something in a Machine spec, usually via KubeadmControlPlane or MachineDeployment.
  2. CAPI checks whether a registered update extension can handle that diff.
  3. If yes, it calls the extension to perform the update in place.
  4. If no extension claims it, CAPI falls back to the classic rollout.

So this is not mutable versus immutable. It is CAPI choosing the safer tool for each change.

Setup

First, enable the feature gates in your clusterctl config or CAPI controller deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: capi-feature-gates
  namespace: capi-system
data:
  feature-gates: "InPlaceUpdates=true,RuntimeSDK=true"

Then deploy an update extension. I wrote a small one for credential rotation and metadata changes. At minimum, implement BeforeUpdateMachine and UpdateMachine:

func (h *Handler) BeforeUpdateMachine(ctx context.Context, req *runtimehooksv1.BeforeUpdateMachineRequest) *runtimehooksv1.BeforeUpdateMachineResponse {
    resp := &runtimehooksv1.BeforeUpdateMachineResponse{}

    oldSpec := req.OldMachine.Spec
    newSpec := req.NewMachine.Spec

    // Only handle changes we know are safe to apply in place
    if onlyLabelsOrAnnotationsChanged(oldSpec, newSpec) {
        resp.RetryAfterSeconds = 0
        resp.Status = runtimehooksv1.ResponseStatusSuccess
        return resp
    }

    // For everything else, let CAPI do a rollout
    resp.Status = runtimehooksv1.ResponseStatusFailure
    return resp
}

Register it with an ExtensionConfig:

apiVersion: runtime.cluster.x-k8s.io/v1alpha1
kind: ExtensionConfig
metadata:
  name: inplace-updater
spec:
  clientConfig:
    service:
      name: inplace-updater
      namespace: capi-system
      port: 8443
  namespaceSelector:
    matchLabels:
      cluster.x-k8s.io/managed: "true"

What I tested

I ran three scenarios on staging (12 nodes, BYOH provider, Ubuntu 22.04):

1. Worker label change

I changed one custom label in the MachineDeployment template. Instead of rolling all 8 workers, CAPI patched labels in place. Total time was about 3 seconds. Before this, the same edit triggered a full rollout that took more than two hours.

2. Kubelet config update

I changed maxPods from 110 to 150. My extension SSHed into each node, patched kubelet config, and restarted kubelet. Nodes stayed in cluster and pods were not evicted.

3. Control plane Kubernetes version update

I bumped KubeadmControlPlane from 1.31.3 to 1.31.4. CAPI correctly decided this should not be in place and performed a standard rolling replacement. That is exactly what I wanted.

Chained upgrades

The other v1.12 feature I really liked is chained upgrades. With ClusterClass, you can bump the cluster topology version and CAPI handles sequencing for you: control plane first, workers after.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: staging
spec:
  topology:
    class: my-cluster-class
    version: v1.31.4  # just bump this
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 8

Change the version, apply, move on. I tested 1.31.3 to 1.31.4 and CAPI did the right thing: control plane nodes rolled one by one, then workers rolled in batches respecting maxUnavailable.

Gotchas

Feature gates must be enabled. You need both InPlaceUpdates and RuntimeSDK.

Your extension must be robust. If it fails halfway, a machine can end up in a bad state. CAPI will not auto-rollback, so make operations idempotent and handle errors carefully.

BYOH support still needs custom work. Metadata changes are straightforward, but OS-level tasks usually require your own extension.

No dry-run yet. There is no built-in preview that tells you whether CAPI will choose in place or rollout for a specific diff. Test in staging first.

Was it worth it?

For our bare-metal setup, yes, absolutely. The label-change case alone saves hours. Chained upgrades also remove a lot of manual coordination and the “did we forget workers after control plane” class of mistakes.

If your nodes spin up in 2-3 minutes on cloud providers, the impact is smaller, but in-place updates still help with low-risk, non-disruptive changes.

My takeaway: start conservative. Let CAPI handle most changes with normal rollouts, and only claim in-place updates where you are very confident it is safe.

Next I want to extend our updater for certificate rotation and kubelet flag changes. I will publish the code once it is cleaned up.