I kept postponing this migration for way too long.

Every time Gateway API came up, I had the same answer: “yeah, I know, I should do it.” Then last week I finally stopped talking about it and migrated three production clusters from Ingress to Gateway API.

After doing it end to end, I wish I had moved sooner.

Why I Finally Did It

The trigger was a multi-tenant cluster where two teams shared the same domain but needed different TLS behavior.

With classic Ingress, that setup gets messy fast. You end up stacking controller-specific annotations, and one team can break routing for the other without meaning to.

Gateway API solves this in a cleaner way:

  • Platform team owns Gateway
  • App teams own HTTPRoute

That separation alone removed a lot of coordination pain for us.

What You Need First

Gateway API is still not built into Kubernetes by default, so first install the CRDs:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Then install a controller that supports it. In my case:

  • New clusters: Envoy Gateway
  • Existing NGINX clusters: NGINX Gateway Fabric
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  -n envoy-gateway-system --create-namespace

Migration Walkthrough

This is a typical Ingress we had:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/rate-limiting: "on"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.example.com
      secretName: app-tls
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app
                port:
                  number: 8080

And this is the Gateway API version.

First the Gateway (platform-managed):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared-gateway
  namespace: gateway-infra
spec:
  gatewayClassName: eg
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"

Then the HTTPRoute (app-managed):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app
  namespace: my-app-ns
spec:
  parentRefs:
    - name: shared-gateway
      namespace: gateway-infra
  hostnames:
    - "app.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: my-app
          port: 8080

The nice part is that there are no free-form annotations here. The API is typed, validated, and far more portable.

Things That Bit Me During Migration

1. Cross-namespace references need ReferenceGrant

If a route in my-app-ns points to a Gateway in gateway-infra, you need this in the Gateway namespace:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-app-routes
  namespace: gateway-infra
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: my-app-ns
  to:
    - group: ""
      kind: Gateway

I forgot this once and stared at “not accepted” for half an hour.

2. Path matching is stricter than old Ingress behavior

With Gateway API, you must be explicit (PathPrefix or Exact).

One service of ours relied on loose matching and broke immediately after cutover.

3. Header-based routing is straightforward now

What used to be awkward with custom annotations is clean in the route spec:

rules:
  - matches:
      - headers:
          - name: X-Canary
            value: "true"
    backendRefs:
      - name: my-app-canary
        port: 8080
  - backendRefs:
      - name: my-app-stable
        port: 8080

This gave us simple canary routing without introducing a service mesh.

4. Weighted traffic splitting is built in

backendRefs:
  - name: my-app-v1
    port: 8080
    weight: 90
  - name: my-app-v2
    port: 8080
    weight: 10

For basic percentage rollouts, this removed a lot of extra tooling.

What About Annotation Features?

Rate limits, timeouts, body size limits still exist, but they move to policy resources (controller-specific):

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: rate-limit
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: my-app
  rateLimit:
    type: Global
    global:
      rules:
        - limit:
            requests: 100
            unit: Second

So yes, some parts are still implementation-specific, but at least they are proper Kubernetes objects with schema validation.

Running Ingress and Gateway API Side by Side

You do not need a big-bang migration.

I ran both for about two weeks using weighted DNS records, validated behavior, then cut over.

# Check Gateway status
kubectl get gateway -A
kubectl get httproute -A

# Check route acceptance
kubectl describe httproute my-app -n my-app-ns

The status field on HTTPRoute is genuinely useful for debugging.

Was It Worth It?

Yes.

The multi-tenant isolation story alone made it worthwhile. But the bigger win is portability. If we change controllers later, the HTTPRoute resources can stay mostly unchanged.

I do not miss annotation chaos.

Quick Migration Checklist

  1. Install Gateway API CRDs
  2. Deploy a Gateway API controller next to your current Ingress controller
  3. Create Gateway resources for shared listeners
  4. Convert Ingress resources to HTTPRoute incrementally
  5. Add ReferenceGrant where cross-namespace access is needed
  6. Test with weighted DNS before full cutover
  7. Remove old Ingress resources after validation
  8. Uninstall the old Ingress controller when everything is migrated

My recommendation: start with one non-critical service, validate your pattern, then batch the rest.