Last week I inherited a cluster with around 40 microservices. Observability was close to nonexistent: basic Prometheus metrics, plus a few random log lines. The team wanted distributed tracing “by next sprint.” There was no realistic way to touch app code across a dozen repos in two weeks.

So I chose OpenTelemetry Operator auto-instrumentation. This is what happened in practice.

The Setup

We run Kubernetes 1.31 on EKS. The goal was simple: get traces and metrics from every service into Grafana Tempo and Mimir without changing application code.

The stack of services was mixed: Python (FastAPI), Node.js (Express), Go, and a couple of Java Spring Boot apps.

Installing the OTel Operator

First, the OpenTelemetry Operator. I used Helm because I wanted to pin the version:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install otel-operator open-telemetry/opentelemetry-operator \
  --namespace otel-system \
  --create-namespace \
  --set manager.collectorImage.repository=otel/opentelemetry-collector-k8s \
  --set admissionWebhooks.certManager.enabled=true \
  --version 0.74.0

Important: cert-manager must be running. If it is missing, webhook certificates fail and pod admission gets stuck. I burned 15 minutes on this because I assumed the cluster already had it. It did not.

# Check if cert-manager is there
kubectl get pods -n cert-manager

# If not:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml

The Collector

Next, I deployed a collector. I chose DaemonSet mode so each node handles its own telemetry. Here is the OpenTelemetryCollector CR:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: otel-system
spec:
  mode: daemonset
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch:
        timeout: 5s
        send_batch_size: 1024
      memory_limiter:
        check_interval: 1s
        limit_mib: 512
        spike_limit_mib: 128
      k8sattributes:
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.deployment.name
            - k8s.pod.name
            - k8s.node.name
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
    exporters:
      otlphttp/tempo:
        endpoint: http://tempo.monitoring:4318
      prometheusremotewrite:
        endpoint: http://mimir.monitoring:9009/api/v1/push
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [otlphttp/tempo]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, k8sattributes, batch]
          exporters: [prometheusremotewrite]

The k8sattributes processor is the real win. It enriches every span and metric with pod name, namespace, deployment, and node. You only realize how useful that is when a trace says “service unknown.”

Auto-Instrumentation Resources

This is where it gets interesting. The operator injects instrumentation during pod admission. You create one Instrumentation CR per language:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-instrumentation
  namespace: otel-system
spec:
  exporter:
    endpoint: http://otel-collector.otel-system:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.50b0
    env:
      - name: OTEL_PYTHON_LOG_CORRELATION
        value: "true"
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.56.0
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.12.0
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:0.19.0-alpha

Then annotate pods, or the whole namespace if you are feeling brave:

# Per deployment
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "otel-system/otel-instrumentation"

Or per namespace for everything:

# Annotate the namespace itself
kubectl annotate namespace production \
  instrumentation.opentelemetry.io/inject-python="otel-system/otel-instrumentation" \
  instrumentation.opentelemetry.io/inject-nodejs="otel-system/otel-instrumentation" \
  instrumentation.opentelemetry.io/inject-java="otel-system/otel-instrumentation"

After that, restart deployments. Injection only happens when pods are created:

kubectl rollout restart deployment -n production --selector=app

What Broke

Go instrumentation is still alpha

Go auto-instrumentation uses eBPF and needs a privileged security context. On EKS with a strict PodSecurityStandard, it simply would not run. I skipped Go services for now and will add manual instrumentation later with the OTel Go SDK. Not ideal, but those services are mostly internal utilities.

Python init container OOM

The Python auto-instrumentation init container tried to install packages into a shared volume. Two of our services had massive dependency trees and the init container hit its 256Mi memory limit. Fix:

python:
  image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.50b0
  resources:
    limits:
      memory: 512Mi

Node.js ESM modules

Three Node.js services used ESM ("type": "module" in package.json). The auto-instrumentation loader does not handle ESM well. It fails silently: no traces, no useful errors. I spent an hour blaming the collector before tracking this down.

The workaround is setting the node options to use the ESM loader:

nodejs:
  env:
    - name: NODE_OPTIONS
      value: "--require @opentelemetry/auto-instrumentations-node/register --experimental-loader=@opentelemetry/instrumentation/hook.mjs"

This still doesn’t cover all ESM cases. For the two services where it still failed, we added three lines of manual instrumentation at the entry point.

Sampling confusion

I set the sampler to parentbased_traceidratio at 0.25 (25%). What I missed at first: with head-based sampling in a service mesh, the entry service makes the decision and downstream services inherit it. That is expected behavior, but it confused the team because they expected every request. For debugging, set it to 1.0, then dial it back.

The Results

After about 20 minutes of real setup work (plus an hour debugging ESM), we had:

  • 34 out of 40 services sending traces to Tempo
  • Automatic span correlation across HTTP calls
  • Pod/namespace/deployment labels on every metric and trace
  • P95 latency dashboards in Grafana within the hour

The 6 missing services: 4 Go (alpha eBPF issue), 2 Node.js ESM edge cases that needed manual SDK setup.

Gotchas I Hit

1. Annotations are per language, not generic. You cannot just say “instrument this pod.” You need to know the runtime and choose the matching annotation. If a pod runs both Python and Node (sidecar pattern), you need both.

2. Init container order matters. If your pod already has init containers that need network access, the OTel init container can race with them. In one service, OTel init ran before the Istio sidecar was ready, so the exporter endpoint was unreachable at startup. A 5-second sleep in the app readiness probe fixed it.

3. Resource overhead is real, just small. Each instrumented pod used about 30-50MB more memory and negligible CPU. For 34 services, that is roughly 1.5GB of extra cluster memory. Plan for it.

4. Version pinning is critical. Operator, collector, and auto-instrumentation images need to be compatible. I had a mismatch between operator 0.74.0 and an older Python image 0.45b0 that caused silent failures. Always check the OTel Operator compatibility matrix.

Finding the Gaps: Which Services Are Not Instrumented?

After the initial rollout, 34 of 40 services had traces. But how do you find the ones that slipped through? I had no dashboard for that, and it took longer than it should have to track them down.

Here is what I do now to catch uninstrumented services before they become blind spots.

Check annotations vs running pods

This one-liner compares what should be instrumented against what actually is:

# List all deployments missing OTel annotations
kubectl get deployments -n production -o json | \
  jq -r '.items[] | select(.spec.template.metadata.annotations == null or
    ((.spec.template.metadata.annotations | keys | map(select(startswith("instrumentation.opentelemetry.io"))) | length) == 0)) |
    .metadata.name'

If a deployment shows up here, it was either skipped intentionally or missed during annotation.

Query the collector for silent services

Even annotated services can be silent. The Python stdlib http.server, certain ESM Node apps, and anything using an unsupported framework will get the init container injected but produce zero spans. You need to check from the other end.

If you use Tempo, query for services that have not sent spans recently:

# Services seen in the last 24h
count by (service_name) (rate(traces_spanmetrics_calls_total[24h]) > 0)

Compare that list against your expected service names. Any service running in production that does not appear in Tempo is either uninstrumented or silently broken.

Build a coverage dashboard

I built a simple Grafana dashboard that joins two data sources:

  1. Kubernetes API (via the Infinity plugin): list of all deployments with their annotation status
  2. Tempo/Mimir: list of services actively sending spans

The panel highlights services in red when they exist in Kubernetes but have no trace data. It took about 30 minutes to build and saved hours of guessing.

Frameworks that auto-instrumentation does not cover

This is the part nobody tells you upfront. Auto-instrumentation relies on monkey-patching known libraries. If your service uses something outside that list, it gets skipped silently.

Python: FastAPI, Flask, Django, aiohttp, Tornado, Falcon, requests, urllib3, SQLAlchemy, psycopg2, redis, celery all work. The stdlib http.server does not. Custom async frameworks are hit or miss.

Node.js: Express, Koa, Hapi, Fastify, and common HTTP clients work. Pure ESM modules are problematic (see earlier section). Custom WebSocket servers usually need manual spans.

Java: Spring Boot, JAX-RS, Servlet, JDBC, and most popular frameworks work well. Java has the most mature auto-instrumentation.

Go: Still alpha. eBPF-based, needs privileged containers. Most teams skip it and use manual SDK.

The safest approach: after deploying auto-instrumentation, run a synthetic request through every service and check Tempo for the resulting trace. If a service handled the request but does not appear in the trace, you found a gap.

What I Would Do Differently

I would start with namespace-level annotations in staging first. I went straight to production (sampler at 0.25) and it worked, but I got lucky. The ESM issue could have caused init container failures and blocked deployments if webhook failurePolicy had been Fail instead of the default Ignore.

I would also use Gateway mode (centralized deployment) instead of DaemonSet for smaller clusters. DaemonSet means one collector per node, which is overkill at 5 nodes. For our 12-node cluster, it made sense.

Wrapping Up

OpenTelemetry auto-instrumentation on Kubernetes works very well for Python, Node.js, and Java workloads. Go support still needs time. The setup is declarative, injection is clean, and you get production-grade observability without editing application code.

If your team keeps saying “we’ll add tracing later,” this approach is a practical shortcut that works. Twenty minutes to get traces flowing beats six months of “we’ll get to it.”