Kubernetes noisy neighbor in a multi-tenant cluster: Find it with ES|QL

A Kubernetes noisy neighbor cut the share of storefront requests served in under 500 ms from 99.6% to 59%. Every pod on the node stayed running the whole time, with no restarts or alerts. One batch workload was using 1.7 of the worker's two cores, leaving the storefront with 60% of the CPU it needed. Elasticsearch Query Language (ES|QL) queries on kubelet metrics from the Elastic Distribution of OpenTelemetry (EDOT) Collector find it and trace it to its deployment, and capping that deployment at 500m brings 99.8% of requests back under 500 ms. A ResourceQuota holds the namespace to that budget, and a latency Service Level Objective (SLO) tracks the storefront from then on.

Prerequisites

• Elasticsearch and Kibana 9.4+. The TS command.

• A Kubernetes cluster with the EDOT Collector installed through the Kubernetes quickstart, plus kubectl and Helm.

• A latency metric for the affected service. The example uses a k6 probe that exports k6_http_req_duration through OpenTelemetry (OTel).

• A Kibana API key with SLO write privileges for the last section.

You can find the companion notebook here. It reproduces the whole lab from this article, including the tenant labels and the SLO, and cleans up at the end.

The scenario: A batch export and a storefront in a multi-tenant cluster

Two tenants share the worker node of a two-node kind cluster. The worker is limited to two CPUs.

• team-storefront runs an NGINX deployment that gzip-compresses an 8 MiB response. It requests 10m of CPU and has no CPU limit.

• team-batch runs batch-export, a stress container with four CPU workers and 384 MiB of allocated memory. It requests one CPU with a limit of two.

• A k6 probe outside the worker sends one request at a time with a 200 ms pause. A good request returns HTTP 200 in under 500 ms. Running alone, the storefront answers 99.6% of the probe's requests within that threshold.

The worker's MemoryPressure condition stays false for the whole run, so this is a CPU contention case. The Docker limit is a CFS quota that the kubelet doesn’t see, so Kubernetes still reports the Docker virtual machine’s (VM's) six CPUs as the node's capacity, and the queries compare CPU cores consumed instead of a percentage of capacity.

Step 1: Add tenant labels to Kubernetes metrics

Put the ownership labels on each deployment's pod template, where the collector can read them from the pod:

spec:
  template:
    metadata:
      labels:
        tenant: batch
        app.kubernetes.io/name: batch-export

Use stable, low-cardinality values, such as a team or application name. Then add this override to the quickstart's Helm values:

collectors:
  daemon:
    config:
      processors:
        k8s_attributes:
          extract:
            labels:
              - tag_name: tenant
                key: tenant
                from: pod
              - tag_name: app
                key: app.kubernetes.io/name
                from: pod

The Kubernetes attributes processor copies the two pod labels onto every metric document as resource.attributes.tenant and resource.attributes.app. Merge the block with any label extraction rules that your installation already has.

Step 2: Confirm the latency symptom with an ES|QL query

Start from the service that’s slow. This query charts the probe's p95 latency per minute:

FROM metrics-generic.otel-*
| WHERE @timestamp >= NOW() - 30 minutes
    AND service.name == "noisy-neighbor-probe"
    AND attributes.tenant == "storefront"
| STATS p95_ms = PERCENTILE(k6_http_req_duration, 95)
    BY minute = BUCKET(@timestamp, 1 minute), tenant = attributes.tenant
| EVAL p95_ms = ROUND(p95_ms, 1)
| SORT minute
| LIMIT 100

The Managed OTLP (mOTLP) endpoint stores the k6 duration histogram as an exponential_histogram field, which PERCENTILE reads directly. If your pipeline writes the legacy histogram type through the Elasticsearch exporter, cast the field to tdigest before calling PERCENTILE. k6 exports the duration with fixed bucket boundaries, and each bucket is stored as a single representative value, so the percentiles repeat a few numbers. A value of 175 ms stands for the 100–250 ms bucket and 625 ms for the 500–750 ms bucket, so read the percentiles as ranges.

In the scenario, p95_ms sits at 175 before the batch export starts and between 601 and 875 for the 13 minutes it runs. Over the same minutes, the storefront pod reports a k8s.container.restarts of 0 and a k8s.pod.phase of 2, which means Running. Both metrics come from the k8s_cluster receiver, in metrics-k8sclusterreceiver.otel-*.

Step 3: Rank every tenant's CPU use on the affected Kubernetes node

The storefront pod runs on noisy-neighbor-worker. You can read that from k8s.node.name on any of its metric documents. This query ranks everything on that node, with no assumption about which tenant is responsible:

TS metrics-kubeletstatsreceiver.otel-*
| WHERE @timestamp >= NOW() - 2 minutes
    AND k8s.cluster.name == "noisy-neighbor"
    AND k8s.node.name == "noisy-neighbor-worker"
| STATS cpu_cores = SUM(AVG_OVER_TIME(k8s.pod.cpu.usage)),
        memory_bytes = SUM(AVG_OVER_TIME(k8s.pod.memory.working_set)),
        cpu_limit_used = MAX(AVG_OVER_TIME(k8s.pod.cpu_limit_utilization))
    BY k8s.node.name, k8s.namespace.name, resource.attributes.tenant
| EVAL cpu_cores = ROUND(cpu_cores, 3),
       memory_mib = ROUND(memory_bytes / 1048576.0, 1),
       cpu_limit_used_pct = ROUND(cpu_limit_used * 100, 0)
| DROP memory_bytes, cpu_limit_used
| SORT cpu_cores DESC
| RENAME k8s.node.name AS node, k8s.namespace.name AS namespace,
    resource.attributes.tenant AS tenant
| LIMIT 5

AVG_OVER_TIME averages each pod's series over the window, and the outer SUM adds those averages per namespace and tenant. Summing raw samples would count the same pod once per sample. For an incident review, replace NOW() with fixed start and end timestamps.

Three numbers describe the contention:

• team-batch consumes 1.708 cores and runs at 85% of its CPU limit. It stays below its two-CPU limit only because the worker has no more CPU to give.

• team-storefront consumes 0.256 cores. The same query over the baseline minutes returns 0.418, so the storefront gets about 60% of the CPU it needs. A pod that waits for CPU reports only the CPU it received.

• k8s.node.cpu.usage for the worker averages 2.0 cores in the same window, which is all the CPU the worker has. The same TS query with STATS MAX(AVG_OVER_TIME(k8s.node.cpu.usage)) BY k8s.node.name returns it.

Rows with a null tenant are unlabeled workloads. Find their owner before you attribute their consumption to a team.

Step 4: Trace the tenant to its Kubernetes deployment

Narrow the query to the top namespace and group by the application label and the owning deployment:

TS metrics-kubeletstatsreceiver.otel-*
| WHERE @timestamp >= NOW() - 2 minutes
    AND k8s.cluster.name == "noisy-neighbor"
    AND k8s.namespace.name == "team-batch"
| STATS cpu_cores = SUM(AVG_OVER_TIME(k8s.pod.cpu.usage)),
        memory_bytes = SUM(AVG_OVER_TIME(k8s.pod.memory.working_set))
    BY resource.attributes.app, k8s.deployment.name, k8s.node.name
| EVAL cpu_cores = ROUND(cpu_cores, 3),
       memory_mib = ROUND(memory_bytes / 1048576.0, 1)
| DROP memory_bytes
| SORT cpu_cores DESC
| RENAME resource.attributes.app AS app,
    k8s.deployment.name AS deployment, k8s.node.name AS node
| LIMIT 10

The label and the deployment metadata agree on batch-export. For workloads owned by a job or a StatefulSet, group by k8s.job.name or k8s.statefulset.name.

Next, compare each container's CPU use with its request:

TS metrics-kubeletstatsreceiver.otel-*
| WHERE @timestamp >= NOW() - 2 minutes
    AND k8s.cluster.name == "noisy-neighbor"
    AND k8s.node.name == "noisy-neighbor-worker"
    AND k8s.namespace.name IN ("team-storefront", "team-batch")
| STATS cpu_request_used = MAX(AVG_OVER_TIME(k8s.container.cpu_request_utilization))
    BY k8s.namespace.name, k8s.deployment.name, k8s.container.name
| EVAL cpu_request_used_pct = ROUND(cpu_request_used * 100, 0)
| DROP cpu_request_used
| SORT cpu_request_used_pct DESC
| LIMIT 10

The storefront's nginx container runs at 2,546% of its request, and stress runs at 170%. Both run above their requests.

On a saturated node, the requests set each container's CPU weight, and batch's 1000m outweighs the storefront's 10m. The split in Step 3 is not 100 to 1, because the storefront only needs CPU while it compresses a response. The weight still decides who waits, and the storefront is the one that waits.

The EDOT kube-stack values enable k8s.container.cpu_request_utilization and k8s.pod.cpu_limit_utilization. In a plain kubeletstats receiver, both are off by default, so enable them under the receiver's metrics block if you use your own collector.

Step 5: Cap the Kubernetes noisy neighbor with CPU limits and a resource quota

Lower the batch ceiling, and raise the storefront's request:

kubectl set resources deployment/batch-export -n team-batch \
  --requests=cpu=500m,memory=512Mi \
  --limits=cpu=500m,memory=768Mi

kubectl set resources deployment/storefront -n team-storefront \
  --requests=cpu=500m,memory=32Mi \
  --limits=memory=128Mi

Batch is now capped at half a CPU. The storefront has the same CPU request as batch and keeps no CPU limit, so it can use spare CPU. Make the same change in your manifests or GitOps repository so the next release keeps it. Kubernetes requests and limits control scheduling, CPU weight, and ceilings. Latency still depends on what the workload needs, so take production values from measurements.

When both rollouts finish, add a namespace budget, so extra batch replicas go through review:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: batch-budget
  namespace: team-batch
spec:
  hard:
    requests.cpu: "1"
    limits.cpu: "1"
    requests.memory: 1Gi
    limits.memory: 1536Mi
    pods: "2"

The budget fits two replicas at the new settings, which also covers the temporary second pod of a rolling update. If you create the quota earlier, the old batch pod still counts against it with its two-CPU limit, and the quota rejects the new pod that the rolling update needs. A pod that requests two CPUs is rejected at admission with exceeded quota: batch-budget. A ResourceQuota applies to new pods across the namespace. It leaves running pods alone and reserves nothing on a specific node.

Step 6: Verify the fix with the same ES|QL queries

Wait for the rollouts to finish, and then run the tenant ranking query from Step 3 again:

The same query now returns a node with CPU to spare:

A pod at its CPU limit is throttled. The kubeletstats receiver has no CFS throttling counters, so k8s.pod.cpu_limit_utilization is the closest signal in this data stream.

The probe ran through all three phases. Median and p95 come from the Step 2 query, read as bucket ranges, and the request counts and the share under 500 ms come from the Service Level Objective (SLO) in the next section. Times are UTC, in the table and in the Kibana screenshots:

The table leaves out 15:24, the minute of the fix. kubectl set resources replaces both pods, and the probe recorded its only failed requests of the run while the storefront pod restarted. The worker is no longer saturated after the fix, so the batch cap alone accounts for this recovery. The larger storefront request takes effect the next time that the node fills up. Batch now runs throttled, which trades export speed for storefront latency. If the export has a deadline, give it its own node or add capacity.

How do you know the Kubernetes noisy neighbor stays fixed?

Requests, limits, and the quota enforce an allocation. An SLO measures whether that allocation is enough for the service. This one counts a probe request as good when it returns HTTP 200 in under 500 ms, with a target of 99% over a rolling 30 days. The threshold comes from the baseline measurement. Running alone, the storefront answers 99.6% of probe requests in under 500 ms, and during the contention, only 59% make it. 500 ms is also one of k6's bucket boundaries, so the SLO counts whole buckets and never splits one.

Run the k6 probe with OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA in its environment. k6 doesn’t document this variable, but its OTel output is built on the OTel Go SDK, whose OTLP exporter reads it. With delta temporality, each exported histogram holds only new requests, so the SLO counts every request once. On k6 2.x, also set K6_OTEL_METRIC_PREFIX=k6_, because that release exports the metric as http_req_duration without the prefix. Create this SLO through the Kibana SLO API, which accepts the exponential_histogram field directly. The SLO form in 9.5.4 doesn’t list it yet. Save the body as storefront-slo.json:

{
  "id": "noisy-neighbor-storefront",
  "name": "Storefront fairness: 99% successful probes under 500 ms",
  "description": "Track storefront HTTP probes while batch work shares its node.",
  "indicator": {
    "type": "sli.histogram.custom",
    "params": {
      "index": "metrics-generic.otel-*",
      "timestampField": "@timestamp",
      "filter": "service.name: \"noisy-neighbor-probe\" and attributes.test-id: \"noisy-neighbor-final\"",
      "good": {
        "field": "k6_http_req_duration",
        "aggregation": "range",
        "from": 0,
        "to": 500,
        "filter": "attributes.status: \"200\""
      },
      "total": {
        "field": "k6_http_req_duration",
        "aggregation": "value_count"
      }
    }
  },
  "budgetingMethod": "occurrences",
  "timeWindow": {"duration": "30d", "type": "rolling"},
  "objective": {"target": 0.99},
  "groupBy": ["attributes.tenant"],
  "settings": {"syncDelay": "1m", "frequency": "1m"}
}

Three parts of the body carry the definition:

• good counts durations from 0 to 500 ms on responses with status 200.

• total counts every duration, so failed requests stay in the denominator.

• occurrences budgets by request. A slow minute with few requests costs less than a slow minute with many.

The attributes.test-id filter isolates one probe run. For a long-lived SLO, filter on a stable service and environment. Submit the file to your Kibana endpoint:

curl --request POST "$KIBANA_URL/api/observability/slos" \
  --header "Authorization: ApiKey $KIBANA_API_KEY" \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: true' \
  --data-binary @storefront-slo.json

Open SLOs, select the storefront instance, and then check Definition. New data appears after the one-minute sync delay and the next transform run:

About 55 minutes after the fix, the six-hour view of the burn rate panel shows the incident as a single spike and reports Acceptable value, with a six-hour burn rate of 5.36x, under the 6x threshold, and a 30-minute burn rate of 0.45x. The historical Service Level Indicator (SLI) reads 94.95% against the 99% objective. The SLO holds a little over an hour of data at this point, so the incident still weighs on it, and the value climbs as good requests accumulate.

Keep that history after a repair. It records how much service quality the incident cost. Add an SLO burn rate rule to get notified when the budget drains again. An SLO with no incoming data stays quiet, so alert on a silent probe, as well.

What capping the noisy neighbor changed

This walkthrough started from a storefront whose p95 latency moved from the 100–250 ms range to over 500 ms with every pod running. One ES|QL query ranked the tenants on its node and showed a batch export using 1.7 of the worker's two cores. A second query traced it to the batch-export deployment, and a third showed the storefront at 2,546% of a 10m CPU request. A 500m cap on batch brought 99.8% of requests back under 500 ms, and a ResourceQuota set a ceiling for the namespace. The SLO recorded the incident and keeps measuring the service.

More Kubernetes monitoring queries and SLO templates

• Common ES|QL queries for Kubernetes monitoring, for restarts, pod phase, and memory utilization

• Kubernetes observability: SLO templates that turn alerts into error budgets

• TS command reference

• Resource management for pods and containers in the Kubernetes documentation

• Create an SLO burn rate rule