The simpler path is hot → frozen: one day on SSD for live troubleshooting, then Index Lifecycle Management (ILM) moves everything else to the frozen tier as searchable snapshots on blob storage. No warm tier. No cold tier. Frozen indices are partially mounted and read-only. You do not keep indexing into them. You do search them in Kibana and ES|QL, without a restore-then-query workflow.

Elasticsearch maps fields dynamically at ingest, so you do not need a rigid schema before you send logs, metrics, traces, or security logs. The same two-tier ILM policy can cover those data streams. Layer Elastic Streams on top when you need per-stream processing, drops, aggregation, downsampling, or retention. For the companion cost walkthrough, see Not every log deserves 90 days: per-stream retention in Elastic Streams.

When four data tiers is more than log storage needs

Traditional Elastic Cloud Hosted deployments often follow a ladder like this:

Each transition requires ILM policy tuning, node sizing for warm and cold hardware profiles, and replica management across tiers. Frozen still depends on a snapshot repository, but you pass through warm and cold first.

At hundreds of terabytes or multiple petabytes per day, those middle tiers become expensive to operate and hard to explain to finance teams. You pay for spinning disks and replica shards on data that nobody queries at second-level latency anymore.

How hot-to-frozen log storage works

A two-tier hot/frozen architecture collapses the ladder:

| Phase | Storage | Purpose | | --- | --- | --- | | Hot (1 day) | SSD, high IO | Active incidents, alerting, SLO burn rates, ongoing ingest | | Frozen | Blob plus cache nodes | Audit, compliance, historical search (read-only searchable snapshots) |

After the first 24 hours, ILM converts indices to searchable snapshots on the found-snapshots repository that Elastic Cloud Hosted provisions against cloud object storage. Data stays queryable. You omit warm and cold entirely.

The hot-to-frozen pattern matches how Elastic documents large time-series Observability and Security deployments that do not require updates after ingest. New events continue to land on hot. Aged backing indices move to frozen.

Elastic benchmarked a related hot-frozen pattern on ECH with 105 TB of logs over 90 days. Frozen-tier Discover queries stayed within single-digit seconds at p99.9 for most tasks, and holding one day in hot plus the rest in frozen cost roughly 16x less than keeping the full dataset on hot nodes alone. See Measuring searchable snapshots performance.

What petabyte-scale log storage costs on Elastic Cloud Hosted

The figures below are an illustrative example based on a telco-scale workload model, not a quote or pricing commitment from Elastic. Your costs will vary with ingest volume, retention, query patterns, cloud region, hardware profile, and licensing tier. Use Elastic Cloud Hosted pricing and your account team for project-specific estimates.

How does Elastic Cloud Hosted pricing differ from Serverless?

Elastic Cloud Hosted (ECH) bills primarily on the memory footprint of the resources you run: hot and frozen tier node capacity (GB RAM-hour), plus snapshot storage and data transfer. Even with a two-tier hot-to-frozen ILM policy, you still size and pay for the Elasticsearch nodes and frozen cache tier that execute queries against blob-backed data.

Elastic Cloud Serverless bills on ingest volume and stored data, not provisioned cluster RAM. That model removes node sizing, capacity planning, and most lifecycle administration. You pay for what you send and what you keep.

This example models ECH only. Serverless is not included in the tables below because Observability Serverless does not yet support writing aged data out to S3 or blob storage for long-term frozen retention. When that capability ships, the same hot-to-blob pattern on Serverless should reduce administration further while keeping ingest-plus-storage pricing. Until then, ECH with hot-to-frozen ILM is the path for petabyte-scale audit and compliance on blob.

The example assumes 1-day hot, then ILM to blob (frozen searchable snapshots) for long-term retention. Treat the per-PB figures as a planning block for mixed telemetry, not as a claim that 1 PB of raw logs occupies the same disk as 1 PB of raw metrics. Time series data streams store metrics far more efficiently than a regular data stream. Calculations use published ECH capacity, snapshot storage, and data transfer rates with Enterprise licensing.

Log storage cost at 1 PB per day without Streams

At 1 PB/day ingest in this example, ECH alone is modeled at $178K/month ($2.14M/year) using published rates from elastic.co/pricing/cloud-hosted.

Scaling linearly:

| Daily ingest | ECH monthly | ECH annual | | --- | --- | --- | | 1 PB/day | $178K | $2.14M | | 2 PB/day | $357K | $4.28M | | 3 PB/day | $535K | $6.42M | | 4 PB/day | $713K | $8.56M | | 5 PB/day | $891K | $10.70M | | 6.5 PB/day | $1.16M | $13.90M | | 8.45 PB/day | $1.51M | $18.07M |

Even without Streams optimization, the two-tier model is built for petabyte-per-day scale: short hot retention, blob-backed frozen search, and no warm or cold hardware profiles. Competitor TCO depends on ingest mix, retention length, and whether other vendors bill long-term searchability as premium hot storage. Compare published ECH rates to your current quotes rather than treating any single percentage as a universal saving.

How Elastic Streams cuts log retention costs by 28%

Adding Elastic Streams controls (drop, aggregate, downsample, retention) on top of the same hot-to-frozen ILM foundation reduces the example run rate by roughly 28%:

| Daily ingest | Streams optimized monthly | Streams optimized annual | | --- | --- | --- | | 1 PB/day | $127K | $1.52M | | 2 PB/day | $254K | $3.04M | | 3 PB/day | $380K | $4.56M | | 4 PB/day | $507K | $6.08M | | 5 PB/day | $634K | $7.60M | | 6.5 PB/day | $824K | $9.88M | | 8.45 PB/day | $1.07M | $12.85M |

At 5 PB/day with 30% year-over-year growth, the three-year comparison looked like this:

| Year | Daily volume | ECH annual | Streams optimized annual | | --- | --- | --- | --- | | Year 1 | 5 PB/day | $10.70M | $7.60M | | Year 2 | 6.5 PB/day | $13.90M | $9.88M | | Year 3 | 8.45 PB/day | $18.07M | $12.85M |

In this example, that is $12.3M lower over three years with Streams optimization compared to ECH alone. Use that delta as a sensitivity check, then adjust for telemetry mix. Metrics in a TSDS, downsampled series, and verbose logs do not share one storage multiplier.

When Serverless adds blob-tier retention for observability data, rerun this comparison on Serverless pricing. Ingest-plus-storage billing plus zero node administration should make Serverless the simpler operational choice for long-retention workloads at scale.

ILM policy example: one day hot, frozen retention for compliance

On ECH, attach a policy that rolls daily on hot and transitions to frozen searchable snapshots. Omit warm, cold, and delete phases when audit requirements call for indefinite retention on blob.

PUT _ilm/policy/obs-hot-frozen
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "frozen": {
        "min_age": "1d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      }
    }
  }
}

The searchable_snapshot action in the frozen phase mounts a read-only, partially mounted snapshot (partial-*). Keep ingest on hot. Point historical Discover and ES|QL at frozen.

Apply the policy through index templates, Kibana Index Management, or the Streams Retention tab. Frozen tier nodes cache frequently queried blob regions locally. First access to a new time range may be slower; repeat queries hit cache and perform closer to hot-tier latency.

For regulated workloads, pair indefinite frozen retention with Streams AI Partitioning so compliance-heavy streams (payments audit, security events) inherit longer lifetimes while verbose debug streams get shorter DSL delete policies on child streams.

Why Elasticsearch data tiers work for petabyte-scale log storage

Among managed observability platforms, Elastic Cloud Hosted is the only option we are aware of that combines:

• Petabyte-scale telemetry ingest on one platform for logs, metrics, traces, security logs, and more.
• Two-tier ILM from hot to frozen without warm or cold hardware profiles.
• Searchable snapshots that query blob storage directly, with no rehydration step.
• Published-rate TCO modeling you can run against ECH pricing before you talk to finance.

Competing observability SaaS tools often charge premium rates for long hot retention, bill per telemetry type, or move aged data to archives that are not searchable with the same query tools you use for live incidents. At petabyte-per-day ingest, those pricing models widen the gap. Elastic built tiering into the search engine itself, which is why Discover, ES|QL, and APM views work across hot and frozen data transparently.

How to size hot and frozen nodes for petabyte-scale log storage

Exact node counts depend on daily ingest, query concurrency, and hardware profile. A practical starting ratio from Elastic benchmarks:

• Hot nodes sized for one day of ingest plus indexing headroom.
• Frozen cache nodes sized for concurrent query patterns, not total blob volume.

Before production rollout:

1. Measure daily telemetry ingest (logs, metrics, traces, security logs).
2. Pilot the one-day hot policy on a representative data stream.
3. Run Discover and ES|QL queries against frozen data to validate cache sizing.
4. Apply Streams drops and partitioning to reduce volume entering hot.

Keep alert evaluation and SLO burn-rate queries on hot. Plan incident workflows that start on the last 24 hours, then expand into frozen for historical context. Frozen queries can show higher latency on first access after cache eviction.

Searchable snapshots and the frozen tier require appropriate Elastic Cloud capabilities. Review Elastic Cloud pricing and your deployment tier before enabling frozen at scale.

How to move your log retention policy to hot-to-frozen

• Replace warm and cold phases with a hot-to-frozen ILM policy on observability data streams.
• Model TCO at your ingest rate using published ECH pricing, then add Streams optimization scenarios.
• Send OpenTelemetry telemetry to wired streams and apply Streams TCO controls.
• Read the searchable snapshots documentation and data tiers guide.

Petabyte observability does not require four storage tiers or choosing between cost and searchability. One day on hot, everything else in frozen searchable snapshots on blob, and Streams dials on top: that is how Elastic Cloud Hosted keeps full history queryable at a TCO finance teams can approve.

The controls work the same on Elastic Cloud Serverless, Elastic Cloud Hosted (ECH), and self-managed clusters. Elasticsearch uses dynamic mapping (schema-on-write with automatic type detection), so the same Streams controls apply to logs, metrics, traces, and any other signal you ingest over OpenTelemetry or native shippers, with no fixed schema required up front.

UI note (Stack 9.5+): the Streams Retention tab was renamed Data lifecycle. This article uses Data lifecycle for current GA. Screenshots captured on earlier builds may still show the old label; the controls themselves are unchanged. See Configure data lifecycle with Streams.

Why observability data storage costs get out of hand

Most teams start with one retention policy for everything. That is simple until:

• Debug logs from a load generator sit next to payment audit events in the same data stream.
• High-cardinality metrics keep growing while only a handful of dimensions matter after a week.
• Trace volume spikes during deploys, but you still pay to store every span at full resolution for 90 days.

The usual fixes (sampling everything globally, turning off logging, or maintaining brittle routing rules in ingest pipelines) trade away visibility for cost. Streams takes a different path: identify patterns first, then apply the right storage policy to each pattern.

Observability data retention controls in Elastic Streams

Streams sits on top of Elasticsearch data streams. Each stream gets its own Data lifecycle tab, processing pipeline, and (for wired streams) a place in a parent/child tree. That structure is what turns generic "reduce observability spend" goals into concrete settings you can audit.

Streams-native controls (configure these in the Streams UI):

| Control | What it does | Typical signal | | --- | --- | --- | | Drop | Stop indexing matched documents before they hit storage | Noisy debug logs, health checks | | Retention (DSL or ILM) | Delete or tier data after a defined age via the Data lifecycle tab | Short-lived app logs vs compliance logs | | Downsample | Replace high-resolution metrics with fixed intervals (from Elastic 9.4) | Infrastructure metrics after 7 days |

Adjacent Elasticsearch capabilities that compound the savings (not separate Streams actions, but important for TCO):

| Capability | What it does | Typical signal | | --- | --- | --- | | TSDS storage efficiency | Dimension-based routing, _tsid/@timestamp sorting, and synthetic _source improve compression; identical _tsid + timestamp data points are rejected | OTel or Prometheus metrics | | Ingest processors / transforms | Processors reshape or drop individual documents before indexing; transforms pivot already-indexed data into summary destinations on a schedule | High-volume counters, enrichment | | ILM tiering | Move older data to warm, cold, or frozen storage (exposed in Streams Data lifecycle on ECH/on-prem) | Long-lived logs and metrics |

On Elastic Cloud Serverless, you also benefit from storage optimizations built into the platform (for example, LogsDB index mode for logs and efficient TSDB storage for metrics). Streams helps you reduce what you store by routing and trimming data before Serverless bills reflect the volume.

On ECH and self-managed deployments, you get the full Elasticsearch lifecycle toolkit: ILM phases, data stream lifecycle (DSL), searchable snapshots, and downsampling. Streams exposes those policies in the UI so you are not hunting through JSON policy files.

Which control to use

Drop, retention and downsampling solve different problems, and the cost of each is paid at a different point.

| Control | What you still pay for | What you give up | Best fit | | --- | --- | --- | --- | | Drop | Nothing. Matched documents are never indexed. | The data, permanently. No reindex, no recovery. | Health checks, load generator output, debug namespaces | | Retention (DSL) | Full storage until the age limit, then nothing. | Any query beyond the retention window. | Short-lived application logs, anything with a compliance floor | | Retention (ILM tiering) | A lower cost per GB as data moves to warm, cold, or frozen. | Query speed on older data. Frozen needs searchable snapshots. | Long-lived logs and metrics on ECH or self-managed | | Downsample | One data point per interval instead of every sample. | Resolution. Original samples are replaced, not kept alongside. | Infrastructure metrics and other high cardinality metrics after the first week |

Step 1: Find log patterns with AI Partitioning

You cannot right-size retention until you know what is in the firehose. Streams AI Partitioning analyzes incoming documents and proposes child streams based on field patterns, service names, log shapes, or plain-English instructions.

A common starting point is the wired logs.otel stream created when you send OpenTelemetry logs to the Managed OTLP endpoint. The same idea applies to metrics and traces wired streams as your deployment ingests them.

How to run AI Partitioning

1. Open Observability > Streams and select a parent wired stream.
2. Run AI Partitioning on a representative time range.
3. Review suggested child streams (for example per team, service, or log level).
4. Refine partitions in natural language if the first pass merges things you want separated.
5. Apply partitions so each child stream inherits processing and retention from the parent, with per-child overrides.

Once partitioned, a payments API stream can keep 90 days while a notifications worker keeps 7. You set that in the UI instead of maintaining reroute processors in Logstash or the OTel Collector.

For a full walkthrough, see One OTLP endpoint, three teams, zero routing rules.

Step 2: Log filtering in Streams instead of your telemetry pipeline

Partitioning tells you who the data belongs to. Processing tells you what to keep.

In the Streams Processing tab you can:

• Add conditional blocks scoped to one child stream or log shape.
• Use AI-generated Grok patterns to parse samples instead of hand-writing regex.
• Add a drop processor for documents that should never be indexed.

This preview-first workflow matters for TCO. You inspect matched, parsed, and dropped samples before anything goes live. That is safer than a pipeline change that silently discards the wrong field.

Example: drop load-generator logs inside a Kubernetes containers stream while keeping application errors. The Kubernetes logs processing walkthrough shows the full flow with the Dropped tab preview.

Drops are the most direct control when the data has zero long-term value. You pay nothing to store what never gets indexed.

Step 3: Set log retention per stream in the Data lifecycle tab

After partitioning, open the Data lifecycle tab (formerly Retention prior to Stack 9.5) on each child stream.

You will see:

• Current storage size and projected ingestion rates.
• Lifecycle phase breakdown when ILM is active (hot, warm, cold, frozen).
• Options to switch between DSL (simple age-based retention) and ILM (tiered storage).

Wired streams support inheritance: set retention on a parent stream and let children inherit it, or override specific children that need longer or shorter lifetimes. Classic streams can follow existing index template policies while still giving you Streams visibility.

For DSL and ILM editing in the UI, see How Streams simplifies retention management and Configure data lifecycle with Streams.

Step 4: Downsample high cardinality metrics and roll up time series

Logs are often about retention length and drops. Metrics are often about resolution over time.

Starting in Elastic 9.4, you can configure downsampling directly in the Streams Data lifecycle tab for both ILM-backed and DSL-managed streams. Pick a phase, set a fixed_interval, and preview the timeline before you save.

Example ILM policy with hot and cold downsampling

{
  "phases": {
    "hot": {
      "min_age": "0ms",
      "actions": {
        "rollover": { "max_age": "1d" },
        "downsample": {
          "fixed_interval": "5m",
          "wait_timeout": "1d"
        }
      }
    },
    "cold": {
      "min_age": "4d",
      "actions": {
        "downsample": {
          "fixed_interval": "10m",
          "wait_timeout": "1d"
        }
      }
    },
    "delete": {
      "min_age": "30d",
      "actions": { "delete": {} }
    }
  }
}

How TSDS reduces metrics storage

Elasticsearch Time Series Data Streams (TSDS) cuts storage for OTel and Prometheus metrics through dimension-based routing (each time series lands on one shard), index sorting by _tsid and @timestamp that co-locates repeated values for better compression, and synthetic _source. TSDS also rejects duplicate data points with an identical _tsid and timestamp. Pair TSDS with downsampling for metrics that only need minute-level granularity after the first week.

See Configure downsampling directly in Elastic Streams for UI steps and shared-policy warnings.

Storage tiers by deployment type

Observability data retention on Elastic Cloud Serverless

Serverless removes cluster sizing work, but you still control what lands in storage. Streams partitioning, drops, and per-stream retention directly affect ingest volume and retention duration, which drives Serverless observability costs.

Platform-level optimizations (LogsDB for logs, columnar TSDB improvements for metrics) stack on top of the policies you set in Streams. You get efficient storage and intentional data lifecycle management.

If you only need log analytics at a lower price point, Logs Essentials on Serverless is another option. Streams TCO controls still apply when you need per-team retention on Complete.

Storage tiers on Elastic Cloud Hosted and self-managed

ECH and on-prem deployments add storage tiering: hot SSD, warm, cold, frozen, and searchable snapshots. ILM moves streams across tiers automatically. DSL offers a simpler delete-after-N-days model when you do not need tiering.

The same Streams UI and APIs apply. The difference is you also choose hardware profiles and snapshot repositories behind the policies.

How retention differs for logs, metrics and traces

Elasticsearch does not require a fixed schema at ingest. OpenTelemetry logs, metrics, and traces can land in wired streams, get partitioned by AI on whatever fields appear in your data, and pick up stream-specific retention.

That matters for TCO because observability signals are not isolated cost lines:

• Logs: drop noise, shorten retention on verbose services, keep audit trails longer.
• Metrics: use TSDS storage efficiency, downsample after incident windows pass.
• Traces: partition by service or environment, align retention with sampling policies.

You are not maintaining three different cost systems. You are applying the same Streams primitives to each signal type.

Suggested observability data retention workflow

1. Ingest to wired streams (for example logs.otel, metrics, or traces OTel endpoints).
2. Partition with AI so each team or service gets its own stream.
3. Drop known noise (health checks, load tests, debug namespaces).
4. Set retention per stream based on compliance and debugging needs.
5. Downsample or tier metrics and high-volume logs on ECH/on-prem.
6. Review the Data lifecycle tab monthly using ingestion graphs to catch drift.

How much storage can you actually save?

Exact savings depend on your workload. Teams that previously kept everything for 30 days often see the largest wins from:

• Dropping 20–40% of debug or duplicate logs at ingest.
• Shortening retention on high-volume, low-value streams.
• Downsampling metrics after the first few days.

Combine those controls with Elasticsearch storage modes (LogsDB, TSDS) for compounding effects. Measure before and after using the storage size metric on each stream's Data lifecycle tab and cluster-level _stats where you have access.

Try per-stream retention on one noisy stream

• Enable wired streams and send sample OTel data to logs.otel.
• Run AI Partitioning on a busy parent stream.
• Set one aggressive retention policy on a noisy child stream and compare storage after 48 hours.
• Read the Streams documentation for API options (Upsert stream, ingest settings) when you want to automate policies in GitOps.

Observability TCO is not about collecting less data by default. It is about storing the right data for the right duration. Streams gives you the map, the scissors, and the dial.

The Datadog Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes - Overview from infra/datadog/dashboards/integrations/kubernetes.json in the migration repository. It is a cluster-wide board with the signals operators check during an incident: pod counts, CPU and memory by host or pod, non-running pods, and containers stuck in CrashLoopBackOff.

Below are representative queries from the source dashboard:

# Pod CPU by host
sum:kubernetes.cpu.usage.total{$scope,$cluster,$label,$node} by {host}

# Pod memory by pod
sum:kubernetes.memory.usage{$scope,$deployment,$statefulset,$replicaset,$daemonset,$cluster,$namespace,!pod_name:no_pod,$label,$service,$node} by {pod_name}

# Pods not running (pressure / scheduling signal)
sum:kubernetes_state.pod.status_phase{$scope,$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,!pod_phase:running,!pod_phase:succeeded,$label,$node,$service} by {kube_cluster_name,kube_namespace,pod_phase}

# CrashLoopBackOff
sum:kubernetes_state.container.status_report.count.waiting{$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,reason:crashloopbackoff,$scope,$daemonset,$label,$node,$service} by {pod_name}

If this board translates cleanly, most production Datadog Kubernetes folders are worth testing with the same workflow.

Why Datadog-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Datadog moves. Elasticsearch stores Kubernetes metrics efficiently and runs the ES|QL queries those panels use. See Elasticsearch as a metrics backend for benchmark context and storage comparisons.

The platform maps Datadog queries to Kibana panels, validates ES|QL against live data, and writes artifacts you can inspect before anything goes to production.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the Observability Migration Platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

To pull dashboards from the Datadog API instead of JSON files, copy datadog_creds.env.example to datadog_creds.env and set DD_API_KEY, DD_APP_KEY, and DD_SITE.

Ingest Kubernetes metrics first

Empty panels after upload usually mean Elasticsearch does not yet have the series the Datadog queries reference. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. OpenTelemetry into managed OTLP with Kubernetes receivers (kubeletstats, k8s_cluster), then explore in Discover
2. Existing Prometheus or agent pipelines that already write pod and node metrics to metrics-*

The migration CLI accepts --field-profile otel to map Datadog tags such as pod_name, kube_namespace, and kube_cluster_name to OpenTelemetry fields like kubernetes.pod.name and kubernetes.namespace. If panels are empty after migration, verify field mapping and the selected time range before you change translator settings.

Run the Datadog dashboard migration CLI

Export the Datadog dashboard JSON from the UI, or copy the sample kubernetes.json from infra/datadog/dashboards/integrations/ in the migration repo. Place files in a directory such as ./datadog_k8s_exports/.

Run the migration from that directory:

datadog-migrate \
  --source files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --field-profile otel maps Datadog Kubernetes fields to OpenTelemetry field names in Elasticsearch
• --assets all includes dashboards and Datadog monitor definitions when present
• --validate runs emitted ES|QL against your cluster before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source datadog \
  --input-mode files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

To fetch a dashboard from Datadog directly:

datadog-migrate \
  --source api \
  --env-file datadog_creds.env \
  --dashboard-ids YOUR_DASHBOARD_ID \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

Validate the migrated Datadog dashboard in Kibana

Open Kibana → Dashboards and locate the migrated Kubernetes - Overview board. Confirm that cluster and namespace pod counts, CPU and memory series, pod phase panels, CrashLoopBackOff widgets, and deployment replica charts return data for your selected time range.

If you migrated monitors, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains monitor translations when monitors were included in the export.

Handle manual-review panels

Some Datadog widget types do not translate on the first pass. Exotic formulas, log-only panels, and unsupported widgets appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names and data_stream.dataset values in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original Datadog query and simplify or redesign the panel | | Monitor never fires | Confirm the rule is enabled and thresholds match your environment |

Datadog coverage is narrower than Grafana in some areas. Read the migration report before you commit to full parity with leadership. The platform prefers conservative failures over uploading panels that look correct but query the wrong fields.

Related Datadog and Grafana migration guides

For the Grafana PromQL version of this workflow, see Migrate your Grafana Kubernetes dashboard to Elastic Observability. For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.

The migration tool keeps panel queries in PromQL instead of rewriting them into a new dialect, validates them when you pass --validate, and uploads compiled dashboards to Kibana. You still review the migration report and enable alerts on your schedule.

Sample Grafana Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes / Views / Global, a community-style Grafana dashboard in the migration repository. It includes the signals operators check during an incident: namespace CPU and memory, throttling pressure, and restart counts.

Below are representative PromQL queries from the source dashboard:

# Pod / container CPU by namespace
sum(rate(container_cpu_usage_seconds_total{image!="", cluster="$cluster"}[$__rate_interval])) by (namespace)

# Memory working set by namespace
sum(container_memory_working_set_bytes{image!="", cluster="$cluster"}) by (namespace)

# Node / CPU pressure style signal: throttled seconds
sum(rate(container_cpu_cfs_throttled_seconds_total{image!="", cluster="$cluster"}[$__rate_interval])) by (namespace) > 0

# Container restart counts
sum(increase(kube_pod_container_status_restarts_total{cluster="$cluster"}[$__rate_interval])) by (namespace) > 0

If this board translates cleanly, most production Grafana Kubernetes folders are worth testing with the same workflow.

Why Grafana-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Grafana moves. Native Prometheus Remote Write and PromQL in Kibana let you keep the dialect your on-call team already uses. The --native-promql flag passes PromQL through unchanged. See Elasticsearch as a metrics engine for storage and query context.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the observability-migration-platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

How do you get Kubernetes metrics into Elasticsearch?

Empty panels after upload usually mean Elasticsearch does not yet have the series the PromQL references. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. Prometheus Remote Write into Elasticsearch from kube-state-metrics, cAdvisor or kubelet metrics, and node exporters.
2. OpenTelemetry into managed OTLP for Kubernetes receivers, then explore in Discover.

Test with a query such as sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace) against Elastic. If that returns data, continue. If it does not, fix ingest first.

Run the Grafana dashboard migration CLI

Export your Grafana dashboard JSON, or copy the sample k8s-views-global.json from infra/grafana/dashboards/ in the migration repo. Place files in a directory such as ./grafana_k8s_exports/.

Run the migration from that directory:

grafana-migrate \
  --source files \
  --input-dir ./grafana_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --esql-index "metrics-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --native-promql keeps pod CPU, memory, throttling, and restart queries in PromQL
• --assets all includes dashboards and Grafana PromQL alert definitions when present
• --validate runs emitted queries against Elasticsearch before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source grafana \
  --input-mode files \
  --input-dir ./grafana_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

Validate the migrated Grafana dashboard in Kibana

Open Kibana → Dashboards and locate Kubernetes / Views / Global. Confirm that namespace or pod CPU utilization, memory working set panels, throttling or pressure widgets, and restart charts return data for your selected time range.

If you migrated alerts, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains alert translations when alert definitions were included in the export.

What do you do when Grafana panels don't migrate automatically?

The Observability Migration Platform flags PromQL expressions that do not translate automatically. Hard joins, unusual arithmetic, and a few Alertmanager-era edge cases appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names exist in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original PromQL and simplify or redesign the panel | | Alert never fires | Confirm the rule is enabled and thresholds match your environment |

This path migrates Grafana PromQL dashboards and Grafana unified PromQL alert definitions into Kibana. It does not ingest a raw alertmanager.yml. The goal is to keep the PromQL your pager already trusts instead of rebuilding the Kubernetes board from zero.

Related guides

For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.