Kibana Universal Alerting will be generally available and on by default in Elastic Cloud Serverless on October 20, 2026, and in Elastic Stack 9.6 for Elastic Cloud Hosted and self-managed deployments. Already have alerts in Kibana? The new system runs alongside your existing Classic Alerting system, so you can adopt what you like at your own pace.

Kibana Classic Alerting continues to handle common scenarios well. Pick a rule type, fill in its parameters, and attach actions as you always have. Classic Alerting is as simple and reliable as ever. Universal Alerting goes further, enabling you to express any ES|QL query as an alerting rule, control flapping with alert and recovery delay, use custom recovery queries, shape alert data to fit your needs, and schedule actions that respond and group alerts across your rules.

When Universal Alerting fits

It's a good fit when one or more of these is true:

1. Your alert scenarios go beyond what Classic Alerting can offer, e.g., custom severities, workflow-powered actions, alert data control, full control over recovery, and more.
2. You want the flexibility of building rules using simplified query builders and later "dropping down" to edit and expand the generated ES|QL.
3. Your alerts come from more than one system, and you want to triage and analyze them in one view.
4. Fine-grained notification control matters, e.g., who is notified and how often, cross-rule grouping and processing, and more.
5. Your team wants all the flexibility offered by Discover, ES|QL, and Workflows. Universal Alerting integrates perfectly with and depends on all of these foundational Elastic features.

If a Classic rule type already covers your scenario, keep using it. We compare the two systems side by side later in this post.

How Universal Alerting works: From ES|QL query rules to alert data

A Universal Alerting rule runs an ES|QL query on a schedule. Each result is written as a rule event to a dedicated data stream, .rule-events. Four core ideas define the system:

• Rules are ES|QL queries, so there are no rule types to pick from. Each row the query returns, identified by its group, becomes a rule event, and the columns it returns are stored with the event. The query defines the shape of the alert.
• Alerts are data. Rule events live in a regular data stream with a documented schema, whether they came from Kibana alerting rules or an external tool. They can be searched in Discover, charted on dashboards, and used as the input for other rules.
• An alert follows one ongoing problem through its lifecycle (pending, active, recovering, inactive) and groups all of its underlying rule events as the alert's history. A rule doesn't trigger a new alert on every run, nor does it update an existing alert document. Rule events can be captured without triggering alerts. These are simply collected as data.
• Action policies live outside rules. They match on alert data, group and throttle them, and send them to workflows. Changing who gets notified doesn't mean editing an action per rule.

Each of these is a separate component, and each one offers flexibility and controls on its own.

The SRE workflow: Author and shape one rule

Say you run production hosts across several availability zones. The goal is to alert when a host stays hot, with the severity level and availability zone the on-call engineer needs. A brief spike shouldn't count. The host examples use metrics from the Elastic Agent System integration (system.* fields), and the pod example later uses the Kubernetes integration (kubernetes.* fields). Swap in your own fields as needed.

Author: One ES|QL rule, written the way you already query

This rule computes average CPU per host, assigns a severity and returns only the hosts worth a look:

/*
  Average CPU per host, with a severity and a minimum-sample guard.
*/
FROM metrics-*
| WHERE system.cpu.total.norm.pct IS NOT NULL
| STATS
    avg_cpu_pct = ROUND(AVG(system.cpu.total.norm.pct) * 100, 0),
    samples = COUNT(*)
  BY host.name, cloud.availability_zone
    /* A column named severity sets the alert's built-in severity field
       (info, low, medium, high or critical). Hosts below every threshold get
       no severity, and the WHERE below drops them. The other returned
       columns, such as avg_cpu_pct and the zone, are stored on the alert and
       available to action policies and dashboards later. */
| EVAL severity = CASE(avg_cpu_pct > 95, "critical", avg_cpu_pct > 90, "high", avg_cpu_pct > 85, "medium")
| WHERE samples >= 10 AND severity IS NOT NULL
    /* A minimum-sample guard skips hosts that just came up and have only
       reported a few data points. */
| KEEP host.name, cloud.availability_zone, avg_cpu_pct, severity, samples

Shape: Decide how the alert behaves over time

The query decides what matches. The rule definition decides what happens as matches come and go. Here's the same rule in YAML, split into a base query, a breach condition, and a recovery condition:

kind: alert

metadata:
  name: host-cpu-saturation
  routing_tags: [infra]                 # links this rule to action policies (lever 5)

schedule:
  every: 1m
  lookback: 5m

query:
  base: |
    FROM metrics-*
    | WHERE system.cpu.total.norm.pct IS NOT NULL
    | STATS
        avg_cpu_pct = ROUND(AVG(system.cpu.total.norm.pct) * 100, 0),
        samples = COUNT(*)
      BY host.name, cloud.availability_zone
    | EVAL severity = CASE(avg_cpu_pct > 95, "critical", avg_cpu_pct > 90, "high", avg_cpu_pct > 85, "medium")
  breach:
    segment: WHERE samples >= 10 AND severity IS NOT NULL

recovery:
  strategy: condition
  segment: WHERE avg_cpu_pct < 70       # recover well below the breach threshold

no_data:
  strategy: keep_last                   # a host that is rebooting keeps its last known status

state_transition:
  pending:
    count: 2                            # alert delay: the alert goes active on the third consecutive breach
  recovering:
    count: 2                            # recovery delay: the alert goes inactive on the third consecutive recovery

grouping:
  fields: [host.name, cloud.availability_zone]   # one alert per host

Recovery, state transitions, grouping and no-data handling are each tunable, and levers 1 to 4 below cover them. Levers 5 and 6 pick up once the alert is active: who gets notified, and how responders triage it.

Six levers for fine-tuning your alerts

Universal Alerting has a control at each stage of the pipeline, listed here from earliest to latest. Use only the ones a scenario calls for.

1. The query: Decide exactly what counts

ES|QL has everything needed to define a condition precisely. WHERE sets the scope, a minimum-sample guard ignores hosts that just came up, CASE assigns severities, and SORT with LIMIT restricts alerts to the worst offenders.

/*
  Alert on only the three hottest pods, not every pod above the line.
*/
FROM metrics-*
| WHERE kubernetes.pod.cpu.usage.node.pct IS NOT NULL
| STATS avg_cpu_pct = ROUND(AVG(kubernetes.pod.cpu.usage.node.pct) * 100, 1) BY kubernetes.pod.name
| SORT avg_cpu_pct DESC
| KEEP kubernetes.pod.name, avg_cpu_pct
| LIMIT 3
    /* LIMIT caps the number of alerts a single run can produce.
       Use it when "the worst few" is the question you care about. */

LIMIT has one side effect to plan for. A pod that drops out of the top three disappears from the results. With no_breach recovery, that counts as recovered, and the alert can activate again if the pod climbs back. If that matters for your rule, pair LIMIT with a recovery delay (lever 3).

2. Grouping: Choose the unit of an alert

STATS ... BY defines what a single alert is. Group by host.name for one alert per host, or by service.name for one per service. Pick the unit someone can act on: a grouping that's too wide hides problems, and one that's too narrow multiplies them. If a grouping field is missing from some documents, those rows fall into a single null group and share one alert, while the other groups keep their own alerts. Handle missing values explicitly in the query if that matters.

Alerting rules return at most 1,000 rows unless you set a LIMIT. If a single run could produce more groups than that, add an explicit LIMIT.

3. Alert delay and recovery delay: Define how an alert starts and ends

Thirty seconds over a threshold isn't always an incident. An instance of recovery isn't always indicative of a real recovery. This kind of flapping is one of the biggest sources of alert noise, and two delay configurations handle it without touching the query.

The alert delay (state_transition.pending) is the number of evaluations an alert spends in the pending state before it goes active. With count: 2, the alert goes active on the third consecutive breach, so a short spike stays pending and never becomes an alert.

The recovery delay (state_transition.recovering) does the same for the recovering state before the alert goes inactive. A signal that bounces around the threshold stays one alert. Each phase accepts a count, a timeframe or both, plus an operator (and or or) that says whether both must be met or just one.

Recovery is a separate setting, and every alert rule must set one (no_breach by default):

• no_breach recovers an alert when its group stops appearing in the breach results.
• condition recovers it when a different recovery condition (as opposed to alert condition) on the same base query is met, as in the rule above.
• query recovers it when a separate ES|QL query returns the alert's group. Recovery is per group, so each alert recovers only when the query returns its group.
• manual never recovers on its own. The alert stays active until someone manually resolves it.

A common pattern is to recover well below the breach threshold; for example, breach at 90% CPU and recover under 67%. That way, an alert doesn't close the moment the metric dips under the line. And because query recovery can run its own ES|QL query, it can check different values than the breach does. The example below requires both CPU and load average to be back to normal before the alert closes:

# A recovery query that looks at more than the breach query does.
recovery:
  strategy: query
  query: |
    FROM metrics-*
    | STATS
        avg_cpu_pct = ROUND(AVG(system.cpu.total.norm.pct) * 100, 0),
        load_avg_5m = AVG(system.load.5)
      BY host.name, cloud.availability_zone
    | WHERE avg_cpu_pct < 67 AND load_avg_5m < 4
    | KEEP host.name, cloud.availability_zone

4. No-data handling: Decide what silence means

When a group with an active alert stops showing up in the breach results, there are two very different explanations. Either the problem cleared up or the data source stopped reporting, and the alert condition alone can't tell them apart. No-data handling lets you decide, per rule, what happens in that moment, so a broken metrics agent doesn't quietly recover a real alert.

Set it with no_data.strategy, or with the no-data setting in the rule builder:

| Option | no_data.strategy | What happens | | --- | --- | --- | | Keep last status | keep_last | Holds the last known state. An active alert stays active, and a recovering alert stays recovering. A status:no_data rule event is written to the data stream.| | Recover | resolve | Closes the alert the first time no data is found, moving it straight to inactive without waiting for the recovery delay. A status:no_data rule event is written to the data stream.| | Do nothing | ignore | Skips the no-data check. With no_breach recovery, a gap in data looks like a recovery. With condition or query recovery, no event is written and the alert stays active. |

The right choice depends on what silence means for the rule. For infrastructure CPU, use Keep last status: if the agent stops sending data, an active CPU breach stays active until data comes back. For a rule that triggers an alert when a job backlog grows too large, Recover makes sense. Once the queue is empty, the query has nothing to return and the alert closes.

5. Action policies: Route, group and throttle

Rules don't carry their own actions. Instead, an action policy selects alerts, bundles them into a payload, controls how often they repeat and dispatches them to a workflow. One action policy can cover alerts from many rules, so changing who gets paged for a hundred rules takes a single edit.

Policy scope selects the alerts an action policy handles, in two parts that can be combined. Routing tags limit the action policy to alerts from rules that carry at least one of those tags. Under Advanced matching, a KQL expression filters on alert fields such as severity, group_hash or data.*. When both are set, an alert has to pass both.

Routing tags connect an action policy to a rule. Add them in the Routing tags field on the Actions step of the rule form, and select the same tags in the action policy form. From then on, the action policy applies to every rule that carries one of those tags, including rules created later. The rule form's Applied policies panel lists every action policy that matches the rule, so the connection is easy to verify. To target a single rule, give it a routing tag that no other rule uses. Attaching or detaching a rule is then a matter of editing its routing tags, and the action policy itself doesn't change.

Routing tags aren't the same as a rule's regular tags. Regular tags group and filter rules in the rule list. They have no effect on notifications, so adding a regular tag to a rule never attaches an action policy to it.

Two more settings shape the notifications. Notify per controls bundling: one notification per alert, one per group of alerts across rules that share a field value such as the availability zone, or a single digest of everything the action policy matches. Frequency limits repeats, for example to at most once every 30 minutes.

Here are two action policies for the CPU rule, written as the settings chosen in the UI. Both are tied to the rule through its infra routing tag:

# Page the on-call engineer only for critical hosts, one page per availability zone.
Name:           infra-oncall-pager
Policy scope:   Routing tags: infra
                Advanced matching (KQL): severity : "critical"
Notify per:     Group, by data.cloud.availability_zone
Frequency:      At most once every 30 minutes
Destination:    pagerduty-infra workflow

# Send the lower severities to the infrastructure channel, batched per zone, at most hourly.
Name:           infra-team-digest
Policy scope:   Routing tags: infra
                Advanced matching (KQL): severity : "high" or severity : "medium"
Notify per:     Group, by data.cloud.availability_zone
Frequency:      At most once every 1 hour
Destination:    slack-infra-team workflow

An alert can match several action policies, and each one applies its own grouping and frequency. If a whole zone runs hot, the pager gets one grouped page rather than one per host, and the team channel gets an hourly summary of the high and medium hosts.

Action policies also work across rules. Give your CPU, memory and disk rules the same routing tag, select that tag in an action policy and set Notify per to Group by data.cloud.availability_zone. A zone where all three rules fire then sends one page instead of three. Switch Notify per to Digest, and the action policy sends a single notification for everything it matches. To alert on the pattern itself, see rules on top of your alerts. Expand commentComment on line R230Resolved Each destination in these examples is a workflow that calls your connector, such as Slack or PagerDuty. To learn how to build and enable one, see the Workflows documentation.

6. Triage controls and maintenance windows: Quiet things on purpose

Once an alert is active, a responder can acknowledge it, snooze the series for a set period or resolve it. Suppressed alerts stay out of notification channels but remain in the data. Nothing is deleted, and everything stays auditable. During a planned rollout, for example, snoozing the series keeps it out of notification channels until the snooze expires. Maintenance windows do the same for scheduled work and apply to Universal Alerting and Classic rules alike.

One place for all your alerts: Classic, Universal, and even external

Few teams have a single source of alerts. Some come from Classic Alerting rules, some from Universal Alerting rules, and some from other monitoring tools. Our new alerts views are built so responders can work from a single queue.

Alerts from other tools, as data

Alerts from external monitoring tools land in the same data stream as the alerts Elastic generates, and are searched, routed and triaged the same way. A workflow writes each one with the alerting.create_alert step, and the source field records where it came from. For the on-call engineer, that's one place to check instead of a tab per tool. External alert ingestion is an experimental feature, so expect it to change and improve as we learn how it's used.

Universal vs. Classic feature comparison

The two systems share a goal but make different tradeoffs. Classic Alerting favors guided setup for known scenarios, while Universal Alerting favors flexibility and control.

| Feature | Classic | Universal | | --- | :---: | :---: | | Prebuilt rule types with guided parameters | ✓ | Partial, more to come | | Rules written as ES|QL queries | ✓ | ✓ | | Basic threshold rules | ✓ | ✓ | | Anomaly detection and APM anomaly rules | ✓ | Coming soon | | Machine learning job health and transform health rules | ✓ | Coming soon | | SLO burn rate rules | ✓ | Coming soon | | Synthetics monitor status rules | ✓ | Coming soon | | Alert fields defined by the rule's query | | ✓ | | Alert event historical data, per run | | ✓ | | Alert delay (consecutive breaches before an alert goes active) | ✓ | ✓ | | Recovery delay (consecutive recoveries before an alert auto-recovers) | | ✓ | | Custom recovery condition or recovery query | | ✓ | | Actions attached to each rule, sent directly to connectors | ✓ | | | Reusable action policies shared across many rules, with grouping and throttling | | ✓ | | Import alerts from external tools (experimental) | | ✓ | | Triage and analysis in the combined Alerts page | ✓ | ✓ | | Maintenance windows | ✓ | ✓ |

Universal and Classic alerts appear together on the combined Alerts page, so responders triage everything in one view. Classic Alerting keeps running exactly as it does today. Maintenance windows, connectors, and workflows are shared across both systems.

New alerting capabilities are actively being built in Universal Alerting. Classic rules remain a good fit for the standard scenarios they were designed for, and you can adopt Universal Alerting at your own pace, one rule at a time.

What Universal Alerting doesn't do (yet)

Universal Alerting doesn't match Classic Alerting feature for feature yet. We'd rather you hear about the gaps now than discover them mid-incident, so here's what to know before moving a rule over.

• Not every Classic rule type has a Universal equivalent yet. Universal rules are ES|QL queries, written by hand or generated by the rule builder. Threshold-style rules, such as custom, metric, log and index threshold rules, map well to the threshold rule builder, and many other conditions can be written directly in ES|QL. Rule types that rely on results computed elsewhere in the stack, such as anomaly detection, don't have a Universal counterpart yet, so those rules stay on Classic for now. The comparison table above lists them, and the rule library on our roadmap will add ready-made starting points for common patterns.
• SLO and synthetics rules need extra work. Their conditions can be written in ES|QL against the SLO and synthetics data, but the multi-window burn rate logic and per-monitor settings of the Classic rules have to be rebuilt by hand. Keep those rules on Classic unless you need the extra control.
• Classic rules are not automatically migrated to the Universal system. They keep running unchanged, and moving one to Universal Alerting means rebuilding its condition as an ES|QL rule.
• Universal Alerting actions depend on workflows. A Classic rule sends its actions straight to connectors. A Universal rule has no actions of its own: action policies hand alerts to workflows, which are the only destination type, and the workflow then calls Slack, PagerDuty or any other connector. Before a Universal rule can notify anyone, at least one workflow must exist and be enabled.
• Workflows require an Enterprise license on Elastic Cloud Hosted and self-managed deployments. Without this license, Universal rules still run and alerts are still written, but action policies cannot deliver to any destination, so no notifications go out. Existing action policies can still be viewed, disabled, snoozed and deleted.
• Universal alerts don't provide flapping state configuration. Alert and recovery delays and custom recovery (lever 3) are the way to keep a noisy alert from firing too often.
• Field names may differ between the two alert types. A query on a Universal alert field such as group_hash won't match any Classic alerts. Our unified alerts table mostly hides this distinction, but use shared filters (status, severity, rule, tags) in your own combined queries, or query each system separately.

Two delivery details apply to every action policy. Notifications are delivered at least once, so in rare cases, such as a restart between dispatch and recording, the same notification can arrive twice. Make workflows idempotent. Notifications are also dispatched on a short polling interval rather than instantly, so don't depend on sub-second paging latency.

Understand your alerting with ES|QL

An alert and its history are documents in a data stream, so the language that defines a rule can also answer questions about your alerting. From an alert, jump into Discover and pivot to that host's metrics and logs without a separate tool. Both examples below query the .rule-events data stream.

/*
  The ten rules that activated the most alerts in the last 7 days.
  rule.id is the rule's saved object ID, not its name.
*/
FROM .rule-events
| STATS alerts = COUNT_DISTINCT(alert.id) BY rule.id
| SORT alerts DESC
| LIMIT 10
    /* Rules at the top are good candidates for tuning. Adjust a threshold,
       raise the alert delay or change the grouping, then re-run this. */

/*
  Alert volume per day for one rule, to confirm a change had the effect you wanted.
  rule.id is the rule's saved object ID, not its name.
*/
FROM .rule-events
| WHERE rule.id == "<your-rule-id>" AND type == "alert"
| STATS alerts = COUNT_DISTINCT(alert.id) BY day = BUCKET(@timestamp, 1 day)
| SORT day ASC

Rules on top of your alerts

Since alerts are data, rules can run over them too. A rule on alert data is an ordinary ES|QL rule that reads .rule-events instead of logs and metrics. That lets it catch a pattern no single alert shows, such as three or more rules firing in the same availability zone. Alerts from Universal rules and from external tools work the same way here, because both land in the same data stream with the same fields. The alerts such a rule produces get the same lifecycle, action policies and triage controls as any other alert.

Signal rules are the building blocks for this. A rule with kind: signal (labeled Events in the UI) records breach and recovery events as data, without triggering alerts or notifying anyone. Use signal rules for conditions that matter mostly in combination, like high CPU, low free disk and elevated I/O wait on the same host. Then write an alert rule that fires when several of those signals line up.

What we're working on next

Try upcoming features per space

New and upcoming Kibana Universal Alerting features will ship behind an experimental feature flag, so teams can try them before they're enabled for everyone. The flag is the Universal Alerting: Experimental Features setting (alerting:v2:experimentalFeatures). It's off by default and set per Kibana space, so you can turn it on in a test or staging space and leave production spaces as they are. Changing it requires a page reload.

The flag gates every upcoming Universal Alerting feature. Everything that ships at GA works without it.

What's coming

While we aren't committing to any specific dates, this is where we're focused after GA:

• Sequence rule builder GA: A guided way to build rules that detect events happening in a particular order, without writing the whole query by hand (already available via "experimental features").
• More rule builders: Provide more guided ways to build rule queries. (SLO, synthetics, anomaly detection, etc.)
• A rule library of ready-made rules to start from and adapt, so a new rule doesn't begin with a blank query.
• Alert deletion, for alerts that are no longer needed.
• External alert import, generally available, with simpler setup and management.
• Bi-directional alert syncing with external systems, so an action taken in one system is reflected in the other.
• Further "no data" features.

If one of these matters more to you than the rest, tell us. It helps us decide what to build first.

Get started

Universal Alerting is generally available on Elastic Cloud Serverless, and on Elastic Cloud Hosted and self-managed deployments starting with Elastic Stack 9.6. On Elastic Cloud Hosted and self-managed deployments, workflow-based action policies require an Enterprise license, which includes 10,000 workflow executions per month (see subscriptions). On Serverless, Elastic manages the alerting system.

Give your team access

Access is controlled with Kibana feature privileges, set per role in Stack Management > Roles. Universal Alerting adds four, listed under Universal Alerting in the role editor, each with its own All or Read level:

| Privilege | Read | All | | --- | --- | --- | | Rules | View rules | Create, edit and delete rules | | Alerts | View alerts | Triage alerts | | Action Policies | View action policies | Create and edit action policies | | Execution history | View execution history | Same as Read |

Most responders need Alerts: All and Rules: Read. Rule authors need Rules: All, plus Action Policies: All if they also manage notifications. Notifications run through workflows, so anyone who builds or edits them also needs All for Analytics > Workflows. See Set up Workflows for the full list of Workflows privileges.

Privileges control what a user can manage, not what data a rule can read. Each rule runs with the API key of the user who last saved it, and that key determines what data the rule can query. Save your rules as a user who can read the indices they query. For how rules, action policies and workflows are authorized, see the authorization documentation.

Create your first rule

Open the Universal rules tab, create your first rule and tell us what you find. If something comes up that we didn't cover here, we want to hear about it.

Don't have a deployment yet? Start an Elastic Cloud free trial.

Production is outrunning the tools that watch it. Teams ship faster than ever, and more of that code is written by AI. Yet keeping production healthy is still manual work. Someone has to write the alert rules that catch problems and work out why each alert fired, and then they have to apply the fix. Much of what they need to know lives only in the minds of a few engineers. Every release changes what needs watching, and this work has become the bottleneck.

Elastic nightshift AI SRE works alongside your team, with the full context about your systems. It watches your telemetry around the clock and detects problems, including the unknown ones that no alert rule covers. It investigates incidents the way that an experienced site reliability engineer (SRE) would, testing hypotheses against your telemetry data, such as logs, metrics, traces, and profiles, ruling out dead ends and showing you evidence for the root cause. Then it recommends a fix to remediate the problem and applies it automatically, if you want it to.

At every step, it uses the most relevant contextabout your data and systems, built from your telemetry, code, runbooks, and more. It learns proactively from your data and prior incidents, along with enterprise knowledge. And, like you and your team, it investigates more accurately with every incident it detects or any alert it investigates. Plus, it can work with you in the Kibana UI, a terminal, an agent of your choice, or anywhere else you work.

When we were building this system, it was incredibly important that we earn your trust. If you've been using large language models (LLMs) for a while, you know how overly confident some tend to be. With Elastic nightshift AI SRE, instead of just getting the conclusion, you can follow all the steps that the agent took to get there and even drill down into the raw data, if you need to, as the following animation shows. You can check out the interactive version of this article here.

How Elastic nightshift AI SRE works

Elastic nightshift consists of four engines that work together (or you can use them on their own): Context, Detection, Investigation, and Remediation. For example, you can have it investigate alerts from the monitoring tools that you already use without turning on detection.

Engine

What it does

What triggers it

What you get

Context

Builds knowledge of your environment from telemetry, code, runbooks, and wikis

Runs continuously from the moment it has access to your data and systems

Entities, relationships, and a long-term memory of past incidents that the other three engines draw on

Detection

Evaluates telemetry continuously and flags what needs attention, with no alert rules to write or maintain

Always on when configured for data in Serverless

A significant event grouping related signals across services, with the parts of your system that it affects

Investigation

Tests hypotheses in parallel against logs, metrics, traces, and profiles, separating symptoms from causes

Any alert, fired in Elastic or a connected tool; optionally a significant event

A root cause in plain language, affected services, proposed actions, and the blind spots that it couldn’t verify

Remediation

Picks a workflow or connector action and fills in the parameters for this incident, or gives you commands to run

A completed investigation

A fix that you review before it runs (or apply automatically, if you choose)

Most importantly, you don't have to move your data to use it. The agents run on Elastic Observability Serverless and reach out to your telemetry and alerts in the systems where they already live. By technical preview later this year, it will work with:

• Data in Elastic Observability Serverless (sending data there is optional).

• Elastic Cloud Hosted (ECH) deployments.

• Self managed clusters (provided that the Elasticsearch endpoint is exposed appropriately).

• Popular third-party observability tools.

• Popular telemetry stores and databases.

When you sign up for the waitlist, tell us which third-party tools, telemetry stores, and databases you want most so we can prioritize accordingly. In addition, we’re working toward a plan to deploy all this functionality in self-managed environments. We’ll have more to share later this year.

Context Engine: Giving an AI SRE knowledge of your systems

When an experienced SRE joins a new team, their skills come with them, but their knowledge of the system doesn't. They spend weeks learning the architecture, the runbooks, and the history of past incidents, along with which services are always noisy. What makes an SRE fast is the combination: years of experience plus deep knowledge of the system they run.

The same is true for AI agents. LLMs bring some of the experience but none of the knowledge of your system. Without context, an agent spends time and tokens working out how things fit together before it gets to the actual problem.

The Context Engine gives Elastic nightshift that knowledge before an incident starts. It's the layer that ties the whole system together.

The Context Engine builds its knowledge in three ways:

• Ahead of time. As soon as it has access to your data and systems, it extracts knowledge from your telemetry; that is, the entities in your environment, such as services and hosts, and the relationships between them. It can also read your wikis, runbooks, and code from systems like GitHub.

• From its own work. Each investigation and detection adds to a long-term memory of what happened and what fixed it.

• From your interactions and input. Give it feedback about detections, or feed it a postmortem doc. You can also manually upload important information.

Remembering facts like these is different from the investigation learning described earlier. The Context Engine remembers what's true about your environment, while the Investigation Engine gets better at how it investigates.

Elastic stores telemetry efficiently enough that you can keep it unsampled, and the agent can search everything within your retention period. Because that telemetry and the knowledge built from it live on Elastic alongside the agent, it can go back to the raw data behind its conclusions.

Investigation Engine: AI root cause analysis from any alert

Alerts pile up, and each one that matters still lands on an engineer who has to piece together logs, metrics, traces, and recent changes by hand before they can even consider a fix.

Your new AI SRE coworker takes on that work, using the Investigation Engine. An investigation can start automatically from any alert, whether it’s fired in Elastic or in another tool that you've connected. From there, it does what a good on-call engineer would:

1. Checks what changed.

2. Forms hypotheses about the cause.

3. Tests them in parallel against your logs, metrics, traces, and profiles.

4. Follows the leads that the evidence supports and drops the ones it doesn't.

5. Separates symptoms from causes.

6. Converges on the most likely root cause.

7. Proposes a fix.

The core logic in our Investigation Engine comes from Deductive AI, which Elastic acquired, and builds on Deductive's three years of work on automated incident investigation.

The nightshift agent plans a step and runs the tools to carry it out. It observes what comes back and checks its conclusions against the evidence before moving on. Across investigations, it learns which paths lead to useful evidence and successful outcomes, so it gets better at investigating with every incident.

We talked earlier about building trust when using agentic systems, and this is one example of earning it from you. You can expand any step of an investigation to see the reasoning behind it and the queries that the agent ran. You can also see the raw data that it ran them against.

When an investigation finishes, you get:

• A conclusion. The root cause, in plain language, and the immediate fix that we’d recommend.

• Impact. Which services are affected and how many problems each one has.

• Proposed actions. Next steps, each waiting for your review.

• Blind spots. What the agent couldn't verify, so you know where its confidence ends.

Detection Engine: How do you catch incidents with no alert rule?

The hardest incidents are often the ones that never triggered an alert to begin with. As a result, the service degrades and users notice. Monitoring stays quiet. No one can write an alert rule for a failure that they haven't imagined, and when something does fire, the related signals are scattered across services, each raising its own alert.

Your new coworker also takes on that work, using the Detection Engine, which continuously evaluates your telemetry and flags what needs attention. You don’t have to write or maintain any alert rules. When related signals show up across several services, it groups them into a single significant event that shows the signals behind it and the parts of your system that it affects. Significant Events can start an investigation automatically, if you choose, so that it’s already underway by the time someone looks.

Remediation Engine: Automated remediation from the diagnosis

Historically, Observability often stopped at the diagnosis. It could tell you that something was wrong, and eventually why, but the fix was up to you. Runbook automation could only replay the fixes that someone had scripted in advance, and real incidents rarely follow the script.

Your same new coworker also takes on this work, using the Remediation Engine, which recommends a fix. It can pick one of your existing workflows or connector actions in Elastic Workflows and fill in the parameters for this incident, or it can give you commands or instructions that you can copy into a terminal or a coding agent. Unlike a runbook script, the action and its parameters come from the diagnosis of the incident in front of it. You decide how much it does on its own; you review each action and the parameters it chose before it runs, or let it apply fixes automatically.

Using the AI SRE from Kibana, your terminal, or Slack

Kibana will always be a first-class home for Elastic nightshift, and we'll keep making it a great experience. The Elastic nightshift home page lists your team’s open and recent investigations, sorted by severity, so whoever is on call can see at a glance what needs attention and what's already been handled.

However, more and more engineers want to use our products from the tools they already live in, like a terminal, a coding agent, or a chat channel, and we're embracing that:

• Elastic CLI. You'll be able to pick up an investigation from your terminal or from a coding agent, like Claude, Cursor, or Codex, and carry on from there. Where Model Context Protocol (MCP) is the easier way to connect an agent, you can use that instead.

• Slack. We're building a Slack integration that makes Elastic nightshift feel like a coworker in your channels. It will post finished investigations (and answer questions about them) where your team already talks. More importantly, it will learn from any interaction in Slack. Nudge it along with additional context, or post a root cause analysis or postmortem document in the Slack thread to automatically trigger a learning loop that will use all of this information to get better for the next incident. 

Join the waitlist for the AI SRE private preview

Elastic's own SRE team already runs Elastic nightshift against our production environment, and what team members learn goes straight back into the product. With the private preview, we're opening it to more teams.

• Who can join: Existing Elastic users, at no cost during the early access period.

• What's included: A preview of the four engines described above, with more capabilities arriving every week.

• What we ask: Honest feedback. We'll work directly with preview teams, and what you tell us will shape what we build next.

• What's next: Technical preview in Elastic Cloud Serverless later this year.

Sign up for the private preview

This is an alert noise reduction exercise done with queries. Kibana writes every alert into a hidden index and keeps it there after recovery, so we put three questions to that alert history with ES|QL. Was this one incident or many, where did it start, and which rule produced more noise than signal? Then we change the grouping key on the noisy rule and replay the same failure, and the same query comes back with one episode that lasts seven minutes.

Kibana stamps every alert document with a rule revision, so the before and after of that tuning change sit in one index. At the end, an Agent Builder agent answers the same questions over the same alert history.

Everything below was tested on Elasticsearch and Kibana 9.5.2 on Elastic Cloud, with OpenTelemetry Collector Contrib 0.157.0.

Prerequisites

• Elasticsearch and Kibana 9.5 or later, on Elastic Cloud or self-managed.
• Kibana privileges to create and edit Elasticsearch query rules and Observability custom threshold rules, plus read access to the .alerts-* indices.
• OpenTelemetry Collector Contrib 0.157.0 or later, and an Elasticsearch API key with ingest privileges for the OTLP endpoint.
• Agent Builder enabled, for the final section only. Everything before it is plain ES|QL and works without it.

Why alert history beats alert notifications

A notification answers one short-lived question: who should look at this right now?

Most of the questions that make alerting better arrive later, and they are questions about data:

• Did 18 alert episodes represent 18 problems, or one incident evaluated from 18 angles?
• Which service showed the first symptom, and did the failure follow the dependency chain?
• Which rules produce short, repetitive episodes that nobody can act on?
• Did last month's threshold change actually reduce noise?

Inside Elastic, this framing has a name: alerts as data.

The idea is that every alert and every state transition is a fact in time, stored with explicit semantics, so correlation becomes a query across streams rather than a bespoke feature.

It also matches what shows up in real operations channels: a single infrastructure failure can open dozens of alerts within seconds, and flapping alerts with miscalibrated thresholds are a familiar source of page fatigue.

Both problems are hard to solve from notifications and straightforward to study from preserved alert history.

What Kibana alerting stores in every alert document

Kibana alerting rules write alerts-as-data documents into hidden indices that you can query like any other index.

The two aliases used in this article are .alerts-stack.alerts-default for Elasticsearch query rules and .alerts-observability.threshold.alerts-default for the Observability custom threshold rule.

The unit of storage is what we will call an episode: one document that covers one alert instance from the moment it becomes active until it recovers.

The fields that matter for investigation:

| Field | What it tells you | | :---- | :---- | | kibana.alert.instance.id | The grouping-key value this episode belongs to | | kibana.alert.start, kibana.alert.end | When the episode opened and recovered | | kibana.alert.duration.us | How long it stayed active | | kibana.alert.status | active or recovered | | kibana.alert.flapping, kibana.alert.flapping_history | Whether Kibana marked it as rapidly changing state | | kibana.alert.rule.name, kibana.alert.rule.revision | Which rule, and which version of it | | kibana.alert.grouping | The structured grouping values, for example service.name | | kibana.alert.reason | The human-readable condition at trigger time | | kibana.alert.url | A deep link back to the rule's data view |

Two properties make the rest of the article possible.

First, a new episode for the same instance creates a new document, so counting documents counts episodes.

Alert documents and notifications are decoupled: whether an episode also pages anyone depends on the actions attached to the rule and on their frequency, which can be every check, only on status changes, or a summary on a custom interval.

Our three rules have no actions attached, so nobody was paged during the lab; everything we count from here on is an alert document.

Second, grouping fields with ECS names such as service.name are copied to the top level of the alert document, which means alert history can join against other indices.

One caveat before you query: alert indices are wide.

The alerts index on this deployment maps roughly 1,400 fields, so every ES|QL query in this article projects a narrow set of columns with KEEP, and you should do the same, especially in tools that feed an LLM.

Because these are hidden system indices, the Discover editor cannot always introspect their fields, so it may flag the query with a warning or even an error badge while still running it and returning rows. Trust the result count in the toolbar, not the badge.

One more orientation note: Kibana screenshots below render timestamps in the browser's local time (UTC-5 in this lab), while the query result tables in the text use UTC.

The lab: how one bad deploy produces an alert storm

To generate an honest storm, we need services that fail in a chain, not a single threshold breach.

The scenario: fraud-scorer 2.3.0 ships with slow model loading, the payments queue backs up, payment-gateway requests start timing out, and customers see failed checkouts on payments-web.

Every service writes structured JSON logs, and an OpenTelemetry Collector tails them with a filelog receiver and ships them over OTLP/HTTP to the deployment's Elastic OpenTelemetry endpoint using an API key.

receivers:
  filelog/payment_gateway:
    include:
      - ${env:LAB_DIR}/events/payment-gateway.jsonl
    start_at: end
    resource:
      service.name: payment-gateway
      service.namespace: payments
      service.version: 5.1.2
      cloud.region: us-central1
    operators:
      - id: parse_json
        type: json_parser
        parse_from: body
      - id: parse_severity
        type: severity_parser
        parse_from: attributes.severity_text
      - id: use_message_as_body
        type: move
        from: attributes.message
        to: body

exporters:
  otlphttp/elasticsearch:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      Authorization: ApiKey ${env:ES_API_KEY}

A transform processor routes everything into one data stream, logs-payments.stormlab-default, and the records land with service.name, log.level, custom string attributes under labels.*, and numeric attributes under numeric_labels.*.

The full collector configuration for all three services, and a notebook that reproduces the whole lab, are in the companion repository.

We also load one small index that will do a lot of work later: a service catalog.

PUT payments_service_catalog
{
  "settings": { "index.mode": "lookup" },
  "mappings": {
    "properties": {
      "service.name": { "type": "keyword" },
      "team":         { "type": "keyword" },
      "tier":         { "type": "keyword" },
      "depends_on":   { "type": "keyword" }
    }
  }
}

Three documents map each service to its owning team, tier, and upstream dependency: fraud-scorer depends on payments-queue, payment-gateway on fraud-scorer, and payments-web on payment-gateway.

POST payments_service_catalog/_bulk
{ "index": {} }
{ "service.name": "fraud-scorer",    "team": "risk-ml",       "tier": "backend",  "depends_on": "payments-queue" }
{ "index": {} }
{ "service.name": "payment-gateway", "team": "payments-core", "tier": "edge",     "depends_on": "fraud-scorer" }
{ "index": {} }
{ "service.name": "payments-web",    "team": "storefront",    "tier": "frontend", "depends_on": "payment-gateway" }

index.mode: lookup is what later allows LOOKUP JOIN from ES|QL.

How grouping keys decide how many alerts you get

The storm is observed by three alerting rules, and their grouping keys are the real subject of this article.

| Rule | Rule type | Grouping key | Window | Threshold | | :---- | :---- | :---- | :---- | :---- | | gateway-5xx-per-endpoint-status | Elasticsearch query rule (ES|QL mode) | labels.endpoint, labels.status_code, cloud.region | 1 minute | 3 or more errors | | fraud-scorer-queue-lag | Elasticsearch query rule (ES|QL mode) | service.name, cloud.region | 1 minute | Max lag of 30 seconds or more | | payments-error-rate-per-service | Observability custom threshold rule | service.name | 2 minutes | More than 5 ERROR documents |

The first rule is deliberately miscalibrated in a way we have all shipped at some point: it groups gateway errors by endpoint and status code with a one-minute window and a low threshold.

FROM logs-payments.stormlab*
| WHERE service.name == "payment-gateway" AND log.level == "ERROR"
| STATS error_count = COUNT(*) BY labels.endpoint, labels.status_code, cloud.region
| WHERE error_count >= 3

It runs every minute as an Elasticsearch query rule in ES|QL mode, and each returned row becomes one alert instance, so /api/payments,503,us-central1 and /api/payments/confirm,504,us-central1 alert separately.

The second rule watches the first domino directly: consumer lag on the fraud queue, reported by the service in numeric_labels.queue_lag_seconds.

FROM logs-payments.stormlab*
| WHERE service.name == "fraud-scorer" AND numeric_labels.queue_lag_seconds IS NOT NULL
| STATS max_lag_seconds = MAX(numeric_labels.queue_lag_seconds) BY service.name, cloud.region
| WHERE max_lag_seconds >= 30

The third rule is an Observability custom threshold rule: more than five ERROR documents in two minutes, grouped by service.name.

It represents the sane default: one stable alert instance per affected service. It also demonstrates that different rule types feed the same investigation because all of them write alerts-as-data documents.

All three rules carry the tag alerts-as-data-v2, which is how every query below isolates them.

Tags on rules are not cosmetic: kibana.alert.rule.tags is stored on every alert document, so a tag is effectively a queryable dataset label for your alert history.

Replaying the incident: 18 alert episodes in seven minutes

We replay the incident: three minutes of healthy baseline, the 2.3.0 deploy, queue lag climbing past the threshold, six minutes of rotating gateway 5xx bursts, customer-visible errors, then a rollback and recovery.

The Alerts page during the storm looks like the pager felt.

Notice the table, not just the count.

The same instance IDs appear both as Active and as Recovered within minutes of each other, which is churn you can see but not yet measure.

When the dust settles, the preserved history can be queried in Discover's ES|QL mode.

The raw material of the storm, the error logs themselves, confirms the cascade shape:

FROM logs-payments.stormlab*
| WHERE labels.incident_id == "payments-brownout-20260722" AND log.level == "ERROR"
| STATS errors = COUNT(*) BY minute = DATE_TRUNC(1 minutes, @timestamp), service.name
| SORT minute

![Discover ES|QL view of error counts per minute and service: fraud-scorer errors first, then payment-gateway at seven per minute, then payments-web]]3

Now we leave the logs behind and interrogate only the alert history.

How to tell if an alert storm was one incident or many

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2"
| STATS episodes = COUNT(*),
        alert_instances = COUNT_DISTINCT(kibana.alert.instance.id),
        first_episode = MIN(kibana.alert.start),
        last_episode = MAX(kibana.alert.start)
  BY rule = kibana.alert.rule.name
| SORT episodes DESC

| rule | episodes | alert_instances | first_episode | last_episode | | :---- | ----: | ----: | :---- | :---- | | gateway-5xx-per-endpoint-status | 14 | 7 | 16:36:38 | 16:42:38 | | payments-error-rate-per-service | 3 | 3 | 16:36:41 | 16:37:41 | | fraud-scorer-queue-lag | 1 | 1 | 16:35:35 | 16:35:35 |

Eighteen episodes, one incident, and every one of them opened inside a seven-minute window.

The gateway rule alone produced 14 of the 18 episodes while tracking only 7 distinct instances, which means half of its episodes were re-opens of something that had just recovered.

That single row is the quantified version of a complaint that usually stays anecdotal: this rule fires more than it helps.

How to find which service failed first with ES|QL

Because the grouping fields were ECS names, each episode carries service.name at the top level, and we can join the alert history against the service catalog.

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2" AND service.name IS NOT NULL
| STATS first_symptom = MIN(kibana.alert.start) BY service.name
| LOOKUP JOIN payments_service_catalog ON service.name
| KEEP first_symptom, service.name, team, tier, depends_on
| SORT first_symptom

| first_symptom | service.name | team | tier | depends_on | | :---- | :---- | :---- | :---- | :---- | | 16:35:35 | fraud-scorer | risk-ml | backend | payments-queue | | 16:36:41 | payment-gateway | payments-core | edge | fraud-scorer | | 16:37:41 | payments-web | storefront | frontend | payment-gateway |

The alert order matches the dependency chain exactly: the first symptom belongs to team risk-ml, the edge degraded 66 seconds later, and customers noticed a minute after that.

There is a subtler deduction available in this table.

The earliest alerting service depends on payments-queue, which never alerted at all, so the most likely true origin sits one hop upstream of the first visible symptom, and that is where a post-incident review should start reading logs.

Also note which rule is absent from this analysis: the noisy gateway rule grouped by endpoint and status code, so its episodes carry no service.name and cannot participate in the topology join.

A grouping key is the data contract for your future investigations.

How to score which alerting rule is noisiest

A noise scorecard query condenses rule quality into one table.

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2"
| EVAL minutes_active = COALESCE(kibana.alert.duration.us, 0) / 60000000.0
| STATS episodes = COUNT(*),
        alert_instances = COUNT_DISTINCT(kibana.alert.instance.id),
        flapping_episodes = SUM(CASE(kibana.alert.flapping == true, 1, 0)),
        median_minutes_active = MEDIAN(minutes_active)
  BY rule = kibana.alert.rule.name, revision = kibana.alert.rule.revision
| SORT episodes DESC

| rule | revision | episodes | instances | flapping | median minutes active | | :---- | ----: | ----: | ----: | ----: | ----: | | gateway-5xx-per-endpoint-status | 0 | 14 | 7 | 0 | 1.0 | | payments-error-rate-per-service | 0 | 3 | 3 | 0 | 7.0 | | fraud-scorer-queue-lag | 0 | 1 | 1 | 0 | 9.0 |

Read the last column first.

A median active time of one minute means the typical episode from the gateway rule was already resolving itself before the on-call could even sign in, while the two well-grouped rules produced episodes long enough to represent the actual incident.

The flapping column is Kibana's own attempt at the same problem, and it is damage control.

The real fix is upstream, in the rule definition, and the alert history has just told us exactly what to change: the grouping key manufactures instances, and the one-minute window manufactures re-opens.

Alert noise reduction: fixing the rule and proving it worked

The rule's own detail page already summarizes the problem: 14 alerts in 24 hours, every one of them from a single incident, produced by the definition on the right.

We update the rule rather than replace it, changing the grouping to service.name, widening the window to five minutes, and raising the threshold.

FROM logs-payments.stormlab*
| WHERE service.name == "payment-gateway" AND log.level == "ERROR"
| STATS error_count = COUNT(*) BY service.name, cloud.region
| WHERE error_count >= 10

Updating a rule in place matters for one specific reason: Kibana stamps every alert document with kibana.alert.rule.revision, so the before and after of a tuning change are separable in the same index with no bookkeeping on our side.

Then we replay a compressed version of the same failure: same deploy, same lag ramp, same rotating gateway bursts, same customer impact, rollback after seven minutes.

The scorecard query runs unchanged, and now the revision column earns its place:

| rule | revision | episodes | instances | flapping | median minutes active | | :---- | ----: | ----: | ----: | ----: | ----: | | gateway-5xx-per-endpoint-status | 0 | 14 | 7 | 0 | 1.0 | | payments-error-rate-per-service | 0 | 6 | 3 | 0 | 6.0 | | fraud-scorer-queue-lag | 0 | 2 | 1 | 0 | 8.0 | | gateway-5xx-per-endpoint-status | 1 | 1 | 1 | 0 | 7.0 |

The unchanged rules now show both incidents in their rows, six and two episodes, which is exactly what stable rules should do.

The gateway rule splits into two rows because the revision changed, and the contrast is the entire argument of this article in four numbers.

Same failure shape, one episode instead of fourteen, and the median episode now lasts seven minutes instead of one, so it spans the incident instead of a fraction of it.

This is the part of alerting work that usually relies on faith, and here it is a query result: the tuning change is verified against the same incident pattern, in the same index, by the same ES|QL.

Nothing about this loop is specific to our lab rule.

Any rule that tags its alerts can be scored, tuned, and re-scored this way, which turns alert tuning from an opinion into a measurement.

Querying alert history with an Agent Builder agent

Every query above is deterministic, which makes them ideal tools for an AI agent.

We register three ES|QL tools in Agent Builder, each one a parameterized version of a query from this article with an explicit KEEP projection, and wire them into an agent called Alert Historian.

POST kbn:/api/agent_builder/tools
{
  "id": "alert_noise_scorecard",
  "type": "esql",
  "description": "Noise scorecard per alerting rule and revision: episodes, distinct instances, flapping episodes, median minutes active.",
  "configuration": {
    "query": "FROM .alerts-* | WHERE kibana.alert.rule.tags == \"alerts-as-data-v2\" AND DATE_DIFF(\"hours\", kibana.alert.start, NOW()) <= ?lookback_hours | EVAL minutes_active = COALESCE(kibana.alert.duration.us, 0) / 60000000.0 | STATS episodes = COUNT(*), alert_instances = COUNT_DISTINCT(kibana.alert.instance.id), flapping_episodes = SUM(CASE(kibana.alert.flapping == true, 1, 0)), median_minutes_active = MEDIAN(minutes_active) BY rule = kibana.alert.rule.name, revision = kibana.alert.rule.revision | KEEP rule, revision, episodes, alert_instances, flapping_episodes, median_minutes_active | SORT episodes DESC",
    "params": {
      "lookback_hours": {
        "type": "integer",
        "description": "How many hours of alert history to analyze, for example 6"
      }
    }
  }
}

The agent's instructions define the vocabulary (an episode is one alert document), the method (timeline first, then propagation, then scorecard), and the honesty rules (report numbers and timestamps from tool output, never invent alerts).

We ask it one question: what happened in the last three hours, was it one incident or many, which team owns the first symptom, and did the gateway rule tuning reduce noise?

The agent called all three tools and got the structure right: it separated the history into two storms with a quiet gap, one per failure wave, rather than blending them into one event.

It attributed the first symptom to team risk-ml, reproduced the upstream deduction about the queue that never alerted, and then answered the tuning question with the revision table.

Its verdict on the fix reads like a review comment backed by data: fourteen episodes at revision 0 against one stable episode at revision 1, with the median episode going from one minute to seven.

The agent is not doing anything we could not do by hand.

That is precisely the point: because alerts are durable, structured data with explicit semantics, a language model can reason over patterns and histories instead of reacting to a single notification, and every number it states is traceable to a tool call you can rerun.

Running alert history queries in production

A few things to decide before you rely on alert history in production.

Retention is a learning-window decision

Alert indices follow an ILM policy, and the default keeps far more than the few hours this lab needed, but if you want quarter-over-quarter rule quality trends, align the policy with that ambition deliberately.

Scope alert index read access narrowly

Alert documents embed rule parameters, reasons, and deep links, so treat .alerts-* read privileges like any other operational dataset and grant the narrowest index patterns that work, for example only the observability alert aliases.

Grouping keys are a data contract

Use ECS field names in BY clauses when they exist, keep cardinality bounded, and never put secrets or personal data into grouping values, because they become kibana.alert.instance.id and live for the retention window.

Project columns aggressively with KEEP

With around 1,400 mapped fields in the alert indices, an unprojected query is unpleasant for humans and actively harmful inside LLM tools, so end every reusable query with KEEP.

Alert storms also stress your automation

If a rule triggers a workflow per alert, fourteen episodes mean fourteen executions doing the same work, which is another argument for pointing automation at the aggregated alert dataset rather than at each notification.

Where to start with your own noisiest rule

The lab in this article is intentionally small, and every piece of it generalizes.

Score your noisiest real rule with the scorecard query, fix its grouping or window, and let kibana.alert.rule.revision tell you whether the fix held.

Join your alert history against whatever catalog you already have; a CMDB export in a lookup index is enough, and the propagation order becomes a query.

If you use Agent Builder, wrap your two or three most-used investigation queries as tools, and the next storm review starts with a conversation instead of a wall of notifications.

The companion notebook builds everything used here: the catalog index, the log data stream, the three rules, both incident replays, the queries, and the Agent Builder tools and agent. It runs for about 25 minutes because the rules evaluate once a minute and the episode durations are the data.

A notification tells you something broke.

The alert history tells you how your detection is performing, and it is already sitting in an index waiting to be queried.

If you run Observability on Elastic Cloud Hosted (ECH), this will feel familiar. Cross-cluster search (CCS) has long enabled teams to query across remote clusters, and it's a mainstay in large scale ECH environments. CPS takes the same idea to Serverless, and makes linking even simpler: no topology mapping, certificate configuration or per connection credentials. You link projects in the Cloud console and Elastic manages the rest.

What is cross-project search?

Serverless organizes your data into projects: Elasticsearch, Observability, or Security, each with its own scaling, permissions, and lifecycle. That isolation is a feature: it preserves data locality where needed, simplifies cost attribution, and lets teams own their own data. The tradeoff, historically, has been visibility. If your logs live in one project and your APM data lives in another, or if every business unit runs its own Observability project, getting a unified view has typically meant duplicating data into a shared store.

CPS removes that tradeoff. You designate an origin project (the project you search from) and link it to other projects you want included in that search. Once linked, an unqualified query like FROM logs* or a Discover session automatically spans the origin plus every linked project; no special syntax, no remote index prefixes. Kibana's project picker lets anyone narrow a query to just the current project or expand it to everything in scope, and administrators can set a default scope per Kibana space so a local incident-response space stays local by default while a global operations space starts broad. Access is enforced per project using your existing Elastic Cloud roles, so linking a project for search doesn't quietly grant broader permissions than a user already has.

What's the difference between cross-project search and cross-cluster search?

Cross-cluster search (ECH, self-managed)

Cross-project search (Serverless)

Where you link

Remote cluster connections, configured per cluster

Link projects in the Elastic Cloud console

Credentials

Certificates or API keys you issue and rotate

Managed by Elastic

Network setup

Network topology you map and maintain

None to configure

Query syntax

Remote cluster name referenced in the query

Unqualified query, no remote index prefix

Scope control

Set per query

Project picker, per-space defaults, and tag-based routing

Where CCS asks administrators to configure remote cluster connections, exchange certificates or API keys, and reference remote cluster names in every query, CPS asks you to link projects in the UI and query the way you already do. It's the same underlying idea: remote search without moving data, adapted to a control plane where infrastructure is meant to disappear.

Why cross-project search matters for Observability teams

Cross-project search matters most the moment your observability data stops living in one place. Observability organizations typically split their data by team, environment, region, or customer, usually for good reasons: isolating noisy neighbors, controlling blast radius, meeting data residency requirements, or giving each team ownership over its own ingest and retention. The problem shows up during an incident or a cross-cutting investigation, when someone needs to see across those boundaries without waiting for a data migration or standing up a consolidated cluster.

A few examples of where CPS changes the equation:

Keep observability data in its region and still search globally

Teams operating in multiple regions or clouds often need to keep observability data in the region it was generated, whether for regulatory reasons or to limit cross-region egress. CPS lets each region keep its own Observability project while a central project links to all of them, so an SRE team can run one query to check whether an anomaly in one region is also showing up elsewhere, without pulling all that data into one place first.

Isolate environments without losing the incident view

Splitting production, staging, and per-service or per-team projects makes scaling and cost predictable, and it limits how far a bad deploy or a runaway job can spread. It also means an on-call engineer used to have to check several places during an incident. With CPS, on-call can work from a single origin project and pull in whichever linked environments matter to the investigation, then narrow back down once things are resolved.

Central teams monitoring many business units or customers

Platform or Observability Center of Excellence (CoE) teams that operate on behalf of many business units, or Managed Service Providers (MSPs) monitoring several customer environments, can give each unit or customer its own project, with its own retention, access, and cost, while the central team links across all of them for cross-cutting reporting and anomaly detection. Each project owner keeps control of their own data; the center gets the aggregate view it needs.

Avoid ingesting the same data twice

When two teams need the same data for different purposes, the old options were duplicate ingestion pipelines or a shared, less isolated deployment. With linked projects, data is ingested once and queried from wherever it's needed, which keeps both storage costs and pipeline maintenance down.

This applies across solution types too: CPS links Elasticsearch, Observability, and Security projects interchangeably, so a Security team and an Observability team can each keep their own project, with their own retention and access controls, while querying the same underlying logs. A security analyst investigating an alert can query infrastructure or service context from a linked Observability project in Discover, dashboards, or ES|QL, and an SRE debugging an incident can query related signals from a linked Security project the same way, all without either team re-ingesting the other's data.

What does cross-project search support in Elastic Observability?

At GA, CPS is available across Elasticsearch, Observability, and Security project types (Observability and Security projects require the Complete tier). For Elastic Observability specifically, application and infrastructure monitoring, alerting rules, SLOs, machine learning anomaly detection (including APM anomaly detection), and Synthetics work across linked projects, alongside the platform-level capabilities every project gets: Discover, dashboards, and ES|QL. Project routing lets you target a query to a subset of linked projects using predefined tags (alias, cloud provider, region) or your own custom tags, combined with boolean logic, so "all EU projects" or "every environment except staging" can be expressed once and reused.

This release also raises the default number of projects you can link to a single origin, well beyond the tech preview limit, with room to go higher for larger deployments. Any existing project can now act as an origin rather than only newly created ones. For programmatic access, you only need a Cloud API key to call CPS from your own tooling. A new dedicated cloud role can now grant those keys for programmatic access.

A few nuances are worth knowing before you design around CPS:

• Alerting: Alerting rules defined in the origin project can evaluate data across linked projects. The alerts they generate stay local to the project where the alerting rule lives.
• SLOs: SLOs don't follow the project picker in the Kibana header. Instead, you set scope in the SLO create and edit form, choosing this project, all linked projects, or a specific subset, since changing scope rebuilds the SLO's underlying computation. You can't view SLOs created in linked projects from the current project, but you can create an SLO here based on data that lives in those projects.
• Synthetics: Overview, monitor details, certificates, errors, and journeys honor the Kibana header project picker and show linked-project monitors as read-only. Creating, editing, and managing monitors stay in the origin project, and Synthetics status and TLS rules still evaluate origin monitors only.
• Streams: Not CPS-aware yet; Streams stay scoped to the project where they were created.

How do you set up cross-project search?

Setting up cross-project search takes a few minutes:

1. From your origin project's settings in the Elastic Cloud console, open Link projects.
2. Select the projects you want to link, individually or in bulk.
3. Save.
4. Open Discover, run an ES|QL query, or build a dashboard as you normally would: linked project data is included automatically, and the project picker lets you narrow or widen the scope at any time.

For the full configuration walkthrough, including linking topologies (hub-and-spoke, shared data project, or full mesh) and how to scope project routing with tags and expressions, see the cross-project search documentation and the Observability-specific guide.

Cross-project search pricing

CPS pricing has two components: a charge for data retained in each linked project you query, and a charge for data egress generated by cross-project query traffic, tracked as its own line item so you can see what CPS is adding to your transfer costs. There's no separate per-link fee. Because actual cost depends on your data volumes, retention settings, and how many projects you link, use the Elastic Cloud Serverless pricing estimator to model costs for your own environment against your own ingest and retention numbers.

Start using cross-project search

If you're already running Elastic Observability on Serverless, cross-project search is available in your project settings today: link your projects and run your first cross-project query in Discover. If you haven't tried Elastic Cloud Serverless yet, start a free trial and see how far a single query can reach.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic, Elasticsearch, and associated marks are trademarks, logos, or registered trademarks of Elasticsearch B.V. in the United States and other countries. All other company and product names are trademarks, logos, or registered trademarks of their respective owners.

Prerequisites for Elastic integration alerting rule templates

Elastic Stack 9.4.0 or later.

Alerting rule templates have been available since 9.2.1, under the integration Assets tab. This article covers three things that need 9.4.0: the dedicated Alerting tab, idle data streams rules, and the NGINX OpenTelemetry Assets package, which is in technical preview.

Step 1: Send NGINX logs and metrics to Elasticsearch with OpenTelemetry

First, get NGINX metrics and logs into Elasticsearch.

Enable the NGINX stub_status module and make the access and error logs readable by the collector. Then, configure an EDOT or upstream OpenTelemetry Collector with the nginx and filelog receivers to export metrics and logs to Elasticsearch.

The integration setup has the full receiver and pipeline configuration.

If you want to reproduce this example, you can use the companion repository.

Both signals matter for alerting, and each group of templates reads a different data stream:

• The log-based templates (4xx and 5xx error rates, error log spike) query logs-nginx.access.otel-* and logs-nginx.error.otel-*, which come from the filelog receiver.
• The metric-based templates (active connections, dropped connections) query metrics-nginxreceiver.otel-*, which comes from the nginx receiver.

This is easy to get wrong: the Fleet Nginx (OpenTelemetry) input package collects stub_status metrics only. Its companion for logs is the classic Nginx integration, which writes ECS-based nginx.access and nginx.error datasets, not the *.otel-* data streams the log-based templates query. If you rely on that pairing alone, the log-based rules have nothing to evaluate and silently never fire. Run the filelog receiver too, not just the nginx receiver.

Step 2: Install the NGINX OpenTelemetry Assets integration

NGINX OpenTelemetry Assets is a content-only package. It ships the dashboards, alerting rule templates, and SLO templates, but it does not collect data itself. The data comes from the collector you set up in Step 1.

You don't need to install it by hand. Once the NGINX OTel data from Step 1 starts arriving, Elastic detects it and installs the Assets package for you, which takes a minute or two. Confirm it under Management > Integrations > Installed integrations, where NGINX OpenTelemetry Assets should appear.

Step 3: Create Elasticsearch alert rules from a rule template

Open the integration and select the Alerting tab.

This package ships six templates: high 4xx and 5xx error rates, high active connections, an error log spike, dropped connections, and a generic High error rate by service template that points at a placeholder logs-myservicereceiver.otel-* index for you to repoint and rename. The five NGINX rules run ES|QL every minute and group results by host.name, so an alert points at the host with the problem. The generic template groups by service.name instead, since it is meant to be repointed at whichever service you choose.

Select a template, for example [Nginx OTel] High 5xx error rate. Kibana opens a prefilled Create rule form built on an Elasticsearch query rule. It runs the template's ES|QL on a schedule and alerts when the query returns rows.

The query looks like this:

FROM logs-nginx.access.otel-*
// Flag each access log entry as a server error (5xx) or not
| EVAL is_5xx = CASE(http.response.status_code >= 500, 1, 0)
// Aggregate total requests and 5xx count per NGINX host
| STATS total = COUNT(*), errors_5xx = SUM(is_5xx) BY host.name
// Minimum sample size to avoid noisy low-traffic hosts
| WHERE total > 50
// Calculate 5xx error rate as a percentage
| EVAL error_rate_pct = ROUND(TO_DOUBLE(errors_5xx) / TO_DOUBLE(total) * 100.0, 2)
// Alert threshold: adjust to tune sensitivity
| WHERE error_rate_pct > 5.0
| SORT error_rate_pct DESC
| LIMIT 10

It counts requests and 5xx responses per host, keeps hosts with enough traffic to matter, and returns those above five percent.

Three things to get right while the form is open:

• Send data first. ES|QL validates column names against the indices that exist when the query runs. Open a template before any NGINX data has been ingested and the editor reports Unknown column "http.response.status_code" and the form shows errors. Once data is flowing (Step 1), the same query validates and the error clears, so collect data before you create the rule.
• Set the time field to @timestamp.
• Leave "Create an alert for each row" selected. Because the query groups by host.name, this makes every affected host raise its own alert.

Add a connector and an action so the alert reaches Slack, email, or PagerDuty, then save and enable the rule.

Step 4: Tune alerting rule template thresholds in ES|QL

The thresholds are starting points, so confirm them against your own traffic. The threshold lives in the ES|QL WHERE clause.

To make the rule stricter, change error_rate_pct > 5.0 to error_rate_pct > 2.0. To require more traffic before it fires, raise total > 50. Use Test query in the rule form to confirm the edited query parses and returns rows before you save.

Three more settings are worth a look:

• Time window: the look-back period the query runs over. A shorter window reacts faster but is noisier on bursty traffic.
• Rule schedule: how often the query runs, every minute by default.
• Alert delay: the number of consecutive runs the condition must hold before an alert is created, which filters out single-run blips.

How do you detect idle data streams in Elasticsearch?

Threshold rules only fire while data keeps arriving. When an agent goes offline or an output breaks, the data stops, and a threshold rule has nothing to evaluate.

Many Elastic integrations include a dynamically generated idle data streams template for exactly this case. It is named [{Integration name}] Idle data streams and appears in the same Alerting tab, though it is generated automatically rather than bundled with the integration. It alerts when no data is written to any of the integration's data stream patterns within a set period.

The NGINX OpenTelemetry packages do not include an idle data streams template, which is why no such template appears in the Alerting tab from Step 3. The end of this section covers what to do instead.

The default period is 24 hours, which is usually too long. A production service can go quiet for most of a day before you hear about it.

When you create the rule, drop the period to match how fast you need to know. Fifteen minutes to one hour works for a critical service. For a batch job, set a period comfortably longer than its run interval, so the quiet gaps between runs do not trigger it.

So why is this example left out? The template is generated from the data stream patterns an integration defines, and it is not generated for input-only packages. A content-only package like NGINX OpenTelemetry Assets defines no data streams of its own either. To catch silence in a collector-based setup like this, recreate the rule by hand with an Elasticsearch query rule. Use the query DSL or KQL variant rather than ES|QL, because an ES|QL rule fires on returned rows and so cannot alert on the absence of data. Point it at the OTel data streams (logs-nginx.access.otel-*, or metrics-nginxreceiver.otel-*) and set the condition to fire when the number of matching documents is below 1 over a window of, say, the last 15 minutes. That reproduces what an idle data streams template does, scoped to the data streams your collector writes.

Get started with Elastic integration alerting rule templates

Alerting rule templates turn alert setup into a few steps: send data, install the integration, create a rule from a template, and adjust the threshold. Treat the bundled thresholds as defaults to confirm, not numbers to trust blindly. And where an idle data streams template is available, reduce its 24-hour default so you find out quickly when a service goes silent.

Resources

• Companion repository, to generate the NGINX demo data used here
• Alerting rule templates
• NGINX OpenTelemetry Assets integration
• End-to-end tracing for NGINX with OpenTelemetry
• Elasticsearch query rule
• Elastic Agent built-in alerts, for monitoring the agents themselves

This walkthrough shows the full path using the OpenTelemetry Demo (Astronomy Shop), where a failing shipping dependency drives every checkout SLI miss. You can reproduce it by running the demo against Elastic Observability, defining an APM availability SLO on checkout, and introducing a failing dependency.

Availability

SLO burn rate analysis is available in Elastic Observability serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

Prerequisites for SLO burn rate alerting in Elastic Observability

You need an instrumented service and an SLO defined on its APM data in Elastic Observability.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages
• Elastic Distributions of OpenTelemetry (EDOT) language SDKs
• OpenTelemetry SDKs sending OTLP via the EDOT Collector, Elastic Agent, APM Server, or the Managed OTLP endpoint. If you run a custom upstream Collector pipeline, include both the elasticapm connector and the elasticapm processor. Those components ship with the EDOT Collector (or a custom EDOT-like build); they are not part of the standard OpenTelemetry Collector Contrib distribution. For wiring details, see the upstream collector setup.
• SLO definition: an SLO based on APM latency or APM availability for the service you want to protect. Elastic creates a default burn rate alert rule when you save the SLO.
• Backend: Elastic Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

SLO burn rate analysis walkthrough: alert to failing dependency

Step 1: Review SLO burn rate on the alert detail page

Open the SLO burn rate alert detail page from the notification.

The chart shows when burn rate rose and how much error budget remains for the rolling period. From there, open the SLI linked to the alert and decide whether you are looking at a short spike or sustained degradation before you dig into events.

In this example, the availability SLO for checkout is burning budget quickly, and the rise is recent, so the change likely landed in the last few hours rather than as long-running drift:

Step 2: Check the SLI error rate driving SLO error budget consumption

On the SLI view, check the error rate that feeds the SLO.

You can look at the SLI as RED metrics in the APM UI for a quick service-level picture, or open Traces in Discover when you want to filter, compare, or break down the spans that count toward the SLO.

Here, error rate on checkout has climbed, which matches the jump in availability SLO burn rate:

Step 3: Compare good and bad event spans for the SLI

Open the events for the SLI and switch between spans for good events and bad events.

Differences between those sets usually show up before you open individual traces. In this case, bad events share a pattern that good events do not, which narrows where to look next:

Step 4: Find the failing span in the trace waterfall

Open a few sample spans for bad events and view each trace in the waterfall.

That puts the failing span in the request path so you can see which hop is causing the SLI miss. In this example, the failing span is a downstream call, not application logic inside checkout itself:

Step 5: Verify which dependency is burning your SLO error budget

Open the service that owns the bad span and check its dependencies.

Here the waterfall points to shipping and a failing call to quote-old. Requests to that dependency fail on every attempt, so the SLO burn is driven by an external dependency rather than checkout code:

SLO burn rate investigation: from alert to root cause

From an SLO burn rate alert in Elastic Observability, you can see when burn rate rose, open the linked SLI, compare good and bad event spans, inspect bad spans in the trace waterfall, and verify failing dependencies to find what is burning error budget.

One of the main components of Elastic's Synthetic Monitoring is the ability to create user journeys, which can be done with or without code. There is a Synthetics agent,, a CLI tool that guides you through the process of creating both heartbeat monitors and user journeys and deploying your code to Elastic Observability. If you are using code to create user journeys, you are using Playwright under the hood with some additional configuration to make it easier to work with Elastic Observability.

To automatically create user journeys using TypeScript, you can create Playwright tests based on a prompt using Warp, an AI-assisted terminal, Gemini 2.5 Pro, and MCP. This application was built using Python and FastMCP, which wraps the synthetic agent to deploy browser tests to Elastic automatically. This blog post will guide you through how the application works, how to use it, and its development process. You can find the complete code on GitHub.

Solution overview

Currently, this solution is set up to run inside Warp as an MCP server; however, you can also use another client, such as Claude Desktop or Cursor. From there, you create a Python script using FastMCP, which allows you to create functions that are callable by an LLM. Within Warp, you can make a configuration file in JSON that enables you to point to your Python script and pass in all the environment variables you are working with. From there, you'll want to toggle agent mode and ask a question about creating synthetic testing or call the MCP function directly. There are many options for which LLM you can select, be sure to check out Warp's documentation to learn more about the options available.

After that, you should ask a question about creating synthetic testing or call the MCP function you are looking for. The following three functions can be used:

• diagnose_warp_mcp_config Used for debugging environment variable issues that may arise. This function likely won't be needed unless there is an issue with your configuration.

• create_and_deploy_browser_test Will automatically create Playwright tests if given the test name, the URL you want to test, and a schedule. This approach uses a template-based method, rather than a machine learning-based method, and all the tests it outputs will appear similar.

• llm_create_and_deploy_test_from_prompt Similar to create_and_deploy_browser_test, but the main difference is that it uses an LLM to create tests based on a prompt you give it. The tests should reflect the prompt you provided. To run this function you'll provide a test name, URL, prompt, and schedule.

Why create this solution as an MCP server?

The reason this was developed as an MCP server, as opposed to just a standalone script or a standard CLI, is that it can be structured and interacted with in a more conversational manner. It enables an LLM to generate dynamic Playwright testing while maintaining consistent arguments, environment variables, and responses to ensure accuracy and reliability. Thus, it becomes a reliable workflow that other agents or developers can compose with additional tools. In other words, the MCP layer turns your LLM-based test authoring into a standardized, reusable capability instead of a one-off script. To learn more about the direction of MCP, be sure to check out our article on the topic.

Implementation considerations

When creating a solution like this one, one thing to be mindful of is your use of tokens. An early version of this solution took approximately twenty minutes to create synthetic tests and ultimately led to severe rate-limiting.

Another issue faced during the building process was striking a balance between creating a template that facilitates the creation of a Playwright script and having an LLM create Playwright scripts based on prompts that didn't feel cookie-cutter. While using a more LLM approach an issue faced was that the scripts often didn't work or were based on parameters that didn't exist and a more templated approach was more reliable but felt repetitive. The final version of this solution attempted to balance this by using elements of the template while adjusting the LLM parameter of temperature, which controls the randomness or creativity of a large language model's output.

While testing this solution, a failing test also emerged that required navigating past a pop-up. In more complex cases, this may serve as a building block that requires additional domain knowledge to create a complete passing Playwright test.

How to get started

Prerequisites

• The version of Python that is used is Python 3.12.1 but you can use any version of Python higher than 3.10.
• This application uses Elastic Observability version 9.1.2, but you can use any version of Elastics Observability that is higher than 8.10. You can also use Elastic Cloud Serverless as well.
• You will also need an OpenAI API key to use the LLM capabilities of this application. You will want to configure an environment variable for your OpenAI API Key, which you can find on the API keys page in OpenAI's developer portal.

Step 1: Install the packages and clone the repository

In order for this MCP server to run locally you will need to install the the following packages:

pip install fastmcp openai
npm install -g playwright @elastic/synthetics

You will use FastMCP 2.0 to create the MCP server, and OpenAI to generate tests based on prompts that you provide. Additionally, you will want to clone the repository to obtain a local copy of the server.

Step 2: Set up a configuration file in Warp

Inside of Warp, you will want to go to the side panel, where it says MCP servers and where it says “add”.

After that, you will be prompted to add a JSON configuration file that should resemble the following. Be sure to add your own Kibana URL, update the correct path, and include your own keys and tokens.

{
 "elastic-synthetics": {
   "command": "python",
   "args": ["elastic_synthetics_server.py"],
   "env": {
     "PYTHONPATH": ".",
     "ELASTIC_KIBANA_URL": "https://your-kibana-url.elastic-cloud.com",
     "ELASTIC_API_KEY": "your-api-key-here",
     "ELASTIC_PROJECT_ID": "mcp-synthetics-demo",
     "ELASTIC_SPACE": "default",
     "ELASTIC_AUTO_PUSH": "true",
     "ELASTIC_USE_JAVASCRIPT": "false",
     "ELASTIC_INSTALL_DEPENDENCIES": "true",
     "OPENAI_API_KEY": "sk-your-openai-key",
     "LLM_MODEL": "gpt-4o"
   },
   "working_directory": "/path/to/your/file",
   "start_on_launch": true 
   }
}

Step 3: Ask a question or call the tools directly

Now that you've set up locally, you will want to toggle agent mode and select the LLM you wish to use. The reason why Gemini-Pro-2.5 was chosen for this blog post is that it provides a straightforward answer, while other LLMs selected returned a very lengthy response.

To start using the MCP tools, from your MCP server, you can ask a question that contains the test name, URL, prompt, and schedule.

You can also call the directly by typing llm_create_and_deploy_test_from_prompt() and the program will prompt you for the relevant details:

Inside Kibana, you should see your monitor listed if you click under Applications and select Monitors listed under Synthetics. You can also find a link to your monitor in the response of your MCP tool.

What's Going On Inside

This code sample consists of three primary functions, which are MCP tools that you can call from your MCP client, including diagnose_warp_mcp_config, create_and_deploy_browser_test and llm_create_and_deploy_test_from_prompt.

Debugging environment issues

There were various issues that came up while creating this application around environment variable loading, so there was a need to create an MCP that could be called depending on errors that may be present.

The tool diagnose_warp_mcp_config kicks off with a decorator @mcp.tool() which allows it to be called and listed in the list of available tools. This tool is designed to help debug issues with Elastic-specific environment variables for troubleshooting purposes. First, it loads in the environment variables and looks for the Elastic specific variables, after it does some security masking so it doesn't show any variables and hides sensitive information like API keys in the output, showing only the first eight characters followed by "…". This tool determines if the minimum required credentials (Kibana URL and API Key) are present to proceed with deployment and provides a report letting you know to address any issues that may exist.

@mcp.tool()
def diagnose_warp_mcp_config() -> Dict[str, Any]:
   """Diagnose Warp MCP environment configuration for Elastic Synthetics"""
   try:
       env_vars = load_env_from_warp_mcp()

       # Check for required variables
       kibana_url = env_vars.get('ELASTIC_KIBANA_URL') or env_vars.get('KIBANA_URL')
       api_key = env_vars.get('ELASTIC_API_KEY') or env_vars.get('API_KEY')
       project_id = env_vars.get('ELASTIC_PROJECT_ID') or env_vars.get('PROJECT_ID')
       space = env_vars.get('ELASTIC_SPACE') or env_vars.get('SPACE', 'default')

       # Mask sensitive values for display
       masked_vars = {}
       for key, value in env_vars.items():
           if 'API_KEY' in key or 'TOKEN' in key:
               masked_vars[key] = f"{value[:8]}..." if value and len(value) > 8 else "***"
           else:
               masked_vars[key] = value

       deployment_ready = bool(kibana_url and api_key)

       return safe_json_response({
           "status": "success",
           "environment_variables": masked_vars,
           "required_check": {
               "kibana_url": bool(kibana_url),
               "api_key": bool(api_key),
               "project_id": bool(project_id),
               "space": bool(space)
           },
           "deployment_ready": deployment_ready,
           "recommendations": [
               "Environment variables detected" if env_vars else "No environment variables found",
               "Kibana URL configured" if kibana_url else "Missing ELASTIC_KIBANA_URL or KIBANA_URL",
               "API Key configured" if api_key else "Missing ELASTIC_API_KEY or API_KEY",
               "Ready for deployment" if deployment_ready else "Missing required credentials"
           ]
       })

   except Exception as e:
       return safe_json_response({
           "status": "error",
           "error": str(e),
           "error_type": type(e).__name__
       })

Creating synthetic tests based on a template

While developing this solution to generate tests based on a prompt, the process wasn't always smooth. Early versions encountered issues with accuracy, hallucinations, and the creation of loops. To make progress, a version that relied on creating a test template to verify the mechanics of the solution, such as whether the test could pass and be deployed to Elastic correctly, was a logical next step.

This solution automates the entire process of creating a synthetic browser test that will regularly check if a website is working correctly, then deploys it to Elastic Observability Synthetics. Similar to diagnose_warp_mcp_config, the MCP tool create_and_deploy_browser_test starts with the decorator @mcp.tool() and checks to make sure that the proper environment variables are loaded.

From there, it creates a TypeScript test file that is based on templates and generates dynamic test steps based on the target website's characteristics, including navigating to the website, verifying the page title exists, checking page load performance, taking a screenshot, verifying page content is visible, and finally saves the test file in a synthetic_tests directory.

Finally, it wraps Elastic's CLI tool @elastic/synthetics to push the test to Kibana, allowing you to set which geographic locations to run tests from, how often to run the test, and the project and workspace settings.

You check out the full code for this MCP tool here.

Creating synthetic tests based on a prompt

While creating browser tests based on a templated approach is a good starting point, it felt generic and cookie-cutter. But it made a helpful structure to build an LLM-based function on top of.

The MCP tool llm_create_and_deploy_test_from_prompt begins by ensuring that basic parameters, including locations, schedule, and directories, are listed. Additionally, it aims to learn more about the target website to inform the AI and initialize the OpenAI client and model, which is GPT-4o.

After setting up the LLM, it converts natural language requests into actual Playwright test code, then cleans and validates the AI-generated code to prevent issues like injection attacks or malformed syntax. It draws inspiration from the templated approach, wrapping AI-generated steps within a proven, reliable test framework template. Finally, it deploys the test to Elastic in a similar manner to the previous tool.

You can find the code for this tool here.

Conclusion and next steps

Synthetic monitoring in Elastic Observability makes it easy to test complete user journeys and keep your site reliable, with simple setup and a Playwright integration. A tool like this can provide a starting point for tests that you can iterate on after.

A solution like this is just the start of an MCP implementation that automatically generates Playwright tests for you and can be expanded in the future to include heartbeat monitors, utilize the Playwright MCP server, or consider experimenting with Claude for Chrome to create synthetic testing.

Check out more articles on Observability Labs on Infrastructure Monitoring

Many users may have a large number of TCP, ICMP, and HTTP monitors and need to migrate them to Synthetics. In this guide, we will explain how to perform this migration easily while ensuring that it will be future-proof or able to develop more advanced checks such as Browser monitors.

First, we must consider the number of monitors to migrate; if the number is small, the easiest way would be to do it manually through the Synthetics UI. However, in this guide we will assume that we have dozens or hundreds of monitors to migrate, and doing it manually in the Synthetics UI is not an option.

Private Location

Traditionally, uptime monitors required a Heartbeat to be deployed in your infrastructure, which indirectly allowed you to monitor endpoints or hosts on your private network. If this is still a requirement, you will need to either configure Private Location or allow Elastic’s global managed infrastructure to access your private endpoints (only on ECH & Serverless).

In this guide, we will use Private Locations, which will allow you to monitor both internal and external resources. More details can be found here: Monitor resources on private networks

Step 1: Set up Fleet Server and Elastic Agent

Private Locations are simply Elastic Agents enrolled in Fleet and managed through an agent policy. 

If you don't have a Fleet Server yet, start setting up a Fleet Server. This step is not necessary if you use ECH, as it comes by default.

Next, you will need to create an Agent Policy. Go to Observability → Monitors (Synthetics) → Settings (top right) → Private Location → + Create Location

Fill in the fields and create a new policy for this Private Location. It is important to know that Private Location should be set up against an agent policy that runs on a single Elastic Agent. 

Step 2: Deploy the Elastic Agent

Now we need to deploy the Elastic Agent that will be responsible for running all the monitors. We can use the same host we were using for Heartbeat. There is only one requirement: we must be able to run Docker containers, since to take advantage of all the features of Synthetics, we must use the elastic-agent-complete Docker Image.

1. Go to Fleet –> Enrollment tokens and note the enrollment token relevant to the policy you just created for the Private Location. Now go to Settings and note the default Fleet server host URL.

2. On the host, run the following commands. For more information on running Elastic Agent with Docker, refer to Run Elastic Agent in a container.

docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL={fleet_server_host_url} \
  --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:9.3.2

Synthetic Project

At this point, we already have the location from which our Synthetic monitors will run. Now we need to load our Uptime monitors as Synthetics.

As we mentioned earlier, there are two ways to do this: either manually through the Synthetics UI or through a Synthetics Project. In our case, since we have so many monitors to migrate and don't want to do it manually, we will use Synthetics Projects. 

The great thing about Synthetics Project is that it has some backward compatibility with the definition of monitors in heartbeat.yml and we will be leveraging it.

What's Synthetics project?

Synthetics project is the most powerful and flexible way to manage synthetic monitors in Elastic, based on the Infrastructure as Code principle and compatible with Git-Ops flows. Instead of configuring monitors from the interface, you define them as code: .yml files for lightweight monitors and JavaScript or TypeScript scripts for browser-type monitors (journeys).

This approach allows you to structure your monitors in a repository, version them with Git, validate them, and deploy them automatically using CI/CD flows, providing traceability, reviews, and consistent deployments.

Step 3: Initialize your Synthetics project

You will no longer need to connect to the hosts where you deployed the Elastic Agent, as the remaining steps can be performed locally as long as you have connectivity to Kibana!

Since Synthetics Projects is based on Node.js, make sure you have it installed. 

1. Install the package:

npm install -g @elastic/synthetics

1. Confirm your system is setup correctly:

npx @elastic/synthetics -h

1. Start by creating your first Synthetics project. Run the command below to create a new Synthetics project named synthetic-project-test in the current directory.

npx @elastic/synthetics init synthetic-project-test

1. Follow the prompt instructions to configure the default variables for your Synthetics project. Make sure to at least select your Private Location. Once that’s done, set the SYNTHETICS_API_KEY environment variable in your terminal, which allows the project to authenticate with Kibana.

2. To generate an API key go to Synthetics Kibana.

3. Click Settings.

4. Switch to the Project API Keys tab.

5. Click Generate Project API key.

More details for all the steps can be found here: Create monitors with a Synthetics project

Step 4: Add your heartbeat.yml files

Once the project is initialized, access the folder it has created and take a look at the project structure:

• journeys is where you’ll add .ts and .js files defining your browser monitors. It currently contains files defining sample monitors.

• lightweight is where we’ll add our heartbeat.yml files defining our lightweight monitors. It currently contains a file defining sample monitors.

Therefore, all we have to do is copy our heartbeat.yml files to this lightweight folder. Before copying heartbeat.yml, keep in mind that we don't need all the content, we are only interested in the heartbeat.monitors part. \ We recommend considering splitting the file into logical groups. Instead of maintaining a single large YAML file, you could create multiple smaller YAML files, with each file representing either a single check or a group of related checks. This approach may simplify management and improve compatibility with GitOps workflows.\ Each YAML file should look like this:

carles@synthetics-migration:synthetic-project-test/lightweight# cat heartbeat.yml

heartbeat.monitors:
- type: icmp
  schedule: '@every 10s'
  hosts: ["localhost"]
  id: my-icmp-service-synth
  name: My ICMP Service - Synthetic
- type: tcp
  schedule: '@every 10s'
  hosts: ["myremotehost:8123"]
  mode: any
  id: my-tcp-service-synth
  name: My TCP Service Synthetic
- type: http
  schedule: '@every 10s'
  urls: ["http://elastic.co"]
  id: my-http-service-synth
  name: My HTTP Service Synthetic

What we just did is define different ICMP, TCP, and HTTP checks as code.

Now we need to ask Synthetics project to create the monitors in Kibana based on what we have defined in our YAML files:

npx @elastic/synthetics push --auth $SYNTHETICS_API_KEY --url <kibana-url>

Unfortunately, we do not support a 1-to-1 mapping of the heartbeat schema to the lightweight schema, so you may encounter some errors during the execution of this command. One example is the definition of schedule. Heartbeat supports the use of crontab expressions, but Project requires the use of @every syntax.

If no syntax errors were found, the command output will show that the monitors have been successfully created in Kibana!

Then, go to Synthetics in Kibana. You should see your newly pushed monitors running. You can also go to the Management tab to see the monitors' configuration settings.

Here's what's new:

Improved Related Alert Grouping with Relevance Scoring & Reasoning

We've enhanced our related alert detection to go beyond surface-level correlations. Alerts are now grouped based on a relevance score that reflects the strength of their relationship across dimensions like:

• Shared entities or resources (e.g. same host, pod, or service)
• Temporal proximity (alerts firing within a suspiciously short window)
• Signal similarity (e.g. spikes in logs, metrics, and traces that point to the same failure mode)

More importantly, we now show the why. You'll see why an alert is grouped, whether it's sharing the same Kubernetes pod, has similar log patterns, or was triggered by the same upstream anomaly. This gives users confidence in the grouping logic and accelerates root cause analysis.

Link Dashboards to Alert Rules and Get Smart Suggestions

You can now link dashboards directly to your alert rules, giving responders an instant visual lens into the metrics or logs that matter most for that alert. No more scrambling to remember which dashboard to check — just click and go.

And we've made this smarter too: Elastic will now suggest relevant dashboards based on the alert's source, rule logic, or monitored entities, helping users land on the right view without needing to configure anything upfront.

Investigation Guides Embedded Into Alerts

Every alert can now be configured with an investigation guide, a set of pre-configured, context-aware instructions or next steps tailored to the alert. Think of it as a playbook that's embedded right where and when you need it.

Use it to:

• Document your team's runbooks and standard triage steps or link to existing runbooks
• Guide junior engineers or on-call responders through unfamiliar territory
• Automate the first few steps of root cause analysis

Why This Matters

These changes are all about reducing time to detect (MTTD) and time to resolve (MTTR). By:

• Grouping alerts more intelligently (and transparently)
• Giving you the dashboards you need, when you need them
• Embedding action-oriented guides in every alert

We're bringing you closer to a truly streamlined incident response workflow; No swivel-chairing, no guesswork, just clarity.

Additionally, look at some of our other articles on Elastic Observability Labs related to analysis:

• Using the AI Assistant in Elastic Observability to Accelerate Root Cause Analysis

• All of the log analytics features in Elastic Observability

• Our latest on OpenTelemetry support in Elastic Observability

Site reliability engineers (SREs) face the challenging task of sifting through vast quantities of data, not just from the applications themselves but also from the underlying infrastructure. In addition to data analysis, they are responsible for ensuring the effective use and development of operational tools. The growing volume of data, the daily resolution of issues, and the continuous evolution of tools and processes can detract from the focus on business performance.

Elastic Observability offers a solution to this challenge. It enables SREs to integrate and examine all telemetry data (logs, metrics, traces, and profiling) in conjunction with business metrics. This comprehensive approach to data analysis fosters operational excellence, boosts productivity, and yields critical insights, all of which are integral to maintaining high-performing applications in a demanding digital environment.

To help manage operations and business metrics, Elastic Observability's SLO (Service Level Objectives) feature was introduced in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

Users can monitor SLOs in real-time with dashboards, track historical performance, and receive alerts for potential issues. Additionally, SLO dashboard panels offer customized visualizations.

Service Level Objectives (SLOs) are generally available for our Platinum and Enterprise subscription customers.

In this blog, we will outline the following:

• What are SLOs? A Google SRE perspective

• Several scenarios of defining and managing SLOs

Service Level Objective overview

Service Level Objectives (SLOs) are a crucial component for Site Reliability Engineering (SRE), as detailed in Google's SRE Handbook. They provide a framework for quantifying and managing the reliability of a service. The key elements of SLOs include:

• Service Level Indicators (SLIs): These are carefully selected metrics, such as uptime, latency, throughput, error rates, or other important metrics, that represent the aspects of the service and are important from an operations or business perspective. Hence, an SLI is a measure of the service level provided (latency, uptime, etc.), and it is defined as a ratio of good over total events, with a range between 0% and 100%.

• Service Level Objective (SLO): An SLO is the target value for a service level measured as a percentage by an SLI. Above the threshold, the service is compliant. As an example, if we want to use service availability as an SLI, with the number of successful responses at 99.9%, then any time the number of failed responses is > .1%, the SLO will be out of compliance.

• Error budget: This represents the threshold of acceptable errors, balancing the need for reliability with practical limits. It is defined as 100% minus the SLO quantity of errors that is tolerated.

• Burn rate: This concept relates to how quickly the service is consuming its error budget, which is the acceptable threshold for unreliability agreed upon by the service providers and its users.

Understanding these concepts and effectively implementing them is essential for maintaining a balance between innovation and reliability in service delivery. For more detailed information, you can refer to Google's SRE Handbook.

One main thing to remember is that SLO monitoring is not incident monitoring. SLO monitoring is a proactive, strategic approach designed to ensure that services meet established performance standards and user expectations. It involves tracking Service Level Objectives, error budgets, and the overall reliability of a service over time. This predictive method helps in preventing issues that could impact users and aligns service performance with business objectives.

In contrast, incident monitoring is a reactive process focused on detecting, responding to, and mitigating service incidents as they occur. It aims to address unexpected disruptions or failures in real time, minimizing downtime and impact on service. This includes monitoring system health, errors, and response times during incidents, with a focus on rapid response to minimize disruption and preserve the service's reputation.

Elastic®’s SLO capability is based directly off the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on an SLI such as KQL (log based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric. Additionally, set the appropriate threshold.

• Utilize occurrence versus time slice based budgeting. Occurrences is the number of good events over the number of total events to compute the SLO. Timeslices break the overall time window into slammer slices of a defined duration and compute the number of good slices over the total slices to compute the SLO. Timeslice targets are more accurate and useful when calculating things like a service’s SLO when trying to meet agreed upon customer targets.

• Manage all the SLOs in a singular location.

• Trigger alerts from the defined SLO, whether the SLI is off, burn rate is used up, or the error rate is X.

• Create unique service level dashboards with SLO information for a more comprehensive view of the service.

SREs need to be able to manage business metrics.

SLOs based on logs: NGINX availability

Defining SLOs does not always mean metrics need to be used. Logs are a rich form of information, even when they have metrics embedded in them. Hence it’s useful to understand your business and operations status based on logs.

Elastic allows you to create an SLO based on specific fields in the log message, which don’t have to be metrics. A simple example is a simple multi-tier app that has a web server layer (nginx), a processing layer, and a database layer.

Let’s say that your processing layer is managing a significant number of requests. You want to ensure that the service is up properly. The best way is to ensure that all http.response.status_code are less than 500. Anything less ensures the service is up and any errors (like 404) are all user or client errors versus server errors.

If we use Discover in Elastic, we see that there are close to 2M log messages over a seven-day time frame.

Additionally, the number of messages with http.response.status_code > 500 is minimal, like 17K.

Rather than creating an alert, we can create an SLO with this query:

We chose to use occurrences as the budgeting method to keep things simple.

Once defined, we can see how well our SLO is performing over a seven-day time frame. We can see not only the SLO, but also the burn rate, the historical SLI, and error budget, and any specific alerts against the SLO.

Not only do we get information about the violation, but we also get:

• Historical SLI (7 days)

• Error budget burn down

• Good vs. bad events (24 hours)

We can see how we’ve easily burned through our error budget.

Hence something must be going on with nginx. To investigate, all we need to do is utilize the AI Assistant, and use its natural language interface to ask questions to help analyze the situation.

Let’s use Elastic’s AI Assistant to analyze the breakdown of http.response.status_code across all the logs from the past seven days. This helps us understand how many 50X errors we are getting.

As we can see, the number of 502s is minimal compared to the number of overall messages, but it is affecting our SLO.

However, it seems like Nginx is having an issue. In order to reduce the issue, we also ask the AI Assistant how to work on this error. Specifically, we ask if there is an internal runbook the SRE team has created.

AI Assistant gets a runbook the team has added to its knowledge base. I can now analyze and try to resolve or reduce the issue with nginx.

While this is a simple example, there are an endless number of possibilities that can be defined based on KQL. Some other simple examples:

• 99% of requests occur under 200ms

• 99% of log message are not errors

Application SLOs: OpenTelemetry demo cartservice

A common application developers and SREs use to learn about OpenTelemetry and test out Observability features is the OpenTelemetry demo.

This demo has feature flags to simulate issues. With Elastic’s alerting and SLO capability, you can also determine how well the entire application is performing and how well your customer experience is holding up when these feature flags are used.

Elastic supports OpenTelemetry by taking OTLP directly with no need for an Elastic specific agent. You can send in OpenTelemetry data directly from the application (through OTel libraries) and through the collector.

We’ve brought up the OpenTelemetry demo on a K8S cluster (AWS EKS) and turned on the cartservice feature flag. This inserts errors into the cartservice. We’ve also created two SLOs to monitor the cartservice’s availability and latency.

We can see that the cartservice’s availability is violated. As we drill down, we see that there aren’t as many successful transactions, which is affecting the SLO.

As we drill into the service, we can see in Elastic APM that there is a higher than normal failure rate of about 5.5% for the emptyCart service.

We can investigate this further in APM, but that is a discussion for another blog. Stay tuned to see how we can use Elastic’s machine learning, AIOps, and AI Assistant to understand the issue.

Conclusion

SLOs allow you to set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Hopefully with the overview we’ve provided in this blog, you can see that:

• SLOs can be based on logs. In Elastic, you can use KQL to essentially find and filter on specific logs and log fields to monitor and trigger SLOs.

• AI Assistant is a valuable, easy-to-use capability to analyze, troubleshoot, and even potentially resolve SLO issues.

• APM Service based SLOs are easy to create and manage with integration to Elastic APM. We also use OTel telemetry to help monitor SLOs.

For more information on SLOs in Elastic, check out Elastic documentation and the following resources:

• What’s new in Elastic Observability 8.12

• Introducing the Elastic AI Assistant

• Elastic OpenTelemetry support

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your SLOs.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Why do we need alerts?

Logs, metrics, and traces are just the base to build a complete monitoring solution for Kubernetes clusters. Their main goal is to provide debugging information and historical evidence for the infrastructure.

While out-of-the-box dashboards, infrastructure topology, and logs exploration through Kibana are already quite handy to perform ad-hoc analyses, adding notifications and active monitoring of infrastructure allows users to deal with problems detected as early as possible and even proactively take actions to prevent their Kubernetes environments from facing even more serious issues.

How can this be achieved?

By building alerts on top of their infrastructure, users can leverage the data and effectively correlate it to a specific notification, creating a wide range of possibilities to dynamically monitor and observe their Kubernetes cluster.

In this blog post, we will explore how users can leverage Elasticsearch’s search powers to define alerting rules in order to be notified when a specific condition occurs.

SLIs, alerts, and SLOs: Why are they important for SREs?

For site reliability engineers (SREs), the incident response time is tightly coupled with the success of everyday work. Monitoring, alerting, and actions will help to discover, resolve, or prevent issues in their systems.

• An SLA (Service Level Agreement) is an agreement you create with your users to specify the level of service they can expect.
• An SLO (Service Level Objective) is an agreement within an SLA about a specific metric like uptime or response time.
• An SLI (Service Level Indicator) measures compliance with an SLO.

SREs’ day-to-day tasks and projects are driven by SLOs. By ensuring that SLOs are defended in the short term and that they can be maintained in the medium to long term, we lay the basis of a stable working infrastructure.

Having said this, identifying the high-level categories of SLOs is crucial in order to organize the work of an SRE. Then in each category of SLOs, SREs will need the corresponding SLIs that can cover the most important cases of their system under observation. Therefore, the decision of which SLIs we will need demands additional knowledge of the underlying system infrastructure.

One widely used approach to categorize SLIs and SLOs is the Four Golden Signals method. The categories defined are Latency, Traffic, Errors, and Saturation.

A more specific approach is the The RED method developed by Tom Wilkie, who was an SRE at Google and used the Four Golden Signals. The RED method drops the saturation category because this one is mainly used for more advanced cases — and people remember better things that come in threes.

Focusing on Kubernetes infrastructure operators, we will consider the following groups of infrastructure SLIs/SLOs:

• Group 1: Latency of control plane (apiserver,
• Group 2: Resource utilization of the nodes/pods (how much cpu, memory, etc. is consumed)
• Group 3: Errors (errors on logs or events or error count from components, network, etc.)

Creating alerts for a Kubernetes cluster

Now that we have a complete outline of our goal to define alerts based on SLIs/SLOs, we will dive into defining the proper alerting. Alerts can be built using Kibana.

See Elastic documentation.

In this blog, we will define more complex alerts based on complex Elasticsearch queries provided by Watcher’s functionality. Read more about Watcher and how to properly use it in addition to the examples in this blog.

Latency alerts

For this kind of alert, we want to define the basic SLOs for a Kubernetes control plane, which will ensure that the basic control plane components can service the end users without an issue. For instance, facing high latencies in queries against the Kubernetes API Server is enough of a signal that action needs to be taken.

Resource saturation

The next group of alerting will be resource utilization. Node’s CPU utilization or changes in Node’s condition is something critical for a cluster to ensure the smooth servicing of the workloads provisioned to run the applications that end users will interact with.

Error detection

Last but not least, we will define alerts based on specific errors like the network error rate or Pods’ failures like the OOMKilled situation. It’s a very useful indicator for SRE teams to either detect issues on the infrastructure level or just be able to notify developer teams about problematic workloads. One example that we will examine later is having an application running as a Pod and constantly getting restarted because it hits its memory limit. In that case, the owners of this application will need to get notified to act properly.

From Kubernetes data to Elasticsearch queries

Having a solid plan about the alerts that we want to implement, it's time to explore the data we have collected from the Kubernetes cluster and stored in Elasticsearch. For this we will consult the list of the available data fields that are ingested using the Elastic Agent Kubernetes integration (the full list of fields can be found here). Using these fields we can create various alerts like:

• Node CPU utilization
• Node Memory utilization
• BW utilization
• Pod restarts
• Pod CPU/memory utilization

CPU utilization alert

Our first example will use the CPU utilization fields to calculate the Node’s CPU utilization and create an alert. For this alert, we leverage the metrics:

kubernetes.node.cpu.usage.nanocores
kubernetes.node.cpu.capacity.cores.

The following calculation (nodeUsage / 1000000000 ) /nodeCap grouped by node name will give us the CPU utilization of our cluster’s nodes.

The Watcher definition that implements this query can be created with the following API call to Elasticsearch:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Node-CPU-Usage?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "10m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.node OR data_stream.dataset: kubernetes.state_node",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "nodes": {
              "terms": {
                "field": "kubernetes.node.name",
                "size": "10000",
                "order": {
                  "_key": "asc"
                }
              },
              "aggs": {
                "nodeUsage": {
                  "max": {
                    "field": "kubernetes.node.cpu.usage.nanocores"
                  }
                },
                "nodeCap": {
                  "max": {
                    "field": "kubernetes.node.cpu.capacity.cores"
                  }
                },
                "nodeCPUUsagePCT": {
                  "bucket_script": {
                    "buckets_path": {
                      "nodeUsage": "nodeUsage",
                      "nodeCap": "nodeCap"
                    },
                    "script": {
                      "source": "( params.nodeUsage / 1000000000 ) / params.nodeCap",
                      "lang": "painless",
                      "params": {
                        "_interval": 10000
                      }
                    },
                    "gap_policy": "skip"
                  }
                }
              }
            }
          }
        },
        "indices": [
          "metrics-kubernetes*"
        ]
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.nodes.buckets": {
        "path": "nodeCPUUsagePCT.value",
        "gte": {
          "value": 80
        }
      }
    }
  },
  "actions": {
    "log_hits": {
      "foreach": "ctx.payload.aggregations.nodes.buckets",
      "max_iterations": 500,
      "logging": {
        "text": "Kubernetes node found with high CPU usage: {{ctx.payload.key}} -> {{ctx.payload.nodeCPUUsagePCT.value}}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Node CPU Usage"
  }
}

OOMKilled Pods detection and alerting

Another Watcher that we will explore is the one that detects Pods that have been restarted due to an OOMKilled error. This error is quite common in Kubernetes workloads and is useful to detect this early on to inform the team that owns this workload, so they can either investigate issues that could cause memory leaks or just consider increasing the required resources for the workload itself.

This information can be retrieved from a query like the following:

kubernetes.container.status.last_terminated_reason: OOMKilled

Here is how we can create the respective Watcher with an API call:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Pod-Terminated-OOMKilled?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.state_container",
                          "analyze_wildcard": true
                        }
                      },
                      {
                        "exists": {
                          "field": "kubernetes.container.status.last_terminated_reason"
                        }
                      },
                      {
                        "query_string": {
                          "query": "kubernetes.container.status.last_terminated_reason: OOMKilled",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "pods": {
              "terms": {
                "field": "kubernetes.pod.name",
                "order": {
                  "_key": "asc"
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.pods.buckets": {
        "path": "doc_count",
        "gte": {
          "value": 1,
          "quantifier": "some"
        }
      }
    }
  },
  "actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/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",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Pod Terminated OOMKilled"
  }
}

From Kubernetes data to alerts summary

So far we saw how we can start from plain Kubernetes fields, use them in ES queries, and build Watchers and alerts on top of them.

One can explore more possible data combinations and build queries and alerts following the examples we provided here. A full list of alerts is available, as well as a basic scripted way of installing them.

Of course, these examples come with simple actions defined that only log messages into the Elasticsearch logs. However, one can use more advanced and useful outputs like Slack’s webhooks:

"actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/T04SW3JHXasdfasdfasdfasdfasdf",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  }

The result would be a Slack message like the following:

Next steps

In our next steps, we would like to make these alerts part of our Kubernetes integration, which would mean that the predefined alerts would be installed when users install or enable the Kubernetes integration. At the same time, we plan to implement some of these as Kibana’s native SLIs, providing the option to our users to quickly define SLOs on top of the SLIs through a nice user interface. If you’re interested to learn more about these, follow the public GitHub issues for more information and feel free to provide your feedback:

• https://github.com/elastic/package-spec/issues/484
• https://github.com/elastic/kibana/issues/150050

For those who are eager to start using Kubernetes alerting today, here is what you need to do:

1. Make sure that you have an Elastic cluster up and running. The fastest way to deploy your cluster is to spin up a free trial of Elasticsearch Service.
2. Install the latest Elastic Agent on your Kubernetes cluster following the respective documentation.
3. Install our provided alerts that can be found at https://github.com/elastic/integrations/tree/main/packages/kubernetes/docs or at https://github.com/elastic/k8s-integration-infra/tree/main/scripts/alerting.

Of course, if you have any questions, remember that we are always happy to help on the Discuss forums.

We’ve recently released a beta feature that provides a Git-friendly IaaC oriented workflow. You can now push Heartbeat monitors with the same ease with which you push code changes in Git or config changes in Terraform. The features discussed in this blog are all currently in beta, and we urge users trying these features out to upgrade to the latest stack version first. When these features become GA, this new workflow will be the preferred way of configuring monitors in the Elastic Stack. If you’re starting a new project, you may want to consider setting it up this way instead of via our more classic configuration.

Today, using Heartbeat is simple. You just need to write a little YAML and monitoring data shows up in Elasticsearch, visible in the Uptime UI. While the UI is indeed simple, there’s some hidden complexity there that we’ve improved with a new UI (the Synthetics app) and augmented with an even more automation friendly CLI workflow via our new Projects feature, which will be discussed below.

How do you manage your configs written in YAML? Many of our users will manage YAML in Git and use tooling such as Ansible, Helm, or similar to manage their infrastructure as code (IaaC). As with any other organization, Elastic also heavily utilizes IaaC in all parts of our operations. Hence it’s only natural we developed a capability to provide you with similar support for the current Heartbeat capability and the upcoming synthetics monitoring capabilities.

Projects: A new way to organize and distribute configs

Let’s dive right into what we’re calling “Synthetics Projects” and how they differ from traditional Heartbeat config files. To use this feature, you would start by creating a project in a Git repo containing your configs. At a high level, setting up a project requires performing the following tasks:

1. Run npx @elastic/synthetics init to create a project skeleton in a directory. See more details on the npmjs.com site.
2. Run git init and git push on the generated directory to version it as a Git repository.
3. Add your lightweight YAML files and browser javascript/typescript files to the journeys folder.
4. Test that it works by running npx @elastic/synthetics push command to sync your project to your Elastic Stack.
5. Configure a CI/CD pipeline to test pull requests to your Git repo and to execute npx @elastic/synthetics push on merges to the main branch.

So, once configured, adding, removing, and editing monitors involves:

1. Editing a monitor’s config, either YAML for lightweight monitors, or Javascript/Typescript for browser based ones locally
2. Testing your local configs with npx @elastic/synthetics journeys
3. Creating a new PR to your main branch via a Git push
4. Waiting for your CI server to perform the same validation and waiting for someone else on your team to review your PR
5. Merging your result to the main branch
6. Waiting for your CI server to push the changes to your Elastic stack

We’ve depicted the flow of data in the diagram below:

This is, in fact, the way many of our users work today, with other software taking the place of npx @elastic/synthetics push as mentioned earlier. Indeed, in the future, we will most likely look into building a Terraform provider, though that isn’t something we’re actively working on now.

Just have a few monitors? Use the GUI!

The above approach is great for sophisticated users with larger numbers of configurations, but if you just want to monitor a few URLs, it’s overkill. If that sounds like you, consider the new Monitor Management UI in the Uptime app! It works in the exact same way, saving configs to your Elastic Stack, but with no need for Git, or a project, or all that other infrastructure. Simply, log in, fill out the form pictured below, and hit save. If you want to set up a private location, that is still done in the same way via Fleet.

What about my existing Fleet monitors?

A small subset of users have monitors configured today using the Synthetics Fleet integration. If that describes you, you’ll want to move onto either the GUI based approach or the Project based approach, as those methods supersede direct usage of the Fleet integration, which will eventually be restricted only to use via the above described methods.

The Fleet approach is inferior in a few ways:

1. It can only configure monitors for a single location.
2. It creates a different UX for monitors configured on the service versus private locations.
3. It’s less fluid of an integration with the Uptime UI.

It’s rare for us to deprecate beta features, but in this case we had a clearly superior alternative. Maintaining both would have created a more confusing and unwieldy product. We don’t yet have an exact date for removing support for these monitors, but you can track this via this GitHub issue.

Outside of cultural challenges, one of the most common reasons for this disconnect is using different tools to achieve similar goals — case in point, end-to-end (e2e) testing versus synthetic monitoring.

This blog shares an overview of these techniques. Using the example repository carlyrichmond/synthetics-replicator, we’ll also show how Playwright, @elastic/synthetics, and GitHub Actions can combine forces with Elastic Synthetics and the recorder to unite development and SRE teams in validating and monitoring the user experience for a simple web application hosted on a provider such as Netlify.

Elastic recently introduced synthetics monitoring, and as highlighted in our prior blog, it can replace e2e tests altogether. Uniting around a single tool to validate the user workflow early provides a common language to recreate user issues to validate fixes against.

Synthetics Monitoring versus e2e tests

If development and operations tools are at war, it’s difficult to unify their different cultures together. Considering the definitions of these approaches shows that they in fact aim to achieve the same objective.

e2e tests are a suite of tests that recreate the user path, including clicks, user text entry, and navigations. Although many argue it’s about testing the integration of the layers of a software application, it’s the user workflow that e2e tests emulate. Meanwhile, Synthetic Monitoring, specifically a subset known as browser monitoring, is an application performance monitoring practice that emulates the user path through an application.

Both these techniques emulate the user path. If we use tooling that crosses the developer and operational divide, we can work together to build tests that can also provide production monitoring in our web applications.

Creating user journeys

When a new user workflow, or set of features that accomplish a key goal, is under development in our application, developers can use @elastic/synthetics to create user journeys. The initial project scaffolding can be generated using the init utility once installed, as in the below example. Note that Node.js must be installed prior to using this utility.

npm install -g @elastic/synthetics
npx @elastic/synthetics init synthetics-replicator-tests

Before commencing the wizard, make sure you have your Elastic cluster information and the Elastic Synthetics integration set on your cluster. You will need:

1. Monitor Management must be enabled within the Elastic Synthetics app as per the prerequisites in the documentation getting started.
2. The Elastic Cloud cluster Cloud ID if using Elastic Cloud. Alternatively, if you are using on-prem hosting you need to enter your Kibana endpoint.
3. An API key generated from your cluster. There is a shortcut in the Synthetics application Settings to generate this key under the Project API Keys tab, as shown in the documentation.

This wizard will take you through and generate a sample project containing configuration and example monitor journeys, with a structure similar to the below:

For web developers, most of the elements such as the README and package.json and lock files will be familiar. The main configuration for your monitors is available in synthetics.config.tsas shown below. This configuration can be amended to include production and development-specific configuration. This is essential for combining forces and reusing the same monitors for e2e tests and allowing for any journeys to be used as e2e tests and production monitors. Although not in this example, details of private locations can be included if you would prefer to monitor from your own dedicated Elastic instance rather than from Elastic infrastructure.

import type { SyntheticsConfig } from "@elastic/synthetics";

export default (env) => {
  const config: SyntheticsConfig = {
    params: {
      url: "http://localhost:5173",
    },
    playwrightOptions: {
      ignoreHTTPSErrors: false,
    },
    /**
     * Configure global monitor settings
     */
    monitor: {
      schedule: 10,
      locations: ["united_kingdom"],
      privateLocations: [],
    },
    /**
     * Project monitors settings
     */
    project: {
      id: "synthetics-replicator-tests",
      url: "https://elastic-deployment:port",
      space: "default",
    },
  };
  if (env === "production") {
    config.params = { url: "https://synthetics-replicator.netlify.app/" };
  }
  return config;
};

Writing your first journey

Although the above configuration applies to all monitors in the project, it can be overridden for a given test.

import { journey, step, monitor, expect, before } from "@elastic/synthetics";

journey("Replicator Order Journey", ({ page, params }) => {
  // Only relevant for the push command to create
  // monitors in Kibana
  monitor.use({
    id: "synthetics-replicator-monitor",
    schedule: 10,
  });

  // journey steps go here
});

The @elastic/synthetics wrapper exposes many standard test methods such as the before and after constructs that allow for setup and tear down of typical properties in the tests, as well as support for many common assertion helper methods. A full list of supported expect methods are listed in the documentation. The Playwright page object is also exposed, which enables us to perform all the expected activities provided in the API such as locating page elements and simulating user events such as clicks that are depicted in the below example.

import { journey, step, monitor, expect, before } from "@elastic/synthetics";

journey("Replicator Order Journey", ({ page, params }) => {
  // monitor configuration goes here

  before(async () => {
    await page.goto(params.url);
  });

  step("assert home page loads", async () => {
    const header = await page.locator("h1");
    expect(await header.textContent()).toBe("Replicatr");
  });

  step("assert move to order page", async () => {
    const orderButton = await page.locator("data-testid=order-button");
    await orderButton.click();

    const url = page.url();
    expect(url).toContain("/order");

    const menuTiles = await page.locator("data-testid=menu-item-card");
    expect(await menuTiles.count()).toBeGreaterThan(2);
  });

  // other steps go here
});

As you can see in the above example, it also exposes the journey and step constructs. This construct mirrors the behavior-driven development (BDD) practice of showing the user journey through the application in tests.

Developers are able to execute the tests against a locally running application as part of their feature development to see successful and failed steps in the user workflow. In the below example, the local server startup command is outlined in blue at the top. The monitor execution command is presented in red further down.

As you can see from the green ticks next to each journey step, each of our tests pass. Woo!

Gating your CI pipelines

It’s important to use the execution of the monitors within your CI pipeline as a gate for merging code changes and uploading the new version of your monitors. Each of the jobs in our GitHub Actions workflow will be discussed in this and the subsequent section.

The test job spins up a test instance and runs our user journeys to validate our changes, as illustrated below. This step should run for pull requests to validate developer changes, as well as on push.

jobs:
  test:
    env:
      NODE_ENV: development
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - run: npm install
      - run: npm start &
      - run: "npm install @elastic/synthetics && SYNTHETICS_JUNIT_FILE='junit-synthetics.xml' npx @elastic/synthetics . --reporter=junit"
        working-directory: ./apps/synthetics-replicator-tests/journeys
      - name: Publish Unit Test Results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          junit_files: "**/junit-*.xml"
          check_name: Elastic Synthetics Tests

Note that, unlike the journey execution on our local machine, we make use of the --reporter=junit option when executing npx @elastic/synthetics to provide visibility of our passing, or sadly sometimes failing, journeys to the CI job.

Automatically upload monitors

To ensure the latest monitors are available in Elastic Uptime, it’s advisable to push the monitors programmatically as part of the CI workflow such as the example task below does. Our workflow has a second job push, shown below, which is dependent on the successful execution of our test job that uploads your monitors to your cluster. Note that this job is configured in our workflow to run on push to ensure changes have been validated rather than just raised within a pull request.

jobs:
  test: …
  push:
    env:
      NODE_ENV: production
      SYNTHETICS_API_KEY: ${{ secrets.SYNTHETICS_API_KEY }}
    needs: test
    defaults:
      run:
        working-directory: ./apps/synthetics-replicator-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - run: npm install
      - run: npm run push

The @elastic/synthetics init wizard generates a push command for you when you create your project that can be triggered from the project folder. This is shown below through the steps and working_directory configuration. The push command requires the API key from your Elastic cluster, which should be stored as a secret within a trusted vault and referenced via a workflow environment variable. It is also vital that monitors pass ahead of pushing the updated monitor configuration to your Elastic Synthetics instance to prevent breaking your production monitoring. Unlike e2e tests running against a testing environment, broken monitors impact SRE activities and therefore any changes need to be validated. For that reason, applying a dependency to your test step via the needs option is recommended.

Monitoring using Elastic Synthetics

Once monitors have been uploaded, they give a regular checkpoint to SRE teams as to whether the user workflow is functioning as intended — not just because they will run on a regular schedule as configured for the project and individual tests as shown previously, but also due to the ability to check the state of all monitor runs and execute them on demand.

The Monitors Overview tab gives us an immediate view of the status of all configured monitors, as well as the ability to run the monitor manually via the card ellipsis menu.

From the Monitor screen, we can also navigate to an overview of an individual monitor execution to investigate failures.

The other monitoring superpower SREs now have is the integration between these monitors to familiar tools SREs already use in scrutinizing the performance and availability of applications such as APM, metrics, and logs. The aptly named Investigate menu allows easy navigation while SREs are performing investigations into potential failures or bottlenecks.

There is also a balance between finding issues and being notified of potential problems automatically. SREs already familiar with setting rules and thresholds for notification of issues will be happy to know that this is also possible for browser monitors. The editing of an example rule is shown below.

The status of browser monitors can be configured not only to consider if any individual or collective monitors have been down several times, such as in the status check above, but also to gauge the overall availability by looking at the percentage of passed checks within a given time period. SREs are not only interested in reacting to issues in a traditional production management way — they want to improve the availability of applications, too.

Recording user workflows

The limitation of generating e2e tests through the development lifecycle is that sometimes teams miss things, and the prior toolset is geared toward development teams. Despite the best intentions to design an intuitive product using multi-discipline teams, users may use applications in unintended ways. Furthermore, the monitors written by developers will only cover those expected workflows and raise the alarm either when these monitors fail in production or when they start to behave differently if anomaly detection is applied to them.

When user issues arise, it’s useful to recreate that problem in the same format as our monitors. It’s also important to leverage the experience of SREs in generating user journeys, as they will consider failure cases intuitively where developers may struggle and focus on happy cases. However, not all SREs will have the experience or confidence to write these journeys using Playwright and @elastic/synthetics.

Enter the Elastic Synthetics Recorder! The above video gives a walkthrough of how it can be used to record the steps in a user journey and export them to a JavaScript file for inclusion in your monitor project. This is useful for feeding back into the development phase and testing developed fixes to solve the problem. This approach cannot be made unless we all combine forces to use these monitors together.

Try it out!

As of 8.8, @elastic/synthetics and the Elastic Synthetics app are generally available, and the trusty recorder is in beta. Share your experiences of bridging the developer and operations divide with Synthetic Monitoring via the Uptime category in the Community Discuss forums or via Slack.

Happy monitoring!

Originally published February 6, 2023; updated May 23, 2023.

1. Why and how to replace end-to-end tests with synthetic monitors
2. Uptime and Synthetic Monitoring
3. Scripting browser monitors
4. Use the Synthetics Recorder
5. Playwright
6. GitHub Actions