Create an APM anomaly rule

To create APM anomaly rules, you need the following:

Note

Rules use an API key to authorize all background tasks, including condition checks and triggered actions. The key type depends on your deployment. For details on how API keys authorize rules, refer to API keys.

This rule queries the ML anomaly indices and the transaction indices defined in the APM index settings. The default index patterns are .ml-anomalies-* and traces-apm*,apm-*,traces-*.otel-*. You cannot override these indices on a per-rule basis.

You can create an anomaly rule to alert you when either the latency, throughput, or failed transaction rate of a service is unusual. Anomaly rules can be set at different levels: environment, service, and transaction type. Add actions to raise alerts via services or third-party integrations (for example, send an email or create a Jira issue).

From an Applications page, open Alerts in the application menu and select Create anomaly rule. If Alerts is not visible, open the More menu first.

  1. In Definition, select the Service, Type, and Environment, or leave All selected.
  2. Under Detector types, select the anomalies that should trigger alerts. All four types are selected by default: latency, throughput, failed transaction rate, and low transaction count.
  3. Set Has anomaly with severity to the minimum severity to alert on: warning, minor, major, or critical.
  4. Set the rule schedule and click Next.
  5. In Actions, optionally configure notifications or other actions, then continue to Details.
  6. In Details, enter the Rule name and optional Tags, then click Create rule.

These steps show how to use the Alerts UI. You can also create an anomaly rule directly from any page within Applications. Click the Alerts and rules button, and select Create anomaly rule. When you create a rule this way, the Name and Tags fields will be prepopulated but you can still change these.

To create your anomaly rule:

  1. In Observability UI, go to Alerts.
  2. Select Manage Rules from the Alerts page, and select Create rule.
  3. Enter a Name for your rule, and any optional Tags for more granular reporting (leave blank if unsure).
  4. Select the APM Anomaly rule type.
  5. Select the appropriate Service, Type, and Environment (or leave ALL to include all options).
  6. Select the desired severity (critical, major, minor, warning) from Has anomaly with severity.
  7. Define the interval to check the rule (for example, check every 1 minute).
  8. (Optional) Set up Actions.
  9. Save your rule.

Select low transaction count under Detector types to alert when a service's transaction count drops abnormally, including when traffic stops reaching it. This detector is selected by default for new rules. Existing rules retain their saved detector selections, so edit them to include it.

The environment's APM machine learning job must also contain the low_count detector. Selecting it in a rule does not update the job. If the detector is missing, follow Add low-count detection to existing jobs.

You can extend your rules with actions that interact with third-party systems, write to logs or indices, or send user notifications. You can add an action to a rule at any time. You can create rules without adding actions, and you can also define multiple actions for a single rule.

To add actions to rules, you must first create a connector for that service (for example, an email or external incident management system), which you can then use for different rules, each with their own action frequency.

Connector types

Connectors provide a central place to store connection information for services and integrations with third party systems. The following connectors are available when defining actions for alerting rules:

Note

Some connector types are paid commercial features, while others are free. For a comparison of the Elastic subscription levels, go to the subscription page.

For more information on creating connectors, refer to Connectors.

Action frequency

After you select a connector, you must set the action frequency. You can choose to create a Summary of alerts on each check interval or on a custom interval. For example, you can send email notifications that summarize the new, ongoing, and recovered alerts every twelve hours.

Alternatively, you can set the action frequency to For each alert and specify the conditions each alert must meet for the action to run. For example, you can send an email only when the alert status changes to critical.

Configure when a rule is triggered

With the Run when menu you can choose if an action runs when the threshold for an alert is reached, or when the alert is recovered. For example, you can add a corresponding action for each state to ensure you are alerted when the rule is triggered and also when it recovers.

Choose between threshold met or recovered
Action variables

Use the default notification message or customize it. You can add more context to the message by clicking the Add variable icon Add variable and selecting from a list of available variables.

Action variables list

The following variables are specific to this rule type. You can also specify variables common to all rules.

context.alertDetailsUrl
Link to the alert troubleshooting view for further context and details. This will be an empty string if the server.publicBaseUrl is not configured.
context.environment
The transaction type the alert is created for.
context.reason
A concise description of the reason for the alert.
context.serviceName
The service the alert is created for.
context.threshold
Any trigger value above this value will cause the alert to fire.
context.transactionType
The transaction type the alert is created for.
context.triggerValue
The value that breached the threshold and triggered the alert.
context.viewInAppUrl

Link to the alert source.