Elasticsearch exporter

Stack Serverless Observability

The Elasticsearch exporter is an OpenTelemetry Collector component that sends logs, metrics, and traces to Elasticsearch. The exporter supports multiple mapping modes and provides flexible configuration options for data routing, authentication, and performance tuning.

To use the Elasticsearch exporter, include it in the exporter definitions of the Collector configuration. The exporter is already included in the default configuration.

The Elasticsearch exporter supports various configuration options for connecting to Elasticsearch, mapping data, and optimizing performance.

You must specify exactly one of the following connection methods:

• endpoint: A single Elasticsearch URL. For example, https://elasticsearch:9200.
• endpoints: A list of Elasticsearch URLs for round-robin load balancing.
• cloudid: An Elastic Cloud ID for connecting to Elastic Cloud.

If none of the previous settings are specified, the exporter relies on the ELASTICSEARCH_URL environment variable.

The exporter supports standard OpenTelemetry authentication configuration. You can also use these simplified authentication options:

• user and password: For HTTP Basic Authentication
• api_key: For Elasticsearch API key authentication

The exporter supports standard OpenTelemetry TLS configuration for secure connections. You can configure TLS certificates, client authentication, and other security settings through the standard TLS configuration options.

The exporter supports several mapping modes that determine how your telemetry data is preprocessed and stored in Elastic. You can configure the mapping mode through the mapping setting:

Stack 8.12.0

The default mapping mode is otel. In the otel mapping mode, the Elasticsearch Exporter stores documents in Elastic's preferred OTel-native schema. In this mapping mode, documents use the original attribute names and closely follow the event structure from the OTLP events.

In ecs mapping mode, the Elasticsearch Exporter maps fields from OpenTelemetry Semantic Conventions (SemConv) to Elastic Common Schema (ECS) where possible. This mode can be used for compatibility with existing dashboards that work with ECS. Refer to ECS & OpenTelemetry for more details.

In bodymap mapping mode, the Elasticsearch Exporter supports only logs and takes the body of a log record as the exact content of the Elasticsearch document without any transformation. Use this mapping mode when you want to have complete control over the Elasticsearch document structure.

In the none mapping mode the Elasticsearch Exporter produces documents with the original field names from the OTLP data structures.

The raw mapping mode is identical to none, except for two differences:

• In none mode attributes are mapped with an Attributes. prefix, while in raw mode they are not.
• In none mode span events are mapped with an Events. prefix, while in raw mode they are not.

Documents are statically or dynamically routed to the target index or data stream. The first routing mode that applies is used, in the following order:

Static mode routes documents to logs_index for log records, metrics_index for data points, and traces_index for spans, if these configs aren't empty respectively.

Dynamic mode (Index attribute) routes documents to index name specified in elasticsearch.index attribute, with the following order of precedence: log record / data point / span attribute -> scope attribute -> resource attribute if the attribute exists.

Dynamic mode (Data stream routing) routes documents to data stream constructed from ${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}, where data_stream.type is logs for log records, metrics for data points, and traces for spans, and is static. The following rules apply:

• data_stream.dataset or data_stream.namespace in attributes, with the following order of precedence: log record / data point / span attribute -> scope attribute -> resource attribute
• Otherwise, if the scope name matches the /receiver/(\w*receiver) regular expression, data_stream.dataset is the first capture group.
• Otherwise, data_stream.dataset falls back to generic and data_stream.namespace falls back to default.

If the mapping mode is set to bodymap, the data_stream.type field can be dynamically set from attributes. The resulting documents contain the corresponding data_stream.* fields. Refer to Data Stream Fields for the restrictions applied to the data stream fields.

These settings allow you to customize document routing:

When using the default OpenTelemetry mapping mode, additional handling is applied to the previous document routing rules:

1. Static mode: Span events are separate documents routed to logs_index if non-empty.
2. Dynamic - Index attribute mode: Span events are separate documents routed using attribute elasticsearch.index, with the following order of precedence: span event attribute -> scope attribute -> resource attribute if the attribute exists.
3. Dynamic - Data stream routing mode: For all documents, data_stream.dataset always ends with .otel. Span events are separate documents that have data_stream.type: logs and are routed using data stream attributes, with the following order of precedence: span event attribute -> scope attribute -> resource attribute.

The elasticsearch.index attribute is removed from the final document if it exists.

The exporter supports both internal batching and OpenTelemetry's standard sending_queue configuration:

By default, the exporter performs its own buffering and batching, as configured through the flush setting, unless the sending_queue and batcher settings are defined.

The Elasticsearch exporter supports the sending_queue setting, which supports both queueing and batching. However, the sending queue is currently deactivated by default. You can turn on the sending queue by setting sending_queue to true. Batching support in sending queue is also deactivated by default and can be turned on by defining sending_queue::batch. For example:

exporters:
  elasticsearch:
    endpoint: https://elasticsearch:9200
    sending_queue:
      enabled: true
      batch:
        enabled: true
        min_size: 1000
        max_size: 10000
        timeout: 5s
		

The Elasticsearch exporter uses the Elasticsearch Bulk API for indexing documents. Configure the behavior of bulk indexing with the following settings:

Starting from Elasticsearch 8.18 and higher, the include_source_on_error query parameter allows users to receive the source document in the error response if there were parsing errors in the bulk request. In the exporter, the equivalent configuration is also named include_source_on_error.

• include_source_on_error:
  • true: Turns on bulk index responses to include source document on error. Stack 8.18.0
  • false: Turns off including source document on bulk index error responses. Stack 8.18.0
  • null (default): Backward-compatible option for older Elasticsearch versions. By default, the error reason is discarded from bulk index responses entirely. Only the error type is returned.

Documents can be passed through an [Elasticsearch Ingest pipeline] before indexing. Use these settings to configure the ingest pipeline:

For example:

exporters:
  elasticsearch:
    endpoint: https://elasticsearch:9200
    pipeline: "my-custom-pipeline"
		

The Elasticsearch Exporter regularly checks Elasticsearch for available nodes. Newly discovered nodes are automatically used for load balancing.

The following settings are related to node discovery:

• discover:
  • on_start (optional): If enabled the exporter queries Elasticsearch for all known nodes in the cluster on startup.
  • interval (optional): Interval to update the list of Elasticsearch nodes.

To turn off node discovery, set discover.interval to 0.

The following are some known limitations of the Elasticsearch exporter:

• Metrics support is currently in development and might have limitations.
• Profile support requires Universal Profiling to be installed in Elasticsearch.
• Some mapping modes might have reduced functionality for certain telemetry types.
• The bodymap mode only supports logs and ignores other telemetry types.

The following are the main known issues with the Elasticsearch exporter:

When you encounter issues with the Elasticsearch exporter, you can try the following:

• Make sure your Elasticsearch version is compatible with your chosen mapping mode.
• Verify your API keys or credentials are valid and have appropriate permissions.
• Check that your Elasticsearch cluster supports the required features for your mapping mode.