« Migration guide Migrating to 8.18 »
Elastic Docs ›Elasticsearch Guide [8.19] ›Migration guide

Migrating to 8.19

edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Migrating to 8.19

edit

This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 8.19.

See also What’s new in 8.19 and Release notes.

Coming in 8.19.0.

Breaking changes

edit

The following changes in Elasticsearch 8.19 might affect your applications and prevent them from operating normally. Before upgrading to 8.19, review these changes and take the described steps to mitigate the impact.

Cluster and node setting changes

edit
Upgrade discovery-ec2 to AWS SDK v2

Details
In earlier versions of Elasticsearch the discovery-ec2 plugin was based on the AWS SDK v1. AWS will withdraw support for this SDK before the end of the life of Elasticsearch 8.19 so we must migrate to the newer AWS SDK v2. Unfortunately there are several differences between the two AWS SDK versions which may require you to adjust your system configuration when upgrading to Elasticsearch 8.19 or later. These differences include, but may not be limited to, the following items.

  • AWS SDK v2 does not support the EC2 IMDSv1 protocol.
  • AWS SDK v2 does not support the aws.secretKey or com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system properties.
  • AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so the discovery.ec2.protocol setting is no longer effective.
  • AWS SDK v2 does not accept an access key without a secret key or vice versa.

Impact
If you use the discovery-ec2 plugin, test your upgrade thoroughly before upgrading any production workloads. Adapt your configuration to the new SDK functionality. This includes, but may not be limited to, the following items.

  • If you use IMDS to determine the availability zone of a node or to obtain credentials for accessing the EC2 API, ensure that it supports the IMDSv2 protocol.
  • If applicable, discontinue use of the aws.secretKey and com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system properties.
  • If applicable, specify that you wish to use the insecure HTTP protocol to access the EC2 API by setting discovery.ec2.endpoint to a URL which starts with http://.
  • Either supply both an access key and a secret key using the keystore settings discovery.ec2.access_key and discovery.ec2.secret_key, or configure neither of these settings.
Upgrade repository-s3 to AWS SDK v2

Details
In earlier versions of Elasticsearch the repository-s3 plugin was based on the AWS SDK v1. AWS will withdraw support for this SDK before the end of the life of Elasticsearch 8.19 so we must migrate to the newer AWS SDK v2. Unfortunately there are several differences between the two AWS SDK versions which may require you to adjust your system configuration when upgrading to Elasticsearch 8.19 or later. These differences include, but may not be limited to, the following items.

  • AWS SDK v2 requires users to specify the region to use for signing requests, or else to run in an environment in which it can determine the correct region automatically. The older SDK would try to determine the region based on the endpoint URL as specified with the s3.client.${CLIENT_NAME}.endpoint setting, together with other data drawn from the operating environment, and would ultimately fall back to us-east-1 if no better value could be found.
  • AWS SDK v2 does not support the EC2 IMDSv1 protocol.
  • AWS SDK v2 does not support the com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system property.
  • AWS SDK v2 does not permit specifying a choice between HTTP and HTTPS so the s3.client.${CLIENT_NAME}.protocol setting is deprecated.
  • AWS SDK v2 does not permit control over throttling for retries, so the the s3.client.${CLIENT_NAME}.use_throttle_retries setting is deprecated and no longer has any effect.
  • AWS SDK v2 requires the use of the V4 signature algorithm, so the s3.client.${CLIENT_NAME}.signer_override setting is deprecated and no longer has any effect.
  • AWS SDK v2 does not support the log-delivery-write canned ACL.
  • AWS SDK v2 counts 4xx responses differently in its metrics reporting.
  • AWS SDK v2 always uses the regional STS endpoint, whereas AWS SDK v1 could use either a regional endpoint or the global https://sts.amazonaws.com one.

Impact
If you use the repository-s3 module, test your upgrade thoroughly before upgrading any production workloads. Adapt your configuration to the new SDK functionality. This includes, but may not be limited to, the following items.

  • Specify the correct signing region using the s3.client.${CLIENT_NAME}.region setting on each node. Elasticsearch will try and determine the correct region based on the endpoint URL and other data drawn from the operating environment but cannot guarantee to do so correctly in all cases.
  • If you use IMDS to determine the availability zone of a node or to obtain credentials for accessing the EC2 API, ensure that it supports the IMDSv2 protocol.
  • If applicable, discontinue use of the com.amazonaws.sdk.ec2MetadataServiceEndpointOverride system property.
  • If applicable, specify the protocol to use to access the S3 API by setting s3.client.${CLIENT_NAME}.endpoint to a URL which starts with http:// or https://.
  • If applicable, discontinue use of the log-delivery-write canned ACL.

ES|QL changes

edit
Allow partial results by default in ES|QL

Details
In earlier versions of Elasticsearch, ES|QL would fail the entire query if it encountered any error. ES|QL now returns partial results instead of failing when encountering errors.

Impact
Callers should check the is_partial flag returned in the response to determine if the result is partial or complete. If returning partial results is not desired, this option can be overridden per request via an allow_partial_results parameter in the query URL or globally via the cluster setting esql.query.allow_partial_results.

Cluster setting "skip_unavailable" catches all runtime errors

Details
If skip_unavailable is set to true, the runtime errors from this cluster do not lead to a failure of the query. Instead, the cluster is set to skipped or partial status, and the query execution continues. This is a breaking change from previous versions, where skip_unavailable only applied to errors related to a cluster being unavailable.

Impact
The errors on remote clusters, e.g. missing indices, will not lead to a failure of the query. Instead, the cluster is set to skipped or partial status in the response metadata.

Disallow mixed quoted/unquoted patterns in FROM

Details
Previously, the ES|QL grammar allowed users to individually quote constituent strings in index patterns such as "remote_cluster":"index_name". This would allow users to write complex malformed index patterns that often slip through grammar and the subsequent validation. This could result in runtime errors that can be misleading. This change simplifies the grammar to early reject such malformed index patterns at the parsing stage, allowing users to write simpler queries and see more relevant and meaningful errors.

Impact
Users can write queries with simpler index patterns and see more meaningful and relevant errors.

Unquoted index patterns do not allow ( and ) characters

Details
Previously, ES|QL accepted unquoted index patterns containing brackets, such as FROM index(1) | ENRICH policy(2). This query syntax is no longer valid because it could conflict with subquery syntax, where brackets are used as delimiters. Brackets are now only allowed in quoted index patterns. For example: FROM "index(1)" | ENRICH "policy(2)".

Impact
This affects existing queries containing brackets in index or policy names, i.e. in FROM, ENRICH, and LOOKUP JOIN commands.

Deprecations

edit

The following functionality has been deprecated in Elasticsearch 8.19 and will be removed in a future version. While this won’t have an immediate impact on your applications, we strongly encourage you to take the described steps to update your code after upgrading to 8.19.

To find out if you are using any deprecated functionality, enable deprecation logging.

Ingest deprecations

edit
Deprecate indices.merge.scheduler.use_thread_pool setting

Details
This deprecates the indices.merge.scheduler.use_thread_pool node setting that was introduced in

Impact
There should be no impact to users since the setting was not released before its deprecation here (and is not documented).

REST API deprecations

edit
Deprecate ability to connect to nodes of versions 8.18 and earlier

Details
Versions 9.1.0 and later of Elasticsearch will not support communication with nodes of versions earlier than 8.19.0, so the ability to connect to nodes of earlier versions is deprecated in this version. This applies both to communication within a cluster and communication across clusters (e.g. for cross-cluster search or cross-cluster replication). Elasticsearch will report in its deprecation logging each time it opens a connection to a node that will not be supported from version 9.1.0 onwards. You must upgrade all your clusters to version 8.19.0 or later before upgrading any of your clusters to 9.1.0 or later.

Impact
Upgrade all of your clusters to at least 8.19.0 before upgrading any of them to 9.1.0 or later.

« Migration guide Migrating to 8.18 »