Edit field mappings

When you build an integration, you need to tell Elasticsearch how to store and index each field in your data. This is called mapping. Mappings define the data type (like keyword, date, or integer) and other properties for every field your integration creates.

Each data stream in your integration has a fields directory. This directory contains YAML files that describe all the fields for that data stream. When you build your integration, these files are combined to create the mapping for the data stream.

Like ingest pipelines, mappings only apply to the data stream dataset, for our example the apache.access dataset.

Example structure:

apache
└───data_stream
    ├── access
    │   └── fields
    │       ├── agent.yml
    │       ├── base-fields.yml
    │       ├── ecs.yml
    │       └── fields.yml
    ├── error
    │   └── fields
    │       ├── agent.yml
    │       ├── base-fields.yml
    │       ├── ecs.yml
    │       └── fields.yml
    └── status
		

ECS is a shared schema for common fields (like host.name, event.dataset, etc.).

• If your integration only supports Elastic Stack 8.13.0 and above: You can rely on the ecs@mappings component template installed by Fleet. This makes explicitly declaring ECS fields unnecessary; the ecs@mappings component template in Elasticsearch will automatically detect and configure them. However, should ECS fields be explicitly defined, they will overwrite the dynamic mapping provided by the ecs@mappings component template.

• If your integration supports older versions (<8.13.0): You can import ECS mappings dynamically by setting import_mappings: true in the ECS section of the _dev/build/build.yml file in the root of the package directory. This introduces a dynamic mapping with most of the ECS definitions. Using this method means that, just like the previous approach, ECS fields don’t need to be defined in your integration, they are dynamically integrated into the package at build time. Explicitly defined ECS fields can be used and will also overwrite this mechanism.

  An example of the aformentioned build.yml file for this method:

  dependencies:
    ecs:
      reference: git@v8.6.0
      import_mappings: true
  		

• Explicitly define or import individual ECS fields: You can always explicitly define ECS fields, which will override dynamic mappings. This can be done in two ways:

  • Use external: ecs to reference a field from ECS.
  • Or, define the field directly in your YAML.

  How does external: ecs work?

  • When you define a field in your integration’s mapping YAML with external: ecs, you’re telling the elastic-package tool not to define the field’s mapping details directly in your package. Instead, you want to import the field definition from the official ECS.

  • The elastic-package CLI will look up the field’s definition in an external ECS reference file, rather than requiring you to copy the full field definition into your package.

  • Important: The ECS reference file location (and version) is controlled by the reference setting in the package’s _dev/build/build.yml file. If the _dev/build/build.yml file does not exist, or if it does not specify an ECS reference, defining fields with external: ecs will fail the build process.

  • By default, if configured, the ECS reference point to the ECS reference file hosted on GitHub, but its value must be explicitly set in _dev/build/build.yml. For example:

    dependencies:
      ecs:
        reference: git@v8.6.0
    		

    This tells elastic-package to use ECS version 8.6.0.

    Example explicit field definition:

    - name: cloud.acount.id
      level: extended
      type: keyword
      ignore_above: 1024
      description: 'The cloud account or organ....'
      example: 43434343
    		

• Local ECS reference file (air-gapped setup): In some environments—such as secure, air-gapped, or offline setups—you may not have internet access to fetch the ECS field definitions directly from GitHub. In these cases, you can download the ECS reference file manually and tell the elastic-package tool to use this local file instead of the remote one.

  • First, download the ECS YAML file to a location on your local machine or network.

  • In your integration’s _dev/build/build.yml file, set the reference to the full file path of your downloaded ECS reference. For example:

  dependencies:
    ecs:
      reference: file:///home/user/integrations/packages/apache/ecs_nested.yml
  		

  (Make sure to use the correct absolute path for your environment.)

  • Now, when you use external: ecs in your field mappings, the elastic-package CLI will look up field definitions in your local ECS file, not on GitHub.

If your integration needs fields that aren’t in ECS, define them in fields.yml:

The example below defines field apache.access.ssl.protocol in the Apache integration.

- name: apache.access
  type: group
  fields:
    - name: ssl.protocol
      type: keyword
      description: |
        SSL protocol version.
		

The agent.yml file defines fields used by default processors. Examples: cloud.account.id, container.id, input.type

In this file, the data_stream subfields type, dataset and namespace are defined as type constant_keyword, the values for these fields are added by the integration. The event.module and event.dataset fields are defined with a fixed value specific for this integration: - event.module: apache - event.dataset: apache.access Field @timestamp is defined here as type date.

Contains ECS fields, either imported or explicitly defined.

Custom fields unique to your integration.

Learn more about fields in the general guidelines.

• Start with ECS fields: Use ECS wherever possible for compatibility and consistency.
• Be descriptive: Add clear descriptions to each field to help users and maintainers.
• Keep fields unique: Each field name should be unique within a data stream.
• Test your mappings: Use elastic-package check to validate your mappings before submitting.