Elastic Cloud Forwarder for AWS

Elastic Cloud Forwarder (ECF) for AWS is a Lambda function that forwards AWS logs to Elastic Cloud Managed OTLP Endpoint using a CloudFormation template. It runs serverless, with no agents to manage and no VPC plumbing. For the AWS log types and sources supported, refer to Log sources.

You can forward logs of multiple types, from multiple S3 buckets and CloudWatch log groups, by deploying a single CloudFormation stack. Each log type is routed automatically based on its S3 path pattern or its CloudWatch log group and stream.

Pick the tab that matches your log schema. Use the ECS tab if your logs need to land in the data streams used by the Elastic AWS integration. Refer to Modes for details.

Before deploying, check the Requirements for your log source and have your Elastic Managed OTLP endpoint and API key on hand.

Forwards logs from S3 buckets and CloudWatch log groups as OTel-native logs.

Launch Stack

After selecting the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example ecf.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticAPIKey API key for authentication with Elastic.
    S3Buckets Comma-separated ARNs of the S3 buckets where your logs are stored. Leave empty if forwarding only from CloudWatch.
    CloudWatchLogGroups Comma-separated ARNs of the CloudWatch Log Groups to subscribe to, each including the trailing :*. Leave empty if forwarding only from S3.
    LogTypes Log types to forward: a comma-separated list or all (default). Supported types: vpcflow, cloudtrail, elbaccess, waf, networkfirewall, s3access.

    Provide at least one of S3Buckets or CloudWatchLogGroups.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

For all available parameters, including per-type pattern overrides, refer to Unified template settings.

To get the out-of-the-box dashboards for your log types, install the OpenTelemetry Assets integrations in Kibana. Refer to Kibana integration setup.

Forwards logs from S3 buckets and CloudWatch log groups into the data streams used by the Elastic integrations.

Launch Stack

After selecting the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example ecf-ecs.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticAPIKey API key for authentication with Elastic.
    S3Buckets Comma-separated ARNs of the S3 buckets where your logs are stored. Leave empty if forwarding only from CloudWatch.
    CloudWatchLogGroups Comma-separated ARNs of the CloudWatch Log Groups to subscribe to, each including the trailing :*. Leave empty if forwarding only from S3.
    LogTypes Log types to forward: a comma-separated list or all (default). Supported types: vpcflow, cloudtrail, elbaccess, waf, guardduty, netskope_transaction, netskope_alerts_events.

    Provide at least one of S3Buckets or CloudWatchLogGroups.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

ECS mode requires the matching Elastic integration installed in Kibana — the Elastic AWS integration for AWS log sources, or the Elastic Netskope integration for Netskope Log Streaming. For all available parameters, refer to Unified template settings and ECS mode parameters.

Forwards CrowdStrike events from an FDR feed into the data stream used by the Elastic CrowdStrike integration.

Launch Stack

After selecting the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example crowdstrike-fdr.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticAPIKey API key for authentication with Elastic.
    FeedClientID Client ID of the FDR feed.
    FeedSecret Secret of the FDR feed.
    FeedSQSURL SQS URL of the FDR feed.
    FeedStorageRegion Storage region of the FDR feed.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

CrowdStrike FDR requires Elastic CrowdStrike Integration installed in Kibana.

Tip

The CloudFormation stack deployment region must match the region of your log sources (S3 buckets or CloudWatch Log Groups).

Single log type templates
Warning

The single log type templates are deprecated in version 1.10.0 and will be removed in a future release. Existing stacks continue to work and are not affected. For new deployments, use the unified templates in the Quick start.

Before version 1.8.0, Elastic Cloud Forwarder used a separate CloudFormation template per log source, and each stack handled a single log type.

S3 logs (OpenTelemetry)

Launch Stack

Required parameters: OTLPEndpoint, ElasticAPIKey, SourceS3BucketARN, and EdotCloudForwarderS3LogsType (vpcflow, elbaccess, cloudtrail, or waf).

CloudWatch logs (OpenTelemetry)

Launch Stack

Required parameters: OTLPEndpoint, ElasticAPIKey, SourceCloudWatchLogGroupARN (including the trailing :*), and EdotCloudForwarderCWLogType (vpcflow or cloudtrail).

S3 logs (ECS)

Launch Stack

Required parameters: OTLPEndpoint, ElasticAPIKey, SourceS3BucketARN, and EdotCloudForwarderS3LogsType (vpcflow, elbaccess, cloudtrail, waf, guardduty, netskope_transaction, or netskope_alerts_events).

For the parameters of the legacy templates, refer to Legacy template settings.

Before deploying Elastic Cloud Forwarder for AWS, confirm you have the following for your log source:

Via S3

To collect VPC Flow logs from S3, you need:

  • A Virtual Private Cloud (VPC).
  • An S3 bucket for storing flow logs.
  • A flow log configured with the S3 bucket as the destination.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via CloudWatch

To collect VPC Flow logs from CloudWatch, you need:

  • A Virtual Private Cloud (VPC) with flow logs delivered to a CloudWatch Log Group.
  • The Log Group ARN (must include the trailing :*).
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect Elastic Load Balancer (ELB) Access logs, you need:

  • An ELB of any type (ALB, NLB, CLB).
  • An S3 bucket to store the access logs.
  • Access logging enabled, with the bucket as the destination.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect CloudTrail logs from S3, you need:

  • A trail that delivers account events as log files to an Amazon S3 bucket.
  • An S3 bucket to store the trail logs.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via CloudWatch

To collect CloudTrail logs from CloudWatch, you need:

  • CloudTrail configured to deliver logs to a CloudWatch Log Group.
  • The Log Group ARN (must include the trailing :*).
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect AWS WAF logs, you need:

  • AWS WAF with logging enabled to an S3 bucket.
  • An S3 bucket to store the WAF logs.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect AWS Network Firewall logs, you need:

  • AWS Network Firewall with logging enabled to an S3 bucket.
  • An S3 bucket to store the firewall logs.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect S3 server access logs, you need:

  • Server access logging enabled on the source bucket, delivering to a logging bucket.
  • The delivery prefix used by your logging configuration. S3 server access logs have no standard delivery prefix, so you must set it through the S3PatternOverrides parameter, for example s3access=my-logs-prefix/*.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect GuardDuty findings, you need:

  • Amazon GuardDuty enabled with findings exported to an S3 bucket.
  • An S3 bucket to store the GuardDuty findings.
  • If the bucket is encrypted with a customer-managed KMS key, the key ARN to pass to GuardDutyKMSKeyARNs (or GuardDutyKMSKeyARN on the legacy template).
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Elastic Cloud Forwarder currently supports collecting CrowdStrike events sourced from CrowdStrike managed S3 buckets.

To collect CrowdStrike events, you need:

  • The FDR feature enabled for your instance. This needs to be done by CrowdStrike support.
  • A feed to export events from CrowdStrike, which you can create after FDR is enabled.
  • Feed client ID, secret, feed SQS URL along with the storage region of the feed you created.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Elastic Cloud Forwarder supports collecting Netskope Log Streaming data delivered to S3. Two log types are available: netskope_transaction (Transaction Events) and netskope_alerts_events (combined Alerts V2 and Events V2). With the unified ECS template, one stack can collect both types; with the legacy s3_logs_ecs template, each stack handles a single type selected with EdotCloudForwarderS3LogsType.

To collect Netskope Log Streaming data, you need:

  • A Netskope Log Streaming configuration delivering to an S3 bucket, with GZIP compression enabled.
  • The delivery prefix of each Netskope signal. Netskope has no standard delivery prefix, so with the unified template you must set each type's prefix through S3PatternOverrides, for example netskope_transaction=txn/*,netskope_alerts_events=alerts/*. With the legacy template, point each stack at a bucket that receives a single Netskope signal.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

If you plan to use ECS mode, install the matching Elastic integration in Kibana so its ingest pipelines and index templates are available:

Refer to Modes for more details.

To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address and API key, follow these steps:

  1. In Elastic Cloud, create an Observability project or open an existing one.
  2. Go to Add data, select Applications and then select OpenTelemetry.
  3. Copy the endpoint and authentication headers values.

Alternatively, you can retrieve the endpoint from the Manage project page and create an API key manually from the API keys page.

  1. Log in to the Elastic Cloud Console.
  2. Find your deployment on the home page or on the Hosted deployments page, and then select Manage.
  3. In the Application endpoints, cluster and component IDs section, select Managed OTLP.
  4. Copy the public endpoint value.
  5. Create an API key in Kibana: Go to Stack Management → API keys and select Create API key.
Important

Trim the API key from Authorization=ApiKey MYKEYVALUE... to just MYKEYVALUE... before using it as the argument to the ElasticAPIKey parameter.

Elastic Cloud Forwarder for AWS supports the following log sources:

AWS log type Description S3 (OTel) S3 (ECS) CloudWatch (OTel) CloudWatch (ECS)
VPC Flow Logs VPC Flow Logs to capture information about IP traffic.
ELB Access Logs Access logs for your Application Load Balancer. Not available Not available
CloudTrail Logs CloudTrail Logs to record account activity.
WAF Logs WAF Logs to capture web request details for security analysis. Not available Not available
Network Firewall Logs Network Firewall logs for traffic inspection. Not available Not available Not available
S3 Server Access Logs S3 server access logs recording requests made to a bucket. Not available Not available Not available
GuardDuty findings GuardDuty findings exported to S3 for threat detection. Not available Not available Not available
CrowdStrike Falcon Data Replicator (FDR) Falcon Data Replicator(FDR) events for threat detection Not available Not available Not available
Netskope Transaction Events Netskope Log Streaming transaction (web proxy) events. Not available Not available Not available
Netskope Alerts and Events V2 Netskope Log Streaming combined Alerts V2 and Events V2 feed. Not available Not available Not available

Elastic Cloud Forwarder for AWS runs in one of two modes. Each mode has its own unified CloudFormation template, covering both S3 and CloudWatch sources.

Mode Log format Log source
OpenTelemetry OTel native S3 and CloudWatch
ECS ECS S3 only

S3 and CloudWatch

OpenTelemetry mode emits OTel-native logs compatible with the OTel content packs, which you can install separately.

ECS mode emits ECS-formatted logs into the data streams used by the Elastic AWS integration, so its packaged dashboards light up. For ECS-specific template parameters, refer to ECS mode parameters.

Note

With the legacy single log type templates, ECS mode supports S3 sources only.

Before deploying Elastic Cloud Forwarder for AWS, consider the following:

  • A single stack forwards multiple log types from multiple S3 buckets and CloudWatch Log Groups. Log types are identified by their S3 delivery path pattern and by CloudWatch log group or stream patterns.
  • S3 sources: A bucket can hold more than one log type, as long as objects follow the standard AWS delivery prefixes or you configure per-type prefixes with S3PatternOverrides. For example, if your CloudTrail logs are delivered under a custom prefix, set S3PatternOverrides to cloudtrail=my-org/audit/cloudtrail/*. Log types without a standard delivery prefix (S3 server access logs, Netskope) always require S3PatternOverrides, for example s3access=my-logs-prefix/*.
  • CloudWatch sources: The stack creates a subscription filter on every configured Log Group. Log types are matched by log group and stream patterns, which you can adjust per type with CloudWatchLogGroupOverrides and CloudWatchLogStreamOverrides, for example vpcflow=eni-*.
  • OpenTelemetry and ECS mode use separate templates. Deploy one stack per mode if you need both.
  • With the legacy single log type templates, each stack handles one log type from one source, and logs of different types must live in separate buckets.

For the override parameter formats and more examples, refer to Unified template settings.

For log types that support both S3 and CloudWatch (VPC Flow Logs and CloudTrail), the choice depends on your delivery requirements and existing setup:

  • Use S3 if your logs are already delivered to S3, or if periodic batch delivery (every ~5 minutes) is acceptable for your use case. S3 is the most cost-effective option for log delivery.
  • Use CloudWatch if you need near-real-time log delivery, or if your logs are already published to a CloudWatch Log Group. Note that CloudWatch log delivery costs are significantly higher than S3.

Both sources produce identical data in Elastic: the same data streams, field mappings, and dashboards apply regardless of the source.

Logs collected by Elastic Cloud Forwarder for AWS are stored in Elasticsearch data streams. The dataset name depends on the mode you deploy.

In OpenTelemetry mode, logs land in OTel-native data streams:

AWS log type Log source Data stream dataset Description
VPC Flow Logs S3 or CloudWatch aws.vpcflow.otel VPC Flow Log records
ELB Access Logs S3 aws.elbaccess.otel ELB Access Log records (ALB, NLB, CLB)
CloudTrail Logs S3 or CloudWatch aws.cloudtrail.otel CloudTrail account activity records
WAF Logs S3 aws.waf.otel AWS WAF web request log records
Network Firewall Logs S3 aws.networkfirewall.otel AWS Network Firewall log records
S3 Server Access Logs S3 aws.s3access.otel S3 server access log records

Both S3 and CloudWatch sources produce the same data stream format for the same log type.

CloudWatch logs that match no configured log type are forwarded as generic CloudWatch records to logs-aws.cloudwatch.otel-default, or to the Elastic wired stream logs.otel when RouteUnmatchedToWiredStream is enabled. Refer to Unified template settings.

For detailed field mappings, refer to the following documentation:

In ECS mode, logs land in the ECS-formatted data streams used by the Elastic AWS integration, in the format logs-aws.<type>-<namespace>:

AWS log type Data stream
VPC Flow Logs logs-aws.vpcflow-default
ELB Access Logs logs-aws.elb.access-default
CloudTrail Logs logs-aws.cloudtrail-default
WAF Logs logs-aws.waf-default
GuardDuty findings logs-aws.guardduty-default
Netskope Transaction Events logs-netskope.transaction-default
Netskope Alerts and Events V2 logs-netskope.alerts_v2-default and logs-netskope.events_v2-default

The namespace component matches the value of the DataStreamNamespace parameter. Refer to ECS mode parameters for details.

CloudWatch logs that match no configured log type land in logs-aws.cloudwatch_logs-<namespace>.

Netskope log types land in the Netskope integration's logs-netskope.* data streams rather than logs-aws.*. The combined Alerts and Events V2 feed is routed to the alerts_v2 and events_v2 data streams by content.

After Elastic Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.

To set up data visualization in Kibana:

  1. Log into your Elastic Cloud deployment and open Kibana.

  2. Go to Management → Integrations in the Kibana navigation menu.

  3. Search for the appropriate integration based on your mode and log type, and install it:

    OpenTelemetry mode:

    AWS log type Integration name Description
    ELB Access Logs AWS ELB OpenTelemetry Assets Dashboards and visualizations for Elastic Load Balancer logs
    VPC Flow Logs AWS VPC Flow Logs OpenTelemetry Assets Dashboards and visualizations for VPC flow log data
    CloudTrail Logs AWS CloudTrail Logs OpenTelemetry Assets Dashboards and visualizations for CloudTrail log data
    WAF Logs AWS WAF Logs OpenTelemetry Assets Dashboards and visualizations for WAF log data

    ECS mode: Install the Elastic AWS integration to pick up the ingest pipelines, index templates, and dashboards for all supported AWS log types. For Netskope log types, install the Elastic Netskope integration instead.

  4. Once installed, navigate to Dashboard to view the pre-built dashboards for your AWS log data.

Elastic Cloud Forwarder for AWS has the following limitations:

Limitation Description
VPC/PrivateLink not supported Elastic Cloud Forwarder cannot be deployed inside a VPC or use AWS PrivateLink endpoints. The Lambda function requires public internet access to forward data to the OTLP endpoint.
Managed OTLP Input only Elastic Cloud Forwarder is tested exclusively with Elastic Cloud Managed OTLP Endpoint. Forwarding to a self-deployed EDOT Collector Gateway is not tested and forwarding to APM Server is not supported.
One mode per stack OpenTelemetry and ECS mode use separate templates. Deploy one stack per mode if you need both.
Same region as sources The stack must be deployed in the same region as its S3 buckets and CloudWatch Log Groups.

The legacy single log type templates have additional limitations:

Limitation Description
Single log type per stack Each legacy stack processes one log type from one source. Logs of different types must live in separate S3 buckets.
Single Log Group per CloudWatch stack Each legacy CloudWatch stack subscribes to one Log Group.
No CloudWatch in ECS mode The legacy ECS template forwards S3 logs only. Use the unified ECS template for CloudWatch sources.
  • Configure the template: Learn about all configuration options, including optional settings, ECS mode parameters, and sizing recommendations.
  • Deployment methods: Explore alternative deployment methods using AWS CLI or AWS Serverless Application Repository.
  • Troubleshooting: Diagnose and resolve issues with log forwarding.