Loading

EDOT Cloud Forwarder for AWS

EDOT Cloud Forwarder (CF) for AWS is a Lambda function that forwards AWS logs to Elastic Cloud Managed OTLP Endpoint using a CloudFormation template. It runs serverless, with no agents to manage and no VPC plumbing. For the AWS log types and sources supported, refer to Log sources.

Pick the tab that matches your log source and schema. Use the ECS tab if your logs need to land in the data streams used by the Elastic AWS integration. Refer to Modes for details.

Before deploying, check the Requirements for your log source and have your Elastic Managed OTLP endpoint and API key on hand.

Launch Stack

After selecting the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example vpc-edot-cf.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticApiKey API key for authentication with Elastic.
    SourceS3BucketARN ARN of the S3 bucket where your logs are stored.
    EdotCloudForwarderS3LogsType The log type: vpcflow, elbaccess, cloudtrail, or waf.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

Launch Stack

After selecting the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example cw-vpc-edot-cf.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticApiKey API key for authentication with Elastic.
    SourceCloudWatchLogGroupARN ARN of the CloudWatch Log Group where your logs are delivered, including the trailing :*.
    EdotCloudForwarderCWLogType The log type: vpcflow or cloudtrail.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

Launch Stack

After selecting the button:

  1. Configure the required parameters:

    Parameter Description
    Stack name Name of the CloudFormation stack, for example vpc-edot-cf-ecs.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticApiKey API key for authentication with Elastic.
    SourceS3BucketARN ARN of the S3 bucket where your logs are stored.
    EdotCloudForwarderS3LogsType The log type: vpcflow, elbaccess, cloudtrail, waf, or guardduty.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

ECS mode requires the Elastic AWS integration installed in Kibana. For the namespace, original-event, and GuardDuty KMS parameters, refer to ECS mode parameters.

Tip

The CloudFormation stack deployment region must match the region of your log source (S3 bucket or CloudWatch Log Group).

Before deploying EDOT Cloud Forwarder for AWS, confirm you have the following for your log source:

Via S3

To collect VPC Flow logs from S3, you need:

  • A Virtual Private Cloud (VPC).
  • An S3 bucket for storing flow logs.
  • A flow log configured with the S3 bucket as the destination.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via CloudWatch

To collect VPC Flow logs from CloudWatch, you need:

  • A Virtual Private Cloud (VPC) with flow logs delivered to a CloudWatch Log Group.
  • The Log Group ARN (must include the trailing :*).
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect Elastic Load Balancer (ELB) Access logs, you need:

  • An ELB of any type (ALB, NLB, CLB).
  • An S3 bucket to store the access logs.
  • Access logging enabled, with the bucket as the destination.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect CloudTrail logs from S3, you need:

  • A trail that delivers account events as log files to an Amazon S3 bucket.
  • An S3 bucket to store the trail logs.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via CloudWatch

To collect CloudTrail logs from CloudWatch, you need:

  • CloudTrail configured to deliver logs to a CloudWatch Log Group.
  • The Log Group ARN (must include the trailing :*).
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect AWS WAF logs, you need:

  • AWS WAF with logging enabled to an S3 bucket.
  • An S3 bucket to store the WAF logs.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

Via S3

To collect GuardDuty findings, you need:

  • Amazon GuardDuty enabled with findings exported to an S3 bucket.
  • An S3 bucket to store the GuardDuty findings.
  • If the bucket is encrypted with a customer-managed KMS key, the key ARN to pass to GuardDutyKMSKeyARN.
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

If you plan to use ECS mode, also install the Elastic AWS integration in Kibana so its ingest pipelines and index templates are available. Refer to Modes for details.

To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address and API key, follow these steps:

  1. In Elastic Cloud, create an Observability project or open an existing one.
  2. Go to Add data, select Applications and then select OpenTelemetry.
  3. Copy the endpoint and authentication headers values.

Alternatively, you can retrieve the endpoint from the Manage project page and create an API key manually from the API keys page.

  1. Log in to the Elastic Cloud Console.
  2. Find your deployment on the home page or on the Hosted deployments page, and then select Manage.
  3. In the Application endpoints, cluster and component IDs section, select Managed OTLP.
  4. Copy the public endpoint value.
Important

Trim the API key from Authorization=ApiKey MYKEYVALUE... to just MYKEYVALUE... before using it as the argument to the ElasticAPIKey parameter.

EDOT Cloud Forwarder for AWS supports the following log sources:

AWS log type Description S3 (OTel) S3 (ECS) CloudWatch (OTel)
VPC Flow Logs VPC Flow Logs to capture information about IP traffic.
ELB Access Logs Access logs for your Application Load Balancer. N/A
CloudTrail Logs CloudTrail Logs to record account activity.
WAF Logs WAF Logs to capture web request details for security analysis. Not yet available
GuardDuty findings GuardDuty findings exported to S3 for threat detection. Not yet available Not yet available

EDOT Cloud Forwarder for AWS runs in one of two modes.

Mode Log format Log source
OpenTelemetry OTel native S3 and CloudWatch
ECS ECS S3 only

OpenTelemetry mode emits OTel-native logs compatible with the OTel content packs, which you can install separately.

ECS mode emits ECS-formatted logs into the data streams used by the Elastic AWS integration, so its packaged dashboards light up. ECS mode supports S3 only. For ECS-specific template parameters, refer to ECS mode parameters.

Before deploying EDOT Cloud Forwarder for AWS, consider the following:

  • Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
  • S3 sources: Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
  • CloudWatch sources: Each CloudFormation stack subscribes to a single CloudWatch Log Group. Deploy a separate stack for each Log Group you want to forward.
  • S3 and CloudWatch stacks are independent — you can deploy both to collect the same log type from different sources.

For log types that support both S3 and CloudWatch (VPC Flow Logs and CloudTrail), the choice depends on your delivery requirements and existing setup:

  • Use S3 if your logs are already delivered to S3, or if periodic batch delivery (every ~5 minutes) is acceptable for your use case. S3 is the most cost-effective option for log delivery.
  • Use CloudWatch if you need near-real-time log delivery, or if your logs are already published to a CloudWatch Log Group. Note that CloudWatch log delivery costs are significantly higher than S3.

Both sources produce identical data in Elastic: the same data streams, field mappings, and dashboards apply regardless of the source.

Logs collected by EDOT Cloud Forwarder for AWS are stored in Elasticsearch data streams. The dataset name depends on the mode you deploy.

In OpenTelemetry mode, logs land in OTel-native data streams:

AWS log type Log source Data stream dataset Description
VPC Flow Logs S3 or CloudWatch aws.vpcflow.otel VPC Flow Log records
ELB Access Logs S3 aws.elbaccess.otel ELB Access Log records (ALB, NLB, CLB)
CloudTrail Logs S3 or CloudWatch aws.cloudtrail.otel CloudTrail account activity records
WAF Logs S3 aws.waf.otel AWS WAF web request log records

Both S3 and CloudWatch sources produce the same data stream format for the same log type.

For detailed field mappings, refer to the following documentation:

In ECS mode, logs land in the ECS-formatted data streams used by the Elastic AWS integration, in the format logs-aws.<type>-<namespace>:

AWS log type Data stream
VPC Flow Logs logs-aws.vpcflow-default
ELB Access Logs logs-aws.elb.access-default
CloudTrail Logs logs-aws.cloudtrail-default
WAF Logs logs-aws.waf-default
GuardDuty findings logs-aws.guardduty-default

The namespace component matches the value of the DataStreamNamespace parameter. Refer to ECS mode parameters for details.

After EDOT Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.

To set up data visualization in Kibana:

  1. Log into your Elastic Cloud deployment and open Kibana.

  2. Go to Management → Integrations in the Kibana navigation menu.

  3. Search for the appropriate integration based on your mode and log type, and install it:

    OpenTelemetry mode:

    AWS log type Integration name Description
    ELB Access Logs AWS ELB OpenTelemetry Assets Dashboards and visualizations for Elastic Load Balancer logs
    VPC Flow Logs AWS VPC Flow Logs OpenTelemetry Assets Dashboards and visualizations for VPC flow log data
    CloudTrail Logs AWS CloudTrail Logs OpenTelemetry Assets Dashboards and visualizations for CloudTrail log data
    WAF Logs AWS WAF Logs OpenTelemetry Assets Dashboards and visualizations for WAF log data

    ECS mode: Install the Elastic AWS integration to pick up the ingest pipelines, index templates, and dashboards for all supported AWS log types.

  4. Once installed, navigate to Dashboard to view the pre-built dashboards for your AWS log data.

EDOT Cloud Forwarder for AWS has the following limitations:

Limitation Description
VPC/PrivateLink not supported EDOT Cloud Forwarder cannot be deployed inside a VPC or use AWS PrivateLink endpoints. The Lambda function requires public internet access to forward data to the OTLP endpoint.
Managed OTLP Input only EDOT Cloud Forwarder is tested exclusively with Elastic Cloud Managed OTLP Endpoint. Forwarding to a self-deployed EDOT Collector Gateway is not tested and forwarding to APM Server is not supported.
Single log type per S3 bucket Each S3 bucket can only contain one log type. Mixed log formats in the same bucket are not supported yet.
Single Log Group per CloudWatch stack Each CloudWatch stack subscribes to one Log Group. Deploy a separate stack for each Log Group you want to forward.
No CloudWatch in ECS mode ECS mode forwards S3 logs only. Use OpenTelemetry mode for CloudWatch sources.
  • Configure the template: Learn about all configuration options, including optional settings, ECS mode parameters, and sizing recommendations.
  • Deployment methods: Explore alternative deployment methods using AWS CLI or AWS Serverless Application Repository.
  • Troubleshooting: Diagnose and resolve issues with log forwarding.