Configure Elastic Cloud Forwarder for AWS
Before deploying Elastic Cloud Forwarder for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses the following settings to deploy and configure the forwarder Lambda function.
The CloudFormation templates are hosted in a public Amazon S3 bucket and are accessible through HTTPS URL. You can reference these templates directly during deployment or download them for local use.
| Log type | Log source | Mode | CloudFormation template |
|---|---|---|---|
| VPC, CloudTrail, ELB, WAF, Network Firewall, S3 Server Access | S3 and CloudWatch | OpenTelemetry | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/otel_logs-cloudformation.yaml |
| VPC, CloudTrail, ELB, WAF, GuardDuty, Netskope | S3 and CloudWatch | ECS | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/ecs_logs-cloudformation.yaml |
| CrowdStrike Falcon Data Replicator (FDR) only | CrowdStrike S3 (sourced through SQS) | ECS | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/crowdstrike_fdr_cloudformation.yaml |
| Log type | Log source | Mode | CloudFormation template |
|---|---|---|---|
| VPC, ELB, CloudTrail, WAF | S3 | OpenTelemetry | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml |
| VPC, ELB, CloudTrail, WAF, GuardDuty, Netskope | S3 | ECS | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs_ecs-cloudformation.yaml |
| VPC, CloudTrail | CloudWatch | OpenTelemetry | https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/cloudwatch_logs-cloudformation.yaml |
For specific versions, edit latest in the URL to the required version in the format vX.Y.Z.
These are the required settings for every template:
| Setting | Description |
|---|---|
stack-name |
Name of the CloudFormation stack, for example, ecf.Do not use the same name for different stacks. |
OTLPEndpoint |
The OTLP endpoint URL used for data ingestion, obtained from Elastic Cloud Serverless or Elastic Cloud Hosted. |
ElasticAPIKey |
API key for authentication with Elastic, obtained from Elastic Cloud Serverless or Elastic Cloud Hosted. |
The unified templates (otel_logs-cloudformation.yaml and ecs_logs-cloudformation.yaml) share the following source and log type settings:
| Setting | Description |
|---|---|
S3Buckets |
Comma-separated ARNs of S3 buckets to forward logs from, for example arn:aws:s3:::flow-logs-prod,arn:aws:s3:::cloudtrail-prod. Leave empty if forwarding only from CloudWatch. All buckets must be in the same region as the stack. |
CloudWatchLogGroups |
Comma-separated ARNs of CloudWatch Log Groups to forward logs from. A subscription filter is created on each one. Each ARN must include the trailing :*. Leave empty if forwarding only from S3. All Log Groups must be in the same region as the stack. |
LogTypes |
Comma-separated list of log types to forward, or all (default) to forward every supported type. OpenTelemetry template: vpcflow, cloudtrail, elbaccess, waf, networkfirewall, s3access. ECS template: vpcflow, cloudtrail, elbaccess, waf, guardduty, netskope_transaction, netskope_alerts_events. vpcflow and cloudtrail are available from both S3 and CloudWatch; the rest are S3 only. |
Provide at least one of the S3 buckets or CloudWatch Log Groups parameters. Log types are identified by their S3 delivery path pattern and by CloudWatch log group or stream patterns, which you can adjust per type:
| Setting | Description |
|---|---|
S3PatternOverrides |
Optional. Override the default S3 path pattern per log type, as type=pattern entries, for example cloudtrail=my-org/audit/cloudtrail/*. Required for log types with no standard delivery prefix: s3access and the netskope_* types. Each overridden type must also be present in LogTypes. |
CloudWatchLogGroupOverrides |
Optional. Override the default log group pattern per log type, as type=pattern entries, for example cloudtrail=my-log-group-*. Log group overrides take precedence over stream overrides. |
CloudWatchLogStreamOverrides |
Optional. Override the default log stream pattern per log type, as type=pattern entries, for example vpcflow=eni-*. |
The OpenTelemetry template additionally supports:
| Setting | Description |
|---|---|
RouteUnmatchedToWiredStream |
CloudWatch only. When true, CloudWatch logs that match no configured log type are forwarded to the Elastic logs.otel wired stream, which must be enabled in your Elastic deployment. Default is false, which leaves unmatched logs in the logs-aws.cloudwatch.otel-default data stream. |
Both unified templates also accept an optional failure bucket parameter:
| Setting | Description |
|---|---|
S3FailureBucketARN |
Optional. ARN of an existing S3 bucket to store failed Lambda invocations. If empty, the stack creates a dedicated failure bucket automatically. |
The unified ECS template (ecs_logs-cloudformation.yaml) uses the Required settings and Unified template settings, with the following additions:
| Setting | Description |
|---|---|
DataStreamNamespace |
The namespace component of the data stream name, for example logs-aws.vpcflow-<namespace>. Use different namespaces to separate data from different environments or teams. Default is default. |
PreserveOriginalEvent |
When set to true, preserves a raw copy of the original event in the event.original field. Useful for reindexing or debugging, but increases storage usage. Default is false. |
PreserveDuplicateCustomFields
|
When set to true, keeps the integration's custom fields that were copied to their ECS counterparts. Some ingest pipelines (for example, Netskope) drop these duplicates unless PreserveDuplicateCustomFields is enabled, and there are dashboards that rely on these fields. Increases storage usage. Default is false. |
DataSetOverrides |
Optional. Override the data stream dataset per log type, as type=dataset entries, for example cloudtrail=audit-prod. Types not listed use the integration's default dataset. |
NamespaceOverrides |
Optional. Override DataStreamNamespace per log type, as type=namespace entries, for example cloudtrail=audit-prod,vpcflow=network-prod. |
PreserveOriginalOverrides |
Optional. Override PreserveOriginalEvent per log type, as type=true\|false entries. |
PreserveDuplicateOverrides |
Optional. Override PreserveDuplicateCustomFields per log type, as type=true\|false entries. |
GuardDutyKMSKeyARNs |
For GuardDuty findings exported with customer-managed KMS keys, the comma-separated key ARNs the Lambda is allowed to decrypt with. Leave empty if not using GuardDuty or if using the default AWS-managed key. |
NetskopeTransactionCsvSeparator
|
Field separator for the Netskope Transaction Events CSV, wrapped in double quotes ("," for a comma or " " for a single space). Only used when netskope_transaction is enabled. Default is ",". |
NetskopeAlertsEventsCsvSeparator
|
Field separator for the Netskope Alerts/Events V2 CSV, wrapped in double quotes ("," for a comma or " " for a single space). Only used when netskope_alerts_events is enabled. Default is ",". |
For context on when to use ECS mode, refer to Modes.
For events sourced from CrowdStrike FDR (crowdstrike_fdr_cloudformation.yaml), use the following settings:
| Setting | Description |
|---|---|
FeedClientID |
Client ID of the FDR feed. |
FeedSecret |
Secret of the FDR feed. |
FeedSQSURL |
SQS URL of the FDR feed. |
FeedStorageRegion |
Storage region of the FDR feed. |
Optional parameters for CrowdStrike FDR:
| Setting | Description | Default |
|---|---|---|
PreserveOriginalEvent |
Preserves a raw copy of the original event in event.original. | false |
EnableObserverGeoIP |
Enable GeoIP enrichment for observer.ip | true |
EnableSourceGeoIP |
Enable GeoIP enrichment for source.ip | true |
EnablDestinationGeoIP |
Enable GeoIP enrichment for destination.ip | true |
These are optional settings you can set in any template.
On the unified and CrowdStrike FDR templates, available in versions 1.8.0 and later, the parameters are prefixed ElasticCloudForwarder.
On the legacy single log type templates — the only templates available before version 1.8.0 — the parameters are prefixed EdotCloudForwarder, for example EdotCloudForwarderMemorySize.
| Setting | Description |
|---|---|
ElasticCloudForwarderConcurrentExecutions |
Sets the maximum number of reserved concurrent executions for the Lambda function. Default value is 20 for the unified templates and the legacy CloudWatch template, and 5 for the legacy S3 templates. CloudWatch sources require significantly higher concurrency than S3 for the same data rate (see CloudWatch Logs sizing). If you notice Lambda throttling, increase concurrent executions. Make sure this value doesn't exceed your AWS account's concurrency limit. |
ElasticCloudForwarderMemorySize |
Sets the allocated memory for the Lambda function, measured in megabytes. The default value is 512 MB. Minimum value is 128 MB. Maximum value is 10240 MB. For S3 sources, 512 MB is recommended. For CloudWatch-only stacks, 256 MB is sufficient — the smaller payload sizes require less memory. More memory increases Lambda CPU allocation, increasing processing speed of events. |
ElasticCloudForwarderTimeout |
Maximum execution time for the Lambda function, measured in seconds. The default is set to 900 seconds or 15 minutes. Accepts values from 1 second to 900 seconds. |
ElasticCloudForwarderVersion |
Version of the forwarder Lambda layer. Expected format is semantic versioning, for example 1.0.0. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. |
EdotCloudForwarderVersion
|
CrowdStrike FDR template only. Former name of ElasticCloudForwarderVersion, renamed in version 1.10.0. If you pinned the layer version under the old name, set it again as ElasticCloudForwarderVersion when updating the stack. On the legacy single log type templates, all parameters keep the EdotCloudForwarder prefix. |
ElasticCloudForwarderExporterMaxQueueSize |
Sets the internal OTLP exporter queue size, measured in bytes. The default value is 75000000 (75 MB) for the unified templates and 50000000 (50 MB) for the legacy templates. This parameter should be used only in exceptional edge cases that require manual tuning of the export queue. |
ElasticCloudForwarderLogLevel |
Sets the log level for the forwarder collector. Accepted values: debug, info, warn, error. Default value is info. Set to debug when troubleshooting to produce detailed logs in the Lambda CloudWatch log group. |
The default values provided have been determined through extensive load testing across different log types and data volumes. For most use cases, these defaults provide a good balance between cost and performance.
Adjust these parameters only if you notice performance issues such as Lambda timeouts, throttling, high memory usage or dropped data. If you need assistance tuning these parameters for your specific workload, refer to Contact support.
The legacy single log type templates are deprecated in version 1.10.0 and will be removed in a future release. Existing stacks continue to work and are not affected. For new deployments, use the unified templates.
The legacy single log type templates handle one log type from one source per stack. Set the following settings based on the log source:
For logs sourced from S3, use the following settings:
| Setting | Description |
|---|---|
EdotCloudForwarderS3LogsType |
The encoding format for logs in the S3 bucket. Supported options on the OpenTelemetry template (s3_logs-cloudformation.yaml):- vpcflow: VPC Flow Logs- elbaccess: ELB Access logs- cloudtrail: CloudTrail Logs- waf: WAF LogsThe ECS template ( s3_logs_ecs-cloudformation.yaml) additionally accepts guardduty, netskope_transaction, and netskope_alerts_events. |
SourceS3BucketARN |
Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the forwarder Lambda function automatically. |
The ECS template also accepts DataStreamNamespace, PreserveOriginalEvent, PreserveDuplicateCustomFields, GuardDutyKMSKeyARN, NetskopeTransactionCsvSeparator, and NetskopeAlertsEventsCsvSeparator, with the same meaning as in ECS mode parameters.
For logs sourced from CloudWatch, use the following settings:
| Setting | Description |
|---|---|
SourceCloudWatchLogGroupARN |
Amazon Resource Name (ARN) of the CloudWatch Log Group where the subscription filter will be created. The ARN must include the trailing :*. |
EdotCloudForwarderCWLogType |
The type of logs in the CloudWatch Log Group. Supported options: - vpcflow: VPC Flow Logs- cloudtrail: CloudTrail Logs |
The log group must already exist in your AWS account and region. If the ARN points to a non-existent log group, stack deployment or updates will fail.
Use the following sizing suggestions to select appropriate reserved concurrency (ElasticCloudForwarderConcurrentExecutions) and Lambda memory (ElasticCloudForwarderMemorySize) based on your expected traffic volumes. This helps maximize performance and prevent Lambda throttling at high log volumes.
The recommendations are per log type. For a unified stack forwarding several log types, reserved concurrency is shared across all sources, so size for the sum of your expected volumes.
Monitor CloudWatch metrics for Lambda throttling and concurrent executions, as well as CloudWatch Logs for execution duration per Lambda invocation.
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| < 5 MB/s | < 50,000 logs/s | 5 | 512 MB | Default configuration |
| 5 - 10 MB/s | 50,000 - 100,000 logs/s | 10 | 512 MB | Increase concurrency |
| > 10 MB/s | > 100,000 logs/s | > 10 | 512 MB | First increase concurrency and then increase memory as needed |
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| < 10 MB/s | < 25,000 events/s | 5 | 512 MB | Default configuration |
| 10 - 40 MB/s | 25,000 - 100,000 events/s | 20 | 512 MB | Increase concurrency |
| > 40 MB/s | > 100,000 events/s | > 20 | 512 MB | First increase concurrency and then increase memory as needed |
ELB logs might produce files with gigabytes of data. The default configurations are confirmed to work up to 3GB log files, which translates roughly to 23,000 requests per second per ELB. If you expect higher request volumes, increase Lambda memory allocation.
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| ≤ 10 MB/s | < 10,000 events/s | 5 | 512 MB | Default configuration |
| 10 - 30 MB/s | 10,000 - 30,000 events/s | 10 | 512 MB | Increase concurrency |
| > 30 MB/s | > 30,000 events/s | 15+ | 512 MB | Scale concurrency as needed |
| Throughput | Log rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| ≤ 25 MB/s | < 35,000 events/s | 5 | 512 MB | Default configuration |
| > 25 MB/s | > 35,000 events/s | 10 | 512 MB | Increase concurrency |
CloudWatch delivers logs differently from S3: instead of large files, it sends many small batches that each trigger a separate Lambda invocation. This results in significantly higher concurrency requirements for the same data rate.
| Throughput | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|
| < 1 MB/s | 20 | 256 MB | Default configuration |
| 1 - 8 MB/s | 100 | 256 MB | Increase concurrency |
| > 8 MB/s | 200 | 256 MB | Scale concurrency as needed |
These recommendations apply to both VPC Flow Logs and CloudTrail logs delivered through CloudWatch. The 256 MB memory recommendation applies to CloudWatch-only stacks. For a unified stack that also forwards from S3, keep 512 MB.
| Throughput | Event rate | Recommended concurrency | Recommended memory | Note |
|---|---|---|---|---|
| ≤ 40 MB/s | < 50,000 events/s | 10 | 512 MB | Default configuration |
| > 40 MB/s | > 50,000 events/s | 10+ | 512 MB | Increase concurrency |
- Deployment methods: Deploy using AWS CLI, AWS Console, or AWS Serverless Application Repository.
- Troubleshooting: Diagnose and resolve issues with log forwarding.