Deploy Elastic Cloud Forwarder for AWS

Elastic Cloud Forwarder for AWS can be deployed using any of the following methods:

Each method achieves the same result and uses CloudFormation templates. Choose the method that best adapts to your workflow.

Use the AWS CLI to deploy the forwarder with CloudFormation templates. This method is ideal for automation and infrastructure-as-code workflows.

The following examples show how to deploy Elastic Cloud Forwarder using AWS CloudFormation CLI. Copy and paste these commands after replacing the placeholder values with your actual configuration.

• Use the --template-url flag to reference a template hosted on S3.
• Use the --region flag to specify the AWS region where the CloudFormation stack will be deployed. The CloudFormation stack deployment region must match the region of your log sources.
• To always use the most recent stable templates, use the latest path. For example, v1/latest.
• To pin a specific version, replace latest with the desired version tag. For example, v1/v{{version.edot-cf-aws}}.

Alternatively, if you have downloaded the template file, use the --template-body file://<path> option with a local template file.

aws cloudformation create-stack \
  --stack-name ecf \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/otel_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=S3Buckets,ParameterValue='"arn:aws:s3:::your-vpc-bucket,arn:aws:s3:::your-cloudtrail-bucket"' \
    ParameterKey=CloudWatchLogGroups,ParameterValue='"arn:aws:logs:eu-central-1:123456789012:log-group:vpc-flow:*"' \
    ParameterKey=LogTypes,ParameterValue='"vpcflow,cloudtrail"'
		

aws cloudformation create-stack \
  --stack-name ecf-ecs \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/ecs_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=S3Buckets,ParameterValue='"arn:aws:s3:::your-vpc-bucket,arn:aws:s3:::your-guardduty-bucket"' \
    ParameterKey=LogTypes,ParameterValue='"vpcflow,guardduty"' \
    ParameterKey=DataStreamNamespace,ParameterValue="default"
		

aws cloudformation create-stack \
  --stack-name ecf-crowdstrike-fdr \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/crowdstrike_fdr_cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=FeedClientID,ParameterValue="<placeholder>" \
    ParameterKey=FeedSecret,ParameterValue="<placeholder>" \
    ParameterKey=FeedSQSURL,ParameterValue="<placeholder>" \
    ParameterKey=FeedStorageRegion,ParameterValue="<placeholder>"
		

aws cloudformation create-stack \
  --stack-name ecf-vpc \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow"
		

aws cloudformation create-stack \
  --stack-name ecf-alb \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess"
		

aws cloudformation create-stack \
  --stack-name ecf-cloudtrail \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-cloudtrail-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="cloudtrail"
		

aws cloudformation create-stack \
  --stack-name ecf-waf \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=arn:aws:s3:::aws-waf-logs-your-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="waf"
		

aws cloudformation create-stack \
  --stack-name ecf-cw-vpc \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/cloudwatch_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=SourceCloudWatchLogGroupARN,ParameterValue="arn:aws:logs:eu-central-1:123456789012:log-group:your-log-group:*" \
    ParameterKey=EdotCloudForwarderCWLogType,ParameterValue="vpcflow"
		

aws cloudformation create-stack \
  --stack-name ecf-vpc-ecs \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs_ecs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow" \
    ParameterKey=DataStreamNamespace,ParameterValue="default"
		

To update an existing CloudFormation stack while preserving some parameter values, follow these steps:

The AWS CloudFormation console supports a one-click flow that uses the Launch Stack button, or a manual flow where you paste the template URL.

For the canonical one-click flow for each mode, refer to the Quick start on the product landing page. The Launch Stack buttons there cover OpenTelemetry mode, ECS mode, and CrowdStrike FDR.

To manually specify the template, follow these steps:

1. Navigate to CloudFormation in the AWS Console.
2. Select Create Stack and choose With new resources (standard) to start a fresh deployment.
3. Select one of the following options under Specify template:
   • Amazon S3 URL (Recommended): Paste the CloudFormation template URL from CloudFormation templates.
   • Upload a template file: Download the template from the S3 URL and upload it manually.
4. Select Next and configure all required parameters using the settings described in Configure the template.
5. Select Next again and check Acknowledge IAM capabilities. This is required because the template creates named IAM roles with permissions to access the required resources.
6. Review your configuration and select Submit to deploy the stack.
7. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

To modify parameters of an existing stack through the AWS Console:

1. Navigate to CloudFormation in the AWS Management Console.
2. Select the stack you want to update.
3. Click Update stack and select either Make a direct update or Create a change set.
4. Choose Use existing template.
5. Select Next.
6. Modify the parameter values as needed (refer to Configure the template for parameter descriptions).
7. Select Next and review your changes.
8. Select Submit to apply the updates. For a change set, select Execute changeset to apply it.
9. Monitor the stack update progress in the console.

In addition to deploying through CloudFormation templates, you can deploy Elastic Cloud Forwarder directly from the AWS Serverless Application Repository (SAR).

To deploy from SAR, follow these steps:

1. Navigate to AWS Serverless Application Repository in the AWS Management Console.
2. Select Available applications and check the box Show apps that create custom IAM roles or resource policies.
3. Search for the appropriate application:
   • OpenTelemetry mode: Search for elastic-cloud-forwarder-logs-otel.
   • ECS mode: Search for elastic-cloud-forwarder-logs-ecs. Refer to Modes for context and to ECS mode parameters for the additional template parameters.
   • CrowdStrike FDR: Search for elastic-cloud-forwarder-fdr.
4. Select the application.
5. Configure the application settings: Under Application settings, fill in the parameters described in the Configure the template section. Refer to that section for details on each parameter.
6. Acknowledge IAM role creation: At the bottom of the page, check the box to acknowledge that the application will create custom IAM roles. This is required for the forwarder to access your log source and send data to Elastic Observability.
7. Select Deploy.

The deployment process will start, and a CloudFormation stack will be created with all the necessary resources. You can monitor the progress in the AWS CloudFormation console under Stacks.

The legacy single log type applications remain available but are deprecated and will be removed in a future release: edot-cloud-forwarder-s3-logs (S3, OpenTelemetry), edot-cloud-forwarder-cloudwatch-logs (CloudWatch, OpenTelemetry), and cloud-forwarder-s3-logs-ecs (S3, ECS). For new deployments, use the unified applications listed above.

The CloudFormation templates create a number of resources to process logs from the configured sources.

The unified templates (otel_logs and ecs_logs) create the following resources. S3 and CloudWatch wiring resources are only created when the corresponding source list is non-empty.

The helper Lambda functions run only during stack create, update, and delete. They keep the S3 notifications and subscription filters in sync with the S3Buckets and CloudWatchLogGroups parameters.

The unified stacks expose the following outputs:

This is a list of resources created by the stack when collecting CrowdStrike events from a Falcon Data Replicator (FDR) feed.

The PollSchedule (with SchedulerExecutionRole) drives the polling loop on a fixed interval, while LambdaInvokeConfig and the failure bucket, S3FailureBucketARN, capture unprocessed events to prevent data loss.

This is a list of resources created by the legacy stacks when processing S3 logs.

The main Lambda function, LambdaFunction, is the core component for processing S3 logs. S3 event notifications are handled dynamically using CustomNotificationUpdater and NotificationUpdaterLambda.

CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while S3FailureBucketARN prevents data loss by capturing unprocessed logs.

This is a list of resources created by the legacy stacks when CloudWatch logs are the source.

The CloudWatch Log Subscription Filter, CloudWatchLogSubscriptionFilter, ensures logs are correctly forwarded to the Lambda function. The Lambda function, LambdaFunction, serves as the core processing unit for CloudWatch logs.

CloudWatch Log Groups help monitor execution performance and debug issues. IAM permissions (LambdaExecutionRole, LambdaPermissionCloudWatch) control interactions between CloudWatch and Lambda, while the failure bucket, S3FailureBucketARN, helps prevent data loss in case of processing errors.

If you no longer need a deployed stack and want to clean up all associated resources, you can remove it using either the AWS CLI or the AWS Console.

Deleting a stack removes all AWS resources created by that stack. However:

• If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket is not removed if it contains objects, because CloudFormation doesn't force-remove non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it.
• If you specified an existing bucket through the S3FailureBucketARN parameter, that bucket is not removed because it is not managed by the stack.

Use the following command to remove a stack:

aws cloudformation delete-stack \
  --stack-name <stack-name> \
  --region <stack-region>
		

You can monitor the deletion progress through this command:

aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --region <stack-region>
		

If the stack deletion fails and remains in a DELETE_FAILED state, you can retry the deletion with force mode:

aws cloudformation delete-stack \
  --stack-name <stack-name> \
  --region <stack-region> \
  --deletion-mode FORCE_DELETE_STACK
		

This forcibly removes the stack's resources, except any that cannot be removed, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion.

aws cloudformation delete-stack \
  --stack-name ecf \
  --region eu-central-1
		

aws cloudformation describe-stacks \
  --stack-name ecf \
  --region eu-central-1
		

To remove a stack using the AWS Management Console:

1. Navigate to CloudFormation in the AWS Management Console.
2. Select the stack you want to remove from the list.
3. Click Delete at the top of the stack details page.
4. Monitor the deletion progress on the Events tab or wait until the stack disappears from the stack list (indicating deletion is complete).

• Configuration settings: Learn about all configuration options, including optional settings and sizing recommendations.
• Troubleshooting: Diagnose and resolve issues with log forwarding.