Troubleshooting Elastic Cloud Forwarder for AWS

This page helps you diagnose and resolve issues with Elastic Cloud Forwarder for AWS when logs are not being forwarded to Elasticsearch as expected.

Use CloudWatch Metrics Explorer to monitor your Elastic Cloud Forwarder Lambda function:

Metric Expected behavior
Duration S3: Increases with file size (2-5s typical). CloudWatch: Stays ~0.5s regardless of throughput.
ConcurrentExecutions Should not consistently hit the configured limit. CloudWatch sources use significantly higher concurrency than S3.
Errors Should be 0.
Throttles Should be 0. This is especially critical for CloudWatch sources, where throttling happens immediately if concurrency is undersized.

The LambdaLogGroup resource created by the CloudFormation stack stores all Lambda execution logs. Check these logs for processing errors, configuration issues, or data export failures.

Lambda execution times out before completing. Check file sizes and execution duration in CloudWatch metrics.

The default 15-minute timeout handles all typical scenarios. For large files (multiple gigabytes), increase memory to allocate more CPU for faster processing.

  • Ingestion lag despite fast individual executions.
  • Throttles metric showing non-zero values in CloudWatch.
  • ConcurrentExecutions metric consistently at the configured limit.

Increase the ElasticCloudForwarderConcurrentExecutions parameter in your CloudFormation stack (named EdotCloudForwarderConcurrentExecutions on the legacy single log type templates).

Note

CloudWatch sources require higher concurrency than S3 sources for the same data rate. See CloudWatch Logs sizing for recommended values.

  • Lambda concurrent executions appear disproportionately high relative to the data rate (for example, 20+ concurrent executions at under 1 MB/s).
  • No throttling or errors are present, but concurrency is much higher than you would expect from an equivalent S3 workload.

This is expected behavior with CloudWatch subscriptions. CloudWatch delivers many small batches (~100-200 KB each) that each trigger a separate Lambda invocation. Concurrency is driven by invocations/sec x avg_duration, not by data volume alone. Unlike S3, where one large file is processed per invocation, CloudWatch sends many short-lived invocations in parallel.

No action is needed unless you see throttling. If you do, increase ElasticCloudForwarderConcurrentExecutions (EdotCloudForwarderConcurrentExecutions on the legacy templates).

Applies to the unified templates, where one stack forwards multiple log types routed by S3 path pattern and CloudWatch log group or stream patterns.

  • Objects arrive in a configured S3 bucket, but one log type produces no data in Elasticsearch, while other types from the same stack work.
  • CloudWatch logs from a configured Log Group produce no data, or land in the catch-all data stream (logs-aws.cloudwatch.otel-default in OpenTelemetry mode, logs-aws.cloudwatch_logs-<namespace> in ECS mode) instead of the expected one.
  1. Check the EnabledLogTypes, WiredS3Buckets, and WiredLogGroups outputs of the CloudFormation stack to confirm the type and its source are configured. If the type is missing, add it to LogTypes (or set LogTypes to all).
  2. For S3 sources, confirm the objects match the type's path pattern. Types are matched by their standard AWS delivery prefix. If your logs are stored under a custom prefix, set it with S3PatternOverrides, for example cloudtrail=my-org/audit/cloudtrail/*.
  3. Some log types have no standard delivery prefix and always require S3PatternOverrides: s3access (S3 server access logs) and the netskope_* types.
  4. For CloudWatch sources, confirm the log group and stream names match the type's patterns, and adjust them with the log group and stream override parameters if needed. Log group overrides take precedence over stream overrides.

Refer to Unified template settings for the override parameters.

You might experience one or more of the following symptoms:

  • Logs are not appearing in Elasticsearch or Kibana dashboards.
  • The S3 failure bucket contains unprocessed event files.
  • CloudWatch logs for the Lambda function show errors.
  • Lambda function metrics show increased error rates or throttling.
  • StatusCode errors when invoking the Lambda function.

  1. Check CloudWatch logs for errors

    Open the AWS CloudWatch console and navigate to the LambdaLogGroup created by the CloudFormation stack. Look for error messages that indicate:

    • Network errors when connecting to the OTLP endpoint.
    • Authentication failures due to invalid or expired API key.
    • Log type mismatches between the source content and the configured log types (LogTypes on the unified templates, EdotCloudForwarderS3LogsType on the legacy templates).

    To get more detailed output, set ElasticCloudForwarderLogLevel (EdotCloudForwarderLogLevel on the legacy templates) to debug in your CloudFormation stack parameters and re-invoke the Lambda function.

  2. Verify your configuration

    Confirm that your CloudFormation stack parameters are correct:

    • OTLPEndpoint points to a valid Managed OTLP endpoint.
    • ElasticApiKey is valid and not expired.
    • Unified templates: LogTypes includes the log types delivered to your sources, and the source lists (S3Buckets, CloudWatchLogGroups) contain the correct ARNs. Refer to A log type is not collected for routing mismatches.
    • Legacy S3 sources: EdotCloudForwarderS3LogsType matches the log format in your S3 bucket (vpcflow, elbaccess, cloudtrail, or waf).
    • Legacy CloudWatch sources: EdotCloudForwarderCWLogType matches the log type (vpcflow or cloudtrail) and SourceCloudWatchLogGroupARN points to an existing Log Group with the trailing :*.
    • The deployment region matches your log source region.

  3. Check Lambda metrics

    In CloudWatch Metrics Explorer, review the Lambda function metrics:

    Metric What to look for
    Errors Increased error count indicates processing failures.
    Throttles High throttle count suggests you need to increase ElasticCloudForwarderConcurrentExecutions (EdotCloudForwarderConcurrentExecutions on the legacy templates).
    Duration Long durations approaching the timeout may cause incomplete processing.
    ConcurrentExecutions Compare against your reserved concurrency limit.

  4. Replay failed events

    If events failed to process, they are stored in the S3 bucket specified by S3FailureBucketARN. Replay them by invoking the Lambda function with the replayFailedEvents trigger:

    aws lambda invoke \
  --function-name <LAMBDA_NAME> \
  --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \
  --cli-binary-format raw-in-base64-out /dev/null

    Replace <LAMBDA_NAME> with the name of your Lambda function from the deployment.

    The following options are available:

    Option Description Default
    dryrun Run without processing events. Useful for understanding what would be replayed. false
    removeOnSuccess Remove the error event from the S3 failure bucket after successful processing. true
    Tip

    Use --timeout with the AWS CLI to increase the Lambda timeout for custom invocations. If a timeout occurs, run the command multiple times to process all error events.

  5. Adjust sizing if needed

    If you're experiencing throttling or timeouts, consider adjusting the Lambda configuration. Refer to Sizing and performance tuning for recommendations based on your log volume.

  • Monitor CloudWatch metrics regularly to catch issues early.
  • Set up CloudWatch alarms for Lambda errors and throttles.
  • Keep your API key up to date and rotate it before expiration.
  • Start with default sizing and increase concurrency or memory only when metrics indicate a need.
  • S3 sources: With the unified templates, verify the delivery prefix of each log type matches its path pattern, and set S3PatternOverrides for custom prefixes. With the legacy templates, ensure each log type uses a dedicated S3 bucket and CloudFormation stack.
  • CloudWatch sources: Start with 20 reserved concurrency and increase if the Throttles metric is non-zero. Memory of 256 MB is sufficient — increasing memory does not significantly improve CloudWatch processing performance.