Loading

EDOT Cloud Forwarder for Azure

Serverless Observability Preview EDOT CF Azure Preview

EDOT Cloud Forwarder (CF) for Azure allows you to stream telemetry data from Azure services to an Elasticsearch cluster. EDOT CF for Azure supports the following sources:

Source Description
Activity log Logs generated by Azure Monitor

Read on to learn how to set up EDOT Cloud Forwarder for Azure.

Note

We are working to support other sources. Get in touch to let us know of any specific requirements that might influence our plans.

Important

EDOT Cloud Forwarder for Azure requires a Managed OTLP endpoint and an API key. Managed OTLP is available for Elastic Cloud Serverless and will soon be available for Elastic Cloud Hosted.

For self-managed deployments, set up an EDOT Collector in Gateway mode that ingests OTel data from the edge setup into the self-managed Elastic Stack.

To collect telemetry data using EDOT Cloud Forwarder for Azure, you need:

  • An Azure subscription
  • Azure CLI
  • Bicep extension for Azure CLI

To use EDOT Cloud Forwarder for Azure, you need an Azure subscription with permissions to create resources.

The Bicep template for EDOT Cloud Forwarder creates the following resources:

  • Function App
  • App Service plan
  • Event Hubs Namespace with two Event Hubs (logs and metrics)
  • Storage account
  • Application Insights workspace

You can use the Azure CLI to deploy and manage resources required for EDOT Cloud Forwarder.

To install Azure CLI, refer to the official documentation.

EDOT Cloud Forwarder for Azure uses Bicep to define all the necessary resources and their dependencies.

After you've installed Azure CLI, install Bicep by running the following command:

az bicep install

Before deploying EDOT Cloud Forwarder for Azure, take the following into consideration:

  • The logs event hub supports Azure resource logs.
  • Support for additional encoding extensions for logs and metrics will be added in future releases.

Follow these steps to deploy EDOT Cloud Forwarder for Azure in a resource group to send telemetry data to the Elastic Cloud Managed OTLP Endpoint endpoint.

  1. Retrieve the OTLP endpoint and API key

    To find out the URL of the managed OTLP endpoint and the API key for authentication, follow these steps:

    Steps to retrieve the OTLP endpoint and API key

    Follow these steps to retrieve the managed OTLP endpoint URL for your Serverless project:

    1. In Elastic Cloud Serverless, open your Observability project.
    2. Go to Add data → Application → OpenTelemetry.
    3. Select Managed OTLP Endpoint in the second step.
    4. Copy the OTLP endpoint configuration value.
    5. Select Create API Key to generate an API key.

    In the Bicep templates, the OTLP endpoint is set as elasticsearchOtlpEndpoint, and the API key is set as elasticsearchApiKey.

    Important

    Trim the API key from Authorization=ApiKey MYKEYVALUE... to just MYKEYVALUE... before using it as the argument to the elasticsearchApiKey parameter.

  2. Set the environment variables

    Define the following environment variables:

    export AZURE_LOCATION="<your_region>"
export RESOURCE_GROUP="<group_name>"
export ELASTICSEARCH_OTLP_ENDPOINT="<your_otlp_endpoint>"
export ELASTICSEARCH_API_KEY="<your_api_key>"

  3. Create the resource group

    Create the resource group that hosts all the resources for EDOT Cloud Forwarder for Azure.

    $ az group create --name $RESOURCE_GROUP --location $AZURE_LOCATION
{
  "id": "/subscriptions/<subscription_id>/resourceGroups/<group_name>",
  "location": "<your_region>",
  "managedBy": null,
  "name": "<group_name>",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

  4. Download the Bicep template

    Download the Bicep template to deploy EDOT Cloud Forwarder for Azure at the following URL: https://ela.st/edot-cf-azure-template

  5. Deploy the Bicep template

    Deploy the Bicep template using the following command:

    az deployment group create \
    --resource-group ${RESOURCE_GROUP} \
    --template-file ecf.bicep \
    --parameters \
        elasticsearchOtlpEndpoint=$ELASTICSEARCH_OTLP_ENDPOINT \
        elasticsearchApiKey=$ELASTICSEARCH_API_KEY \
        logsDecoder=ds \
        metricsDecoder=dcr \
        eventHubPartitionCount=8 \
        eventHubMessageRetentionInDays=1 \
    1. To collect Activity logs from a diagnostic settings, set logsDecoder to ds (diagnostic settings).
    2. To collect Storage Account metrics from a Data Collection Rule, set metricsDecoder to dcr (data collection rules).

    For a complete list of parameters, refer to Configure the template.

  6. Collect telemetry data

    Follow these instructions to collect telemetry data depending on your use case.

    Create a diagnostic setting to collect Activity logs from an Azure subscription.

    1. In the Azure portal, open Subscriptions from the search bar.
    2. Select the Azure subscription from which you want to collect Activity logs.
    3. Go to Activity log → Export Activity Logs → Add diagnostic setting.
    • In Diagnostic setting name, enter a name for the diagnostic setting.
    • Select all categories in Logs > Administrative.
    • In Destination details, select Stream to an event hub.
    • In Event hub namespace, select the namespace created by the EDOT Cloud Forwarder (ECF) for Azure.
    • In Event hub name (optional), select the logs event hub created by the EDOT Cloud Forwarder (ECF) for Azure.
    1. Select Save to create the diagnostic setting.
    2. After a few minutes, the diagnostic setting starts streaming the Activity logs to the logs event hub.
    3. Go to Discover in your Elastic Cloud deployment or Serverless project, and select the logs-* data view.
    4. Filter the docs by data_stream.dataset, to browse the Activity logs streamed to Elasticsearch.

EDOT Cloud Forwarder for Azure uses a Bicep template to deploy the EDOT CF.

The following parameters are available in the Bicep template:

Parameter Default Description Available options
elasticsearchOtlpEndpoint OTLP service endpoint.
elasticsearchApiKey Elasticsearch API key.
logsDecoder ds Decoder to decode the logs. ds (diagnostic settings)
metricsDecoder ds Decoder to decode the metrics. ds (diagnostic settings) and dcr (Data Collection Rules)
eventHubPartitionCount 4 Number of partition in the event hubs.
eventHubMessageRetentionInDays 1 Number of days for the event hub data retention.

The Bicep template creates a number of resources to process logs from a specific log source.

This is the list of resources created by the Bicep template.

Resource name Type Description
eventHubNamespace Microsoft.EventHub/namespaces Event Hubs namespace to host the event hubs to stream logs and metrics.
logsEventHub Microsoft.EventHub/namespaces/eventhubs Event hub for the logs.
metricsEventHub Microsoft.EventHub/namespaces/eventhubs Event hub for the metrics.
storageAccount Microsoft.Storage/storageAccounts Storage Account container to store the Event Hubs trigger checkpoint information and the failed messages.
hostingPlan Microsoft.Web/serverfarms Hosting plan that runs the function app.
functionApp Microsoft.Web/sites The function app that runs the EDOT Collector.
applicationInsights Microsoft.Insights/components Application Insights instance to monitor the application.

Note the following:

  • The eventHubNamespace hosts the two event hubs responsible for receiving logs and metrics from Azure services.
  • The storageAccount stores the checkpoint information for the Event Hub triggers, and it also hosts the failed logs and metrics in the logs-error-container and metrics-error-container containers.
  • The functionApp is the core component responsible for running the OTel collector inside the Azure Function.

If you no longer need the resources and want to remove them, use the following command:

az group delete --name $RESOURCE_GROUP

This removes all the resources in the resource group.

Warning

If you remove the resource group, data that's still unprocessed will be lost.