Azure storage metricset

Stack

This is the storage metricset of the module azure.

This metricset allows users to retrieve all metrics from specified storage accounts.

refresh_list_interval

Resources will be retrieved at each fetch call (period interval), this means a number of Azure REST calls will be executed each time. This will be helpful if the azure users will be adding/removing resources that could match the configuration options so they will not added/removed to the list. To reduce on the number of API calls we are executing to retrieve the resources each time, users can configure this setting and make sure the list or resources will not be refreshed as often. This is also beneficial for performance and rate/ cost reasons (https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits).

resources

This will contain all options for identifying resources and configuring the desired metrics

resource_id

([]string) The fully qualified ID’s of the resource, including the resource name and resource type. Has the format /subscriptions/{{guid}}/resourceGroups/{{resource-group-name}}/providers/{{resource-provider-namespace}}/{resource-type}/{{resource-name}}. Should return a list of resources.

resource_group

([]string) This option will return all storage accounts inside the resource group.

service_type

([]string) This configuration key can be used with any of the 2 options above, for example:

resources:
    - resource_id: ""
      service_type: ["blob", "table"]
    - resource_group: ""
      service_type: ["queue", "file"]
		

It will filter the metric values to be returned by specific metric namespaces. The supported metrics and namespaces can be found here https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccounts. The service type values allowed are blob, table, queue, file based on the namespaces Microsoft.Storage/storageAccounts/blobServices,Microsoft.Storage/storageAccounts/tableServices,Microsoft.Storage/storageAccounts/fileServices,Microsoft.Storage/storageAccounts/queueServices. If no service_type is specified all values are applied.

Also, if the resources option is not specified, then all the storage accounts from the entire subscription will be selected. The primary aggregation value will be retrieved for all the metrics contained in the namespaces. The aggregation options are avg, sum, min, max, total, count.

A default non configurable timegrain of 5 min is set so users are advised to configure an interval of 300s or a multiply of it.

default_timegrain:

(string) Sets the default time grain to use when collecting storage account metrics. Defaults to PT5M.

To collect storage account metrics with a PT1M time grain, we recommend using one of the following configurations:

# (1) With `period: 60s` and `default_timegrain: "PT1M"`, the metricset
# collects 1 data point every 60s.
- module: azure
  metricsets:
  - storage
  enabled: true
  period: 60s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 3600s
  enable_batch_api: true
  default_timegrain: "PT1M"
		

1. 1h

# (2) With `period: 300s` and `default_timegrain: "PT1M"`, the metricset
# collects 5 data points every 300s (5 minutes) — one for each minute,
# but all data points arrive after 5 minutes
- module: azure
  metricsets:
  - storage
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 3600s
  enable_batch_api: true
  default_timegrain: "PT1M"
		

1. 1h

These two configurations trade off scalability and freshness. Configuration (1) prioritizes freshness over scalability, while configuration (2) prioritizes scalability over freshness.

Suggested changes:

• enable_batch_api: true: Retrieves metric values for multiple Azure resources in one API call, supporting more storage accounts.
• refresh_list_interval: 3600s: Looks for new storage accounts every 60 minutes instead of 10 minutes, helping to avoid or reduce gaps when monitoring many storage accounts.

Note: By setting the collection period: 1m, the metricset only has 60s to collect all metric values instead of 300s, so it can handle fewer storage accounts. Keep in mind that the storage accounts metricset collects metrics for five different namespaces (storage account, blob, file, queue, and table).

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "azure": {
        "namespace": "Microsoft.Storage/storageAccounts/queueServices",
        "resource": {
            "group": "obs-infrastructure",
            "type": "Microsoft.Storage/storageAccounts"
        },
        "storage": {
            "queue_capacity": {
                "avg": 0
            },
            "queue_count": {
                "avg": 0
            },
            "queue_message_count": {
                "avg": 0
            }
        },
        "subscription_id": "fd675b6f-b5e5-426e-ac45-d1f876d0ffa6",
        "timegrain": "PT1H"
    },
    "cloud": {
        "instance": {
            "id": "/subscriptions/fd675b6f-b5e5-426e-ac45-d1f876d0ffa6/resourceGroups/obs-infrastructure/providers/Microsoft.Storage/storageAccounts/urcbyscmrkbygsawinvm/queueServices/default",
            "name": "urcbyscmrkbygsawinvm"
        },
        "provider": "azure",
        "region": "westeurope"
    },
    "event": {
        "dataset": "azure.storage",
        "duration": 115000,
        "module": "azure"
    },
    "metricset": {
        "name": "storage",
        "period": 10000
    },
    "service": {
        "type": "azure"
    }
}