Loading

abuse.ch integration for Elastic

Stack 9.1.0 Serverless Observability Serverless Security

Version 3.4.0 (View all)
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic
Ingestion method(s) API

The abuse.ch integration for Elastic allows you to collect logs from abuse.ch, which provides actionable, community-driven threat intelligence data and helps identify, track, and mitigate malware and botnet-related cyber threats. With this integration, threat intelligence indicators can be ingested into Elastic for enhanced threat detection and event enrichment.

The abuse.ch integration is compatible with v1 version of abuse.ch URLhaus, MalwareBazaar, ThreatFox, and SSLBL APIs.

This integration periodically queries the abuse.ch APIs to retrieve threat intelligence indicators.

This integration collects threat intelligence indicators into the following datasets:

  • ja3_fingerprints: Collects JA3 fingerprint based threat indicators identified by SSLBL via SSLBL API endpoint.
  • malware: Collects malware payloads from URLs tracked by URLhaus via URLhaus Bulk API.
  • malwarebazaar: Collects malware payloads from MalwareBazaar via MalwareBazaar API.
  • sslblacklist: Collects SSL certificate based threat indicators blacklisted on SSLBL via SSLBL API endpoint.
  • threatfox: Collects threat indicators from ThreatFox via ThreatFox API.
  • url: Collects malware URL based threat indicators from URLhaus via URLhaus API.

The abuse.ch integration brings threat intel into Elastic Security, enabling detection alerts when Indicators of Compromise (IoCs) like malicious IPs, domains, or hashes match your event or alert data. This data can also support threat hunting, enrich alerts with threat context, and power dashboards to track known threats in your environment.

This integration installs Elastic latest transforms. For more details, check the Transform setup and requirements.

abuse.ch requires an Auth Key (API key) for request authentication. Any requests made without this key will be rejected by the abuse.ch APIs.

  1. Sign up for a new account, or login into the abuse.ch authentication portal.
  2. Connect with at least one authentication provider: Google, Github, X, or LinkedIn.
  3. Select Save profile.
  4. In the Optional section, click the Generate Key button to generate Auth Key.
  5. Copy the generated Auth Key.

For more details, check the abuse.ch Community First - New Authentication blog.

This integration supports both Elastic Agentless-based and Agent-based installations.

Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to Agentless integrations and the Agentless integrations FAQ.

Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.

Elastic Agent must be installed. For more details, check the Elastic Agent installation instructions. You can install only one Elastic Agent per host.

  1. In the top search bar in Kibana, search for Integrations.

  2. In the search bar, type abuse.ch.

  3. Select the abuse.ch integration from the search results.

  4. Select Add abuse.ch to add the integration.

  5. Enable and configure only the collection methods which you will use.

    • To Collect abuse.ch logs via API, you'll need to:

      • Configure Auth Key.
      • Enable/Disable the required datasets.
      • For each dataset, adjust the integration configuration parameters if required, including the URL, Interval, etc. to enable data collection.
  6. Select Save and continue to save the integration.

  1. In Kibana, navigate to Dashboards.
  2. In the search bar, type abuse.ch.
  3. Select a dashboard for the dataset you are collecting, and verify the dashboard information is populated.
  1. In Kibana, navigate to Management > Stack Management.
  2. Under Data, select Transforms.
  3. In the search bar, type abuse.ch.
  4. All transforms from the search results should indicate Healthy under the Health column.
  • When creating the Auth Key inside the abuse.ch authentication portal, make sure you connect at least one additional authentication provider to ensure seemless access to the abuse.ch platform.
  • Check for captured ingestion errors inside Kibana. Ingestion errors, including API errors, are captured into error.message field.
    1. Navigate to Analytics > Discover.
    2. In Search field names, search and add fields error.message and data_stream.dataset into the Discover view. For more details on adding fields inside Discover, check Discover getting started.
    3. Search for the dataset(s) that are enabled by this integration. For example, in the KQL query bar, use the KQL query data_stream.dataset: ti_abusech.url to search on specific dataset or KQL query data_stream.dataset: ti_abusech.* to search on all datasets.
    4. Search for errors that are captured into error.message field using KQL query error.message: *. You can combine queries using KQL boolean expressions, such as AND. For example, to search for errors inside url dataset, you can use KQL query: data_stream.dataset: ti_abusech.url AND error.message: *.
  • Common API errors: All the abusec.ch API errors are captured inside the error fields.
    1. abuse.ch APIs return HTTP status 403 Forbidden when the Auth Key is invalid. In such case, the error.message field is populated with message query_status: unknown_auth_key and error.id with 403 Forbidden. To fix this, you need to regenerate the Auth Key in the abuse.ch authentication portal and update the integration policy with newly generated Auth Key.
    2. abuse.ch APIs return HTTP status 500 Internal Server Error when experiencing problem on the abuse.ch service. In such case, error.message field is populated with message POST:500 Internal Server Error (500) and error.id with 500 Internal Server Error. This is likely a one-off scenario and the ingestion should resume normally in the subsequent request.
  • Since this integration supports the expiration of Indicators of Compromise (IoCs) using Elastic latest transform, the threat indicators are present in both source and destination indices. While this may appear to be duplicate ingestion, it is an implementation detail necessary for properly expiring threat indicators.
  • Because the latest copy of threat indicators is now indexed in two places, that is, in both source and destination indices, users must anticipate storage requirements accordingly. The ILM policies on source indices can be tuned to manage their data retention period.
  • For help with Elastic ingest tools, check Common problems.

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

These inputs can be used in this integration:

This integration datasets use the following APIs:

All abuse.ch datasets now support indicator expiration. For the URL dataset, a full list of active threat indicators are ingested at every interval. For other datasets, namely Malware, MalwareBazaar, and ThreatFox, the threat indicators are expired after the duration IOC Expiration Duration is configured in the integration setting. An Elastic Transform is created for every source index to make sure only active threat indicators are available to the end users. Each transform creates a destination index named logs-ti_abusech_latest.dest_* which only contains active and unexpired threat indicators. The indicator match rules and dashboards are updated to list only active threat indicators. Destinations indices are aliased to logs-ti_abusech_latest.<data_stream_name>.

Source Data stream Destination Index Pattern Destination Alias
logs-ti_abusech.url-* logs-ti_abusech_latest.dest_url-* logs-ti_abusech_latest.url
logs-ti_abusech.malware-* logs-ti_abusech_latest.dest_malware-* logs-ti_abusech_latest.malware
logs-ti_abusech.malwarebazaar-* logs-ti_abusech_latest.dest_malwarebazaar-* logs-ti_abusech_latest.malwarebazaar
logs-ti_abusech.threatfox-* logs-ti_abusech_latest.dest_threatfox-* logs-ti_abusech_latest.threatfox

To facilitate IoC expiration, source data stream-backed indices .ds-logs-ti_abusech.<data_stream_name>-* are allowed to contain duplicates from each polling interval. ILM policy logs-ti_abusech.<data_stream_name>-default_policy is added to these source indices, so it doesn't lead to unbounded growth. This means that in these source indices data will be deleted after 5 days from ingested date.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.