Collective Intelligence Framework v3 Integration


Version	1.24.0 (View all)
Managed integration release status	GA
Subscription level What's this?	Basic
Developed by What's this?	Community
Ingestion method(s)	API
Minimum Kibana version(s)	9.1.0 8.19.0

Important - Managed integrations on Elastic Cloud Hosted

You can use this integration as an Elastic Managed integration on Elastic Cloud Hosted deployments running this version or later.

This integration connects with the REST API from the running CIFv3 instance to retrieve indicators.

Agentless Enabled Integration

Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to Agentless integrations and the Agentless integrations FAQ. Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.

Expiration of Indicators of Compromise (IOCs)

Indicators are expired after a certain duration. An Elastic Transform is created for a source index to allow only active indicators to be available to the end users. The transform creates a destination index named logs-ti_cif3_latest.dest_feed* which only contains active and unexpired indicators. Destination indices are aliased to logs-ti_cif3_latest.feed. The indicator match rules and dashboards are updated to show only active indicators.

Indicator Type	Indicator Expiration Duration
`ipv4-addr`	`45d`
`ipv6-addr`	`45d`
`domain-name`	`90d`
`url`	`365d`
`file`	`365d`
All Other Types	Derived from `IOC Expiration Duration` setting

Data retention

Threat indicators are re-collected across polling intervals, and the latest transform keeps the active, deduplicated view in its destination index. The source data streams therefore hold repeated copies of the same indicators.

The package bounds the growth of these source data streams with a retention that depends on the deployment type:

Data stream	Self-managed and Elastic Cloud Hosted (ILM policy)	Serverless (data stream lifecycle)
`logs-ti_cif3.feed-*`	`logs-ti_cif3.feed-default_policy`: roll over after 2d, delete 3d after rollover	delete 5d after rollover

On self-managed and Elastic Cloud Hosted deployments the ILM policy applies. The data stream lifecycle shipped with the package is not used there. ILM counts the delete age from the rollover of a backing index, so a document can remain for up to the rollover age plus the delete age. On Serverless, ILM is not available and the data stream lifecycle applies instead. It also works per backing index: Elasticsearch rolls the write index over automatically on age, size, or document count, and deletes a backing index once the retention has passed since it rolled over. A document therefore stays for the retention plus up to one rollover interval. The rollover age is derived from the retention and is an implementation detail that Elasticsearch may change. Where the package installs a transform, the transform's destination indices are not affected by either.

To keep data for a different period:

Self-managed and Elastic Cloud Hosted: edit the ILM policy in Kibana under Stack Management → Index Lifecycle Policies, or with PUT _ilm/policy/<policy name>. A package upgrade reinstalls the package's ILM policies, so check your change after upgrading.
Serverless: set the retention on the data stream, for example PUT _data_stream/logs-ti_cif3.feed-default/_lifecycle with the body {"data_retention": "90d"}. Replace default with your namespace.

Data Streams

Feed

The CIFv3 integration collects threat indicators based on user-defined configuration including a polling interval, how far back in time it should look, and other filters like indicator type and tags.

CIFv3 confidence field values (0..10) are converted to ECS confidence (None, Low, Medium, High) in the following way:

CIFv3 Confidence	ECS Conversion
Beyond Range	None
0 - <3	Low
3 - <7	Medium
7 - 10	High

Field	Description	Type
@timestamp	Event timestamp.	date
cif3.application	The application used by the indicator, such as telnet or ssh.	keyword
cif3.asn	AS Number of IP.	integer
cif3.asn_desc	AS Number org name.	keyword
cif3.cc	Country code of GeoIP.	keyword
cif3.city	GeoIP city information.	keyword
cif3.confidence	The confidence on a scale of 0-10 that the tags appropriately contextualize the indicator.	float
cif3.count	The number of times the same indicator has been reported with the same metadata by the same provider.	integer
cif3.deleted_at	The indicator expiration timestamp.	date
cif3.description	A description of the indicator.	keyword
cif3.expiration_duration	The configured expiration duration.	keyword
cif3.indicator	The value of the indicator, for example if the type is fqdn, this would be the value.	keyword
cif3.indicator_iprange	IPv4 or IPv6 IP Range.	ip_range
cif3.indicator_ipv4	IPv4 address.	ip
cif3.indicator_ipv4_mask	subnet mask of IPv4 CIDR.	integer
cif3.indicator_ipv6	singleton IPv6 address.	keyword
cif3.indicator_ipv6_mask	subnet mask of IPv6 CIDR.	integer
cif3.indicator_ssdeep_chunk	SSDEEP hash chunk.	text
cif3.indicator_ssdeep_chunksize	SSDEEP hash chunk size.	integer
cif3.indicator_ssdeep_double_chunk	SSDEEP hash double chunk.	text
cif3.itype	The indicator type, can for example be "ipv4, fqdn, email, url, sha256".	keyword
cif3.latitude	Latitude of GeoIP.	keyword
cif3.location	Lat/Long of GeoIP.	geo_point
cif3.longitude	Longitude of GeoIP.	keyword
cif3.portlist	The port or range of ports used by the indicator.	text
cif3.protocol	The protocol used by the indicator.	text
cif3.provider	The source of the indicator information.	keyword
cif3.rdata	Extra text or descriptive content related to the indicator such as OS, reverse lookup, etc.	keyword
cif3.reference	A reference URL with further info related to the indicator.	keyword
cif3.region	GeoIP region information.	keyword
cif3.tags	Comma-separated list of words describing the indicator such as "malware,exploit".	keyword
cif3.timezone	Timezone of GeoIP.	text
cif3.uuid	The ID of the indicator.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Name of the module this data is coming from.	constant_keyword
input.type	Type of Filebeat input.	keyword
labels.is_ioc_transform_source	Indicates whether an IOC is in the raw source data stream, or the in latest destination index.	constant_keyword
log.flags	Flags for the log file.	keyword
log.offset	Offset of the entry in the log file.	long
threat.feed.name	Display friendly feed name	constant_keyword
threat.indicator.first_seen	The date and time when intelligence source first reported sighting this indicator.	date
threat.indicator.last_seen	The date and time when intelligence source last reported sighting this indicator.	date
threat.indicator.modified_at	The date and time when intelligence source last modified information for this indicator.	date
threat.indicator.tls.client.ja3	An md5 hash that identifies clients based on their TLS handshake.	keyword

		{
    "@timestamp": "2024-08-01T08:05:14.040Z",
    "agent": {
        "ephemeral_id": "b351d699-2fd0-49f7-99e1-a7a471a29a62",
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "cif3": {
        "deleted_at": "2022-09-03T20:25:53.000Z",
        "expiration_duration": "45d",
        "indicator": "20.206.75.106",
        "itype": "ipv4",
        "portlist": "443",
        "uuid": "ac240898-1443-4d7e-a98a-1daed220c162"
    },
    "data_stream": {
        "dataset": "ti_cif3.feed",
        "namespace": "26457",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-08-01T08:05:14.040Z",
        "dataset": "ti_cif3.feed",
        "ingested": "2024-08-01T08:05:26Z",
        "kind": "enrichment",
        "original": "{\"application\":\"https\",\"asn\":8075,\"asn_desc\":\"microsoft-corp-msn-as-block\",\"cc\":\"br\",\"city\":\"campinas\",\"confidence\":10,\"count\":1,\"firsttime\":\"2022-07-20T20:25:53.000000Z\",\"group\":[\"everyone\"],\"indicator\":\"20.206.75.106\",\"indicator_ipv4\":\"20.206.75.106\",\"itype\":\"ipv4\",\"lasttime\":\"2022-07-20T20:25:53.000000Z\",\"latitude\":-22.9035,\"location\":[-47.0565,-22.9035],\"longitude\":-47.0565,\"portlist\":\"443\",\"protocol\":\"tcp\",\"provider\":\"sslbl.abuse.ch\",\"reference\":\"https://sslbl.abuse.ch/blacklist/sslipblacklist.csv\",\"region\":\"sao paulo\",\"reporttime\":\"2022-07-21T20:33:26.585967Z\",\"tags\":[\"botnet\"],\"timezone\":\"america/sao_paulo\",\"tlp\":\"white\",\"uuid\":\"ac240898-1443-4d7e-a98a-1daed220c162\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "network": {
        "protocol": "https",
        "transport": "tcp"
    },
    "related": {
        "ip": [
            "20.206.75.106"
        ]
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "cif3-indicator",
        "botnet"
    ],
    "threat": {
        "indicator": {
            "as": {
                "number": 8075,
                "organization": {
                    "name": "microsoft-corp-msn-as-block"
                }
            },
            "confidence": "High",
            "first_seen": "2022-07-20T20:25:53.000Z",
            "geo": {
                "country_iso_code": "br",
                "location": {
                    "lat": -22.9035,
                    "lon": -47.0565
                },
                "region_name": "sao paulo",
                "timezone": "america/sao_paulo"
            },
            "ip": "20.206.75.106",
            "last_seen": "2022-07-20T20:25:53.000Z",
            "marking": {
                "tlp": "WHITE"
            },
            "modified_at": "2022-07-21T20:33:26.585967Z",
            "name": "20.206.75.106",
            "provider": "sslbl.abuse.ch",
            "reference": "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv",
            "sightings": 1,
            "type": "ipv4-addr"
        }
    }
}
		
	

Changelog

Version	Details	Minimum Kibana version
1.24.0	Enhancement (View pull request) Document the retention applied to source data streams on stateful and Serverless deployments, and how to change it.	9.1.0 8.19.0
1.23.0	Enhancement (View pull request) Remove redundant ECS field definitions and the catch-all dynamic template from the transforms, relying on the `ecs@mappings` component template.	9.1.0 8.19.0
1.22.0	Enhancement (View pull request) Enable request trace log removal.	9.0.5 8.19.2
1.21.0	Enhancement (View pull request) Add tags to ingest pipeline processors and add `preserve_original_event` to pipeline-level `on_failure` handlers.	9.0.5 8.19.2
1.20.2	Enhancement (View pull request) Set agentless deployment mode `release` field to `ga`.	9.0.5 8.19.2
1.20.1	Bug fix (View pull request) Fix transform destination to use keyword for data_stream.namespace, allowing multi-namespace deployments.	9.0.5 8.19.2
1.20.0	Enhancement (View pull request) Enable Agentless deployment.	9.0.5 8.19.2
1.19.0	Enhancement (View pull request) Use num_failure_retries instead of unattended mode for transform failure recovery.	9.0.0 8.13.0
1.18.2	Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines.	9.0.0 8.13.0
1.18.1	Bug fix (View pull request) Remove duplicate security-solution-default tag references	9.0.0 8.13.0
1.18.0	Enhancement (View pull request) Allow transforms to run in unattended mode.	9.0.0 8.13.0
1.17.0	Enhancement (View pull request) Update Kibana constraint to support 9.0.0.	9.0.0 8.13.0
1.16.0	Enhancement (View pull request) Do not remove `event.original` in main ingest pipeline.	8.13.0
1.15.0	Enhancement (View pull request) Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".	8.13.0
1.14.4	Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines.	8.13.0
1.14.3	Bug fix (View pull request) Fix labels.is_ioc_transform_source values	8.13.0
1.14.2	Bug fix (View pull request) Add missing fields in transform	8.13.0
1.14.1	Bug fix (View pull request) Fix ECS date mapping on threat fields.	8.13.0
1.14.0	Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0
1.13.1	Bug fix (View pull request) Adjust field mappings for transform destination index.	8.12.0
1.13.0	Enhancement (View pull request) Improve handling of empty responses.	8.12.0
1.12.0	Enhancement (View pull request) Support for IOC expiration	8.12.0
1.11.0	Enhancement (View pull request) Set sensitive values as secret.	8.12.0
1.10.2	Bug fix (View pull request) Clean up null handling	8.7.1
1.10.1	Enhancement (View pull request) Changed owners	8.7.1
1.10.0	Enhancement (View pull request) Limit request tracer log count to five.	8.7.1
1.9.0	Enhancement (View pull request) ECS version updated to 8.11.0.	8.7.1
1.8.0	Enhancement (View pull request) Set 'community' owner type.	8.7.1
1.7.0	Enhancement (View pull request) ECS version updated to 8.10.0.	8.7.1
1.6.0	Enhancement (View pull request) The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest.	8.7.1
1.5.0	Enhancement (View pull request) Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.	8.7.1
1.4.0	Enhancement (View pull request) Update package-spec to 2.9.0.	8.7.1
1.3.0	Enhancement (View pull request) Update package to ECS 8.9.0.	8.7.1
1.2.0	Enhancement (View pull request) Document duration units.	8.7.1
1.1.0	Enhancement (View pull request) Document valid duration units.	8.7.1
1.0.0	Enhancement (View pull request) Release Collective Intelligence Framework as GA.	8.7.1
0.8.0	Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors.	8.7.1
0.7.0	Enhancement (View pull request) Update package to ECS 8.8.0.	8.7.1
0.6.0	Enhancement (View pull request) Add a new flag to enable request tracing	8.7.1
0.5.0	Enhancement (View pull request) Update package to ECS 8.7.0.	8.0.0
0.4.1	Enhancement (View pull request) Honor `preserve_original_event` setting.	8.0.0
0.4.0	Enhancement (View pull request) Update package to ECS 8.6.0.	8.0.0
0.3.1	Bug fix (View pull request) Use ECS definition for `threat.indicator.geo.location`.	8.0.0
0.3.0	Enhancement (View pull request) Update package to ECS 8.5.0.	—
0.2.2	Bug fix (View pull request) Remove duplicate field.	8.0.0
0.2.1	Enhancement (View pull request) Fix documentation build error	8.0.0
0.2.0	Enhancement (View pull request) Labelling with Threat Intelligence category	8.0.0
0.1.0	Enhancement (View pull request) Initial draft of the package	8.0.0