Juniper SRX Integration for Elastic

The Juniper SRX integration for Elastic enables you to collect and analyze logs from Juniper SRX Series Firewalls. By ingesting these logs into the Elastic Stack, you gain comprehensive visibility into your network's security posture and operational status. You can use this data to identify threats, monitor traffic patterns, and troubleshoot network issues effectively.

This integration is compatible with Juniper SRX Series Firewalls running Junos versions that support structured-data logging. Your device must be capable of generating syslog messages in the structured-data + brief format for successful parsing.

This integration collects logs from Juniper SRX firewalls by receiving syslog data over TCP or UDP, or reading a log file. You deploy an Elastic Agent on a host that is configured as a syslog receiver. The agent listens for incoming logs, parses the structured messages, and forwards the data to your Elastic deployment into the log data stream. The agent can also watch log files written locally to the Elastic Agent's host. This allows you to visualize and search your security data in real-time.

The Juniper SRX integration collects log messages in structured-data format from various security and system processes. It processes the following types of data:

• Firewall session logs: Information on session creation, closing, and denials (RT_FLOW).
• Intrusion detection and prevention (IDP) logs: Security screen events and attack log events (RT_IDS, RT_IDP).
• Unified threat management (UTM) logs: Events related to web filtering, antivirus, and antispam detection (RT_UTM).
• Advanced anti-malware logs: Records of malware actions and host infection events (RT_AAMW).
• Security intelligence logs: Data related to security intelligence actions (RT_SECINTEL).
• Juniper SRX logs: Captures all the security and system processes mentioned above in structured-data format.

The following processes and tags are supported:

Integrating your Juniper SRX logs with the Elastic Stack provides several security and operational benefits. You can use this integration for the following:

• Real-time threat monitoring: Identify and respond to threats detected by IDP, UTM, and anti-malware systems as they happen.
• Network visibility: Analyze firewall session data to understand traffic patterns and identify potential bottlenecks or unusual activity.
• Security auditing: Maintain a searchable, long-term archive of security events and intelligence actions to support compliance requirements.
• Incident investigation: Use detailed session and security logs to trace the origin and impact of security incidents across your network.

To collect logs from your Juniper SRX devices, you'll need to meet these vendor-specific requirements:

• Gain administrative access to the Juniper SRX CLI using SSH or a console connection to perform configuration changes.
• Ensure network connectivity exists between the Juniper SRX management or data interfaces and the host running the Elastic Agent.
• Set up firewall rules to allow traffic on the configured syslog port, which defaults to 9006 for this integration.
• Use an SRX device running a Junos version that supports structured-data syslog formatting.

You'll also need to prepare your Elastic environment with the following:

• Install and enroll an Elastic Agent in Fleet or configure it to run in standalone mode.
• Install the Juniper SRX integration package in Kibana.
• Ensure network access is available for the agent to receive inbound syslog traffic on your specified port.

You must install Elastic Agent on a host that can receive syslog data or access log files from your Juniper SRX device. For more details, check the Elastic Agent installation instructions. You'll only need one Elastic Agent per host.

You'll need the Elastic Agent to stream data from the syslog or log file receiver and ship it to Elastic. From there, the integration's ingest pipelines will process the events.

You can configure your Juniper SRX device to send logs using syslog or write them to a file for collection.

Follow these steps to configure remote syslog:

1. Log in to the Juniper SRX device using SSH or the console port.
2. Enter configuration mode by typing configure.
3. Set the remote syslog destination to the IP address of your Elastic Agent. Replace <AGENT_IP> with your agent's IP and <PORT> with your configured port (for example, 9006).

   set system syslog host <AGENT_IP> any any
   set system syslog host <AGENT_IP> port 9006
   		

4. Configure the mandatory log format. This integration only supports the structured-data format with the brief option:

   set system syslog host <AGENT_IP> structured-data brief
   		

5. Set the security logging mode to event to ensure security logs are sent to the syslog process:

   set security log mode event
   set security log format syslog
   		

6. Verify your configuration by running show system syslog. Ensure the host entry includes the structured-data { brief; } block.
7. Commit your changes by typing commit.

Follow these steps to write logs to a local file:

1. Log in to the Juniper SRX device or the intermediate log host.
2. Configure the SRX to write logs to a file using the structured format:

   set system syslog file juniper-srx.log any any
   set system syslog file juniper-srx.log structured-data brief
   		

3. Ensure the Elastic Agent has read permissions for the file. The default path is /var/log/juniper-srx.log.
4. Commit the changes on the SRX device.

For more information, refer to these Juniper resources:

• Junos CLI reference | structured-data
• Direct system log messages to a remote destination
• SRX Getting Started - Configure System Logging
• SRX Getting Started - Configure Traffic Logging (Security Policy Logs) for SRX Branch Devices

You'll need to enable and configure the integration input method that matches your Juniper configuration. Follow the setup instructions that match your configuration.

This input collects logs over a UDP socket.

1. In Kibana, navigate to Management > Integrations and search for Juniper SRX.
2. Click Add Juniper SRX.
3. Locate the Collecting syslog from Juniper SRX via UDP input.
4. Configure the syslog_host (default: localhost) and syslog_port (default: 9006).
5. If you want to keep the raw log, enable Preserve original event to store it in event.original.
6. Optionally, configure Custom UDP Options like read_buffer (default 100MiB) or max_message_size (default 50KiB).
7. Save the integration to your Agent policy.

This input collects logs over a TCP socket.

1. In Kibana, navigate to Management > Integrations and search for Juniper SRX.
2. Click Add Juniper SRX.
3. Locate the Collecting syslog from Juniper SRX via TCP input.
4. Configure the syslog_host (default: localhost) and syslog_port (default: 9006).
5. If you're using encryption, configure the SSL Configuration settings with your certificate and key.
6. Under Custom TCP Options, you can set the framing method (default delimiter) or max_connections.
7. Save the integration to your Agent policy.

This input collects logs directly from files on the host where the Elastic Agent is running.

1. In Kibana, navigate to Management > Integrations and search for Juniper SRX.
2. Click Add Juniper SRX.
3. Locate the Collecting syslog from Juniper SRX via file input.
4. Set the paths to the absolute path of your log files (default: /var/log/juniper-srx.log).
5. Save the integration to your Agent policy.

After you've finished the configuration, verify that data is flowing correctly into Elasticsearch.

To trigger data flow, perform these actions on your Juniper SRX:

• Generate traffic events: From a device behind the firewall, visit a website or ping an external IP that matches a logged security policy.
• Generate configuration events: Enter configuration mode in the CLI, make a minor change like an interface description, and run commit.
• Generate authentication events: Log out and back into the SRX CLI or J-Web interface.

To check the data in Kibana:

1. Navigate to Analytics > Discover.
2. Select the logs-* data view.
3. Filter for data_stream.dataset: "juniper_srx.log".
4. Verify that logs appear and contain expected fields like event.action, source.ip, or event.outcome.
5. Navigate to Analytics > Dashboards and search for "Juniper SRX" to see the pre-built visualizations.

For help with Elastic ingest tools, check Common problems.

You might encounter the following issues when configuring the Juniper SRX integration:

• Missing structured data format: If logs appear in Kibana as a raw string in the message field and are not parsed, ensure you've committed the set system syslog host <IP> structured-data brief command on the SRX device. This integration requires the structured-data format to identify fields.
• Security logs not sent: If you receive system logs but traffic or IDP logs (like RT_FLOW or RT_IDS) are missing, verify that set security log mode event is configured in the Junos CLI. By default, SRX devices might send security logs using the data plane, bypassing system syslog settings.
• Port mismatch: Confirm that the port configured on the Juniper SRX (for example, set system syslog host <IP> port 9006) matches the syslog_port value in your Elastic Agent integration settings.
• Network connectivity issues: Verify there are no firewalls or network Access Control Lists (ACLs) blocking UDP or TCP traffic on port 9006 between the SRX management interface and the Elastic Agent host.
• Parsing failures: Check the error.message field in Kibana. If it contains "Provided Grok expressions do not match", it typically indicates the device is sending logs in standard syslog format instead of the required structured-data format.
• Incomplete or truncated logs: If log messages are cut off, you might need to increase the max_message_size in the integration's UDP or TCP options to accommodate large structured-data payloads.

To ensure optimal performance in high-volume environments, consider the following:

• Manage your data volume by configuring the Juniper SRX appliance to forward only necessary events, such as RT_FLOW or RT_IDS, using Junos syslog facility and severity filters. Don't forward excessive debug-level logs at the source because it can overwhelm the ingest pipeline.
• Scale your deployment by placing multiple Elastic Agents behind a network load balancer to distribute the ingestion load in high-throughput environments. You should place agents in close network proximity to the SRX devices to reduce latency and minimize the impact of network congestion on log delivery.

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

These inputs can be used with this integration:

ssl.enabled: true
ssl.certificate: "/path/to/server.crt"
ssl.key: "/path/to/server.key"
ssl.certificate_authorities: ["/path/to/ca.crt"]
ssl.client_authentication: "optional"
		

The following resources provide additional information about Juniper SRX logging and configuration:

• Juniper SRX Product Page
• Junos Documentation on Structured Data
• Direct System Log Messages to a Remote Destination - Juniper Networks
• Example: Forward structured system syslog messages from SRX - Juniper Support
• KB16502 - Configure System Logging

The log data stream provides events from Juniper SRX devices. These logs include the following types:

• Traffic logs
• Security logs
• Authentication logs
• Junos OS system events

{
    "@timestamp": "2016-02-18T01:32:50.391Z",
    "agent": {
        "ephemeral_id": "54aa3cbe-60b4-41ae-9a50-c2f871846983",
        "id": "3bf92588-2ea8-4747-8efa-294ffad051db",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.10.2"
    },
    "client": {
        "ip": "192.168.1.100",
        "port": 58071
    },
    "data_stream": {
        "dataset": "juniper_srx.log",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "as": {
            "number": 35908
        },
        "geo": {
            "continent_name": "Asia",
            "country_iso_code": "BT",
            "country_name": "Bhutan",
            "location": {
                "lat": 27.5,
                "lon": 90.5
            }
        },
        "ip": "67.43.156.13",
        "port": 80
    },
    "ecs": {
        "version": "8.17.0"
    },
    "elastic_agent": {
        "id": "3bf92588-2ea8-4747-8efa-294ffad051db",
        "snapshot": false,
        "version": "8.10.2"
    },
    "event": {
        "action": "web_filter",
        "agent_id_status": "verified",
        "category": [
            "network",
            "malware"
        ],
        "dataset": "juniper_srx.log",
        "ingested": "2023-10-03T10:08:52Z",
        "kind": "alert",
        "outcome": "success",
        "severity": 12,
        "timezone": "+00:00",
        "type": [
            "info",
            "denied",
            "connection"
        ]
    },
    "input": {
        "type": "tcp"
    },
    "juniper": {
        "srx": {
            "category": "cat1",
            "process": "RT_UTM",
            "profile": "uf1",
            "reason": "BY_BLACK_LIST",
            "tag": "WEBFILTER_URL_BLOCKED"
        }
    },
    "log": {
        "level": "warning",
        "source": {
            "address": "172.25.0.6:36430"
        }
    },
    "observer": {
        "name": "utm-srx550-b",
        "product": "SRX",
        "type": "firewall",
        "vendor": "Juniper"
    },
    "process": {
        "name": "RT_UTM"
    },
    "related": {
        "hosts": [
            "www.baidu.com"
        ],
        "ip": [
            "192.168.1.100",
            "67.43.156.13"
        ],
        "user": [
            "user01"
        ]
    },
    "server": {
        "ip": "67.43.156.13",
        "port": 80
    },
    "source": {
        "ip": "192.168.1.100",
        "port": 58071,
        "user": {
            "name": "user01"
        }
    },
    "tags": [
        "juniper-srx",
        "forwarded"
    ],
    "url": {
        "domain": "www.baidu.com",
        "path": "/"
    }
}