COLLECT - Understand how to write a [COLLECT] section in a Parsing Rules file, and the syntax to use. - Administrator Guide - Cortex XSIAM - Cortex

COLLECT - Understand how to write a [COLLECT] section in a Parsing Rules file, and the syntax to use. - Administrator Guide - Cortex XSIAM - Cortex - Security Operations

Cortex XSIAM Documentation

Product

Cortex XSIAM

Creation date

2024-03-06

Last date published

2026-01-25

Note

The CSV Collector applet is not affected by the COLLECT rules applied to a Broker VM.

To avoid performance issues on the Broker VM, Cortex XSIAM does not permit all Parsing Rules to run on the Broker VM by default, but only the Parsing Rules that you designate.

The Broker VM is directly affected by the [COLLECT] rules you create, so depending on the complexity of the rules more hardware resources on the Broker VM may be required. As a result, ensure that your Broker VM meets the following minimum hardware requirements to run [COLLECT] rules:

8-core processor
8GB RAM
512GB disk
Plan for a max of 10K eps (events per second) per core.

COLLECT syntax is derived from Cortex Query Language (XQL) with a few modifications as explained in the Parsing Rules syntax. In addition, COLLECT rules contain the following syntax add-ons:

COLLECT rules can have more than one XQLp statement, separated by a semicolon (;). Each statement creates a different data reduction and manipulation at the Broker VM for a different vendor and product.
While the XQL stages alter and fields are permitted in COLLECT rules for various vendors and products, you should avoid using them for supported vendors that can be used for Analytics as these stages can disrupt the operation of the Analytics Engine. For a list of these vendors, see the Visibility of logs and alerts from external sources table specifically those vendors with Normalized Log Visibility.
Another new stage is available called drop.
- drop takes a condition similar to the XQL filter stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, so drop <condition> is not equivalent to filter not <condition>.
- drop can only appear last in a statement. No other XQLp syntax can follow.

COLLECT sections take parameters, where some are mandatory and others optional.

[COLLECT:vendor=<vendor>, product=<product>, target_brokers = (<broker_ID1, brokerID2,...>), no_hit = <keep\drop>];

Example 128.

Here's an example of how to define the COLLECT section with a single broker_ID:

[COLLECT:vendor="PANW", product="NGFW_CEF", target_brokers=(BROKER_ID), no_hit=drop]

Here's an example of how to define the COLLECT section with multiple broker_IDs:

[COLLECT:vendor="PANW", product="NGFW_CEF", target_brokers=(BROKER_ID1, BROKER_ID2, BROKER_ID3), no_hit=drop]

The parameter descriptions are explained in the following table:

Parameter	Description
`vendor`	The vendor that the specified `COLLECT` rule for data reduction and data manipulation at the Broker VM applies to (mandatory).
`product`	The product that the specified `COLLECT` rule for data reduction and data manipulation at the Broker VM applies to (mandatory).
`target_brokers`	Specifies the list of Brokers to run the `COLLECT` rule for data reduction and data manipulation based on the vendor and product configured (mandatory). When `target_brokers=*`, the `COLLECT` rule applies to all the data collected by the Broker VM applets. Note The CSV Collector applet is not affected by the `COLLECT` rules applied to a Broker VM.
`no_hit`	No-match strategy to use for the entire specified group of `COLLECT` rules (optional). The default is `keep`. If `no_hit = drop`, then in a scenario where none of the `COLLECT` rules in the group generates output for a given event, that event is discarded. If `no_hit = keep`, then in a scenario where none of the `COLLECT` rules in the group generates output for a given event, that event is passed to the Cortex XSIAM server.

The following is an example of using a COLLECT rule to filter data for a specific vendor and product that will run before the INGEST section.

Example 129.

[COLLECT:vendor="Apache", product="ApacheServer", target_brokers = (bvm1, bvm2, bvm3), no_hit = drop]
alter source_log = json_extract_scalar(_raw_log, "$.source") 
| filter source_log = "WebApp-Logs"
| fields source_log, _raw_log;
[INGEST:vendor="Apache", product="ApacheServer", target_dataset = "dvwa_application_log"]
alter log_timestamp = json_extract_scalar(_raw_log, "$.timestamp")
| alter log_msg = json_extract_scalar(_raw_log, "$.msg")
| alter log_remote_ip = json_extract_scalar(_raw_log, "$.Remote_IP")
| alter scanned_ip = json_extract_scalar(_raw_log, "$.Scanned_IP")
| fields log_msg ,log_remote_ip ,log_timestamp ,source_log ,scanned_ip , _raw_log;

A few more points to keep in mind when writing COLLECT rules:

There are no COLLECT rules by default, so all collected events are forwarded by the Broker VM to the Cortex XSIAM server.
Tip
To reduce the amount of data transmitted to Cortex XSIAM from the broker, use filters to drop logs. Yet, be aware that once the logs are modified using alter or fields stages, the Broker VM will convert the original log into a JSON format, which could increase the data size being sent from the broker to Cortex XSIAM.
When COLLECT rules are defined, the designated Broker VMs check every collected event versus each rule. When there is a match for a given product or vendor, the Broker VM checks if it meets the filter criteria.
- If it meets the criteria, the event is passed to the Cortex XSIAM server.
- If it doesn’t meet the criteria, it depends on the no_hit parameter.
  -If no_hit=drop, then this COLLECT rule will not pass the event. Yet, the event still goes through other rules on this Broker VM.
  -If no_hit=keep, the event is passed to the Cortex XSIAM server, and goes through other rules on this Broker VM.
When the evaluated event, doesn’t match any product or vendor for a defined COLLECT rule, the event is passed to the Cortex XSIAM server.