Parsing Rules file structure and syntax - Administrator Guide - Cortex XSIAM - Cortex - Security Operations

Cortex XSIAM Documentation
Product
Cortex XSIAM
Creation date
2024-03-06
Last date published
2024-10-13
Category
Administrator Guide
Abstract

The Parsing Rules file consists of multiple sections of three types, which also represent the custom syntax specific to Parsing Rules.

Note

Only a user with Cortex Account Administrator or Instance Administrator permissions can access Parsing Rules.

File structure

The Parsing Rules file consists of multiple sections of these three types, which also represent the custom syntax specific to Parsing Rules.

  • INGEST: This section is used to define the resulting dataset.

  • COLLECT (Optional): This section defines a rule that enables data reduction and data manipulation at the Broker VM to help avoid sending unnecessary data to the Cortex XSIAM server and reduce traffic, storage, and computing costs. In addition, the COLLECT section is used to manipulate, alter, and enrich the data before it’s passed to the Cortex XSIAM server. While this rule is optional to configure, once added this rule runs before the INGEST section.

  • CONST (Optional): This section is used to define strings and numbers that can be reused multiple times within Cortex Query Language (XQL) statements in other INGEST sections by using $constName.

  • RULE (Optional): Rules are part of the XQL syntax, which are tagged with a name, and can be reused in the code in the INGEST sections by using [rule:ruleName].

  • EXTEND (Optional): This section is used to chain your Parsing Rules logic to extend your existing default RULE sections, which are added by a Content Package you installed from the Marketplace. An EXTEND section runs immediately after the default RULE section that it extends and enables data manipulation without overriding or interfering with the existing vendor Parsing Rules.

The order of the sections is unimportant. The data of each section type gets grouped together during the parsing stage. Before any action takes place all COLLECT, CONST, RULE, EXTEND, and INGEST objects are grouped together and collected to the same list.

Syntax

The syntax used in the Parsing Rules file is derived from XQL, but with a few modifications. This subset of XQL is called XQL for Parsing (XQLp).

Note

For more information on the XQL syntax, see Cortex XQL Language Reference.

The COLLECT, CONST, INGEST, RULE, and EXTENDsyntax is derived from XQL, but with the following modifications for XQLp:

  • A statement never starts with a dataset or preset selection. The query's data source is meaningless. It is transparent to the user where the raw logs are coming from, fully handled by the system.

  • Only the following XQL stages are permitted: alter, fields, filter, and join. In addition, a new call stage is supported, which is used to invoke another rule.

    Note

    • An inner type of join stage is only supported in CONST, INGEST, and RULE sections and is not supported in a COLLECT section.

    • You cannot call a RULE section that exists in Default Rules from the User Defined Rules section.

  • Only the following XQL functions are permitted in all sections: parse_timestamp, parse_epoch, and regexcapture.

    Note

    The regexcapture function is only supported in Parsing Rules and cannot be used in any other XQL query.

  • No output stages are supported.

  • A Rule object can only contain a single statement.

  • A join inner query is restricted to using a lookup as a data source and is only supported in XQLp stages.

    There is no default lookup, so all join inner queries must start with dataset=<lookup> | ....

  • CONST reference ($MY_CONST) is supported.

  • An IN condition can only take a sequence list, such as device_name in (“device1”, “device2”, “device3”) and not another XQL or XQLp inner queries.

Comments in C programming language can be used anywhere throughout the Parsing Rules file:

// line comment
/* inner comment */

Note

Every statement in the Parsing Rules file must end with a semicolon (;).