INGEST - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Documentation
Product
Cortex XDR
Creation date
2024-03-06
Last date published
2024-10-09
Category
Administrator Guide
Abstract

Understanding how to write a [INGEST] section in a Parsing Rules file and the syntax to use.

An INGEST section is used to define the resulting dataset. The COLLECT, CONST, and RULE sections are only add-ons, used to help organize the INGEST sections, and are optional to configure. Yet, a Parsing Rules file that contains no INGEST sections, generates no Parsing Rules. Therefore, the INGEST section is mandatory to configure.

INGEST syntax is derived from Cortex Query Language (XQL) with a few modifications as explained in the Parsing Rules syntax. In addition, INGEST sections contain the following syntax add-ons:

  • INGEST sections can have more than one XQLp statement, separated by a semicolon (;). Each statement creates a different Parsing Rule.

  • The following XQL functions and stages are also supported in the INGEST section:

  • Another new stage is available called drop.

    • drop takes a condition similar to the XQL filter stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, so drop <condition> is not equivalent to filter not <condition>.

    • drop can only appear last in a statement. No other XQLp rules can follow.

  • INGEST sections take parameters, and not names as RULE sections use, where some are mandatory and others optional.

    [ingest:vendor=<vendor>, product=<product>, dataset=<dataset>, no_hit=<keep\drop>, ingestnull=<true\false>]
filter raw_log not contains "alert";

The parameter descriptions are explained in the following table:

Parameter

Description

vendor

The vendor that the specified Parsing Rules apply to (mandatory).

product

The product that the specified Parsing Rules apply to (mandatory).

dataset

The name of the dataset to insert every row with the results after applying any of the specified Parsing Rules (mandatory).

no_hit

No-match strategy to use for the entire specified group of rules (optional). The default is keep.

  • If no_hit = drop, then in a scenario where none of the rules in the group generates output for a given log record, that record is discarded.

  • If no_hit = keep, then in a scenario where none of the rules in the group generates output for a given log record, that record is kept in the _raw_log field. This record is inserted into the group's dataset once, but every column holds NULL except for _raw_log, which holds the original JSON log record.

ingestnull

Defines whether null value fields are ingested (optional). By default this is set to true, so you only need to set this parameter when you want to overwrite the default definition.

Each statement represents a different Parsing Rule in the same group as depicted in the following example:

Example 45. 
[CONST]
DEVICE_NAME = "ngfw"; 
[rule:use_two_rules]
filter severity = "medium" | call basic_rule | call use_xql_and_another_rule; 
[rule:basic_rule]
fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something"; 
[rule:use_xql_and_another_rule]call multiline_statement | filter severity = "medium"; 
[rule:multiline_statement]
alter url = json_extract(_raw_log, "$.url")
| join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url 
|filter severity = "medium"; 
[ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=drop]
filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url");
call use_two_rules | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME;

This generates 1 group of 2 Parsing Rules for panw/ngfw, where all the ingested data into panw_ngfw_ds dataset.

The following represents the syntax for the rules:

Rule #1:
filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); 
Rule #2:
filter severity = "medium"
| fields log_type, severity
| filter log_type="eal" and severity="HIGH" and type="something"
| alter url = json_extract(_raw_log, "$.url")
| join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url
| filter severity = "medium"
| filter severity = "medium"
| join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity
| fields severity, log_type
| drop device_name = $DEVICE_NAME

A few more points to keep in mind when writing INGEST sections:

  • INGEST parameter names are not case-sensitive. Therefore, vendor=PANW and vendor=panw are the same.

  • Since section order is unimportant, you do not have to declare a RULE or a CONST before using it in an INGEST section.

  • You can have multiple INGEST sections with the same vendor, product, dataset , and no_hit values. Yet, this can lead to unexpected results. Consider the following example:

    Example 46. 
    [ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=keep]
filter raw_log not contains "alert"; 
[ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=keep]
filter device_type not contains "agent";

    Let lw be a log row. If lw.raw_log contains an alert and lw.device_type contains an agent, then lw is inserted twice into the pan_ngfw_ds dataset as every section is standalone.

    • To eliminate these kind of errors and misunderstandings, it is highly advised to group all rules having the same vendor, product, dataset , and no_hit values in a single INGEST section.

    • Logs that were discarded by a drop stage are considered ingested with a no-match policy. This means they are not kept even if no_hit = keep.

    • Keep in mind that all rules inside a group get evaluated independently. This is in contrast to firewall-like rules, which stop evaluating the first rule that is able to make a decision. Therefore, without proper filtering, it is possible to ingest the same log more than once.

  • You can override the default raw dataset in INGEST sections. For more information, see Parsing Rules Raw Dataset.

  • Cortex XDR supports configuring case sensitivity in Parsing Rules only within the INGEST section using the following configuration stage:

    config case_sensitive = true | false

  • You can add a single tag to the ingested data as part of the ingestion flow that you can easily query. You can add tags as part of the INGEST section or use both the INGEST and RULE sections. The following are examples of each:

    • INGEST section:

      Example 47. 

      Adding a single tag:

      [INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ]
tag add "New Event"

      Adding a list of tags:

      [INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ]
tag add "New Event1", "New Event2", "New Event3"

    • INGEST and RULE sections:

      Example 48. 

      Adding a single tag:

      [INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop  , ingestnull = true ]
alter xx = call new_tag_rule; 
      [RULE:new_tag_rule]
tag add "test";

      Adding a list of tags:

      [INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop  , ingestnull = true ]
alter xx = call new_tag_rule; 
      [RULE:new_tag_rule]
tag add  "test1", "test2", "test3";