Understanding how to write a [INGEST] section in a Parsing Rules file and the syntax to use.
An INGEST
section is used to define the resulting dataset. The COLLECT
, CONST
, and RULE
sections are only add-ons, used to help organize the INGEST
sections, and are optional to configure. Yet, a Parsing Rules file that contains no INGEST
sections, generates no Parsing Rules. Therefore, the INGEST
section is mandatory to configure.
INGEST
syntax is derived from Cortex Query Language (XQL) with a few modifications as explained in the Parsing Rules syntax. In addition, INGEST
sections contain the following syntax add-ons.
INGEST
sections can have more than one XQLp statement, separated by a semicolon (;
). Each statement creates a different Parsing Rule.The following XQL functions and stages are also supported in the
INGEST
section:Functions—
arrayfilter
,arraycreate
,arraymerge
, andobject_create
.Stages—
iploc
andarrayexpand
.
Another new stage is available called
drop
.drop
takes a condition similar to the XQLfilter
stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, sodrop <condition>
is not equivalent tofilter not <condition>
.drop
can only appear last in a statement. No other XQLp rules can follow.
INGEST
sections take parameters, and not names asRULE
sections use, where some are mandatory and others optional.[ingest:vendor=<vendor>, product=<product>, dataset=<dataset>, no_hit=<keep\drop>, ingestnull=<true\false>] filter raw_log not contains "alert";
The parameter descriptions are explained in the following table.
Parameter | Description |
---|---|
| The vendor that the specified Parsing Rules apply to (mandatory). |
| The product that the specified Parsing Rules apply to (mandatory). |
| The name of the dataset to insert every row with the results after applying any of the specified Parsing Rules (mandatory). |
| No-match strategy to use for the entire specified group of rules (optional). The default is
|
| Defines whether null value fields are ingested (optional). By default this is set to |
Each statement represents a different Parsing Rule in the same group as depicted in the following example.
[CONST] DEVICE_NAME = "ngfw"; [rule:use_two_rules] filter severity = "medium" | call basic_rule | call use_xql_and_another_rule; [rule:basic_rule] fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something"; [rule:use_xql_and_another_rule]call multiline_statement | filter severity = "medium"; [rule:multiline_statement] alter url = json_extract(_raw_log, "$.url") | join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url |filter severity = "medium"; [ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=drop] filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); call use_two_rules | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME;
This generates 1 group of 2 Parsing Rules for panw/ngfw, where all the ingested data into panw_ngfw_ds
dataset.
The following represents the syntax for the rules.
Rule #1: filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); Rule #2: filter severity = "medium" | fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something" | alter url = json_extract(_raw_log, "$.url") | join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url | filter severity = "medium" | filter severity = "medium" | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME
A few more points to keep in mind when writing INGEST
sections.
INGEST
parameter names are not case-sensitive. Therefore,vendor=PANW
andvendor=panw
are the same.Since section order is unimportant, you do not have to declare a
RULE
or aCONST
before using it in anINGEST
section.You can have multiple
INGEST
sections with the samevendor
,product
,dataset
, andno_hit
values. Yet, this can lead to unexpected results. Consider the following example:[ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=keep] filter raw_log not contains "alert"; [ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=keep] filter device_type not contains "agent";
Let
lw
be a log row. Iflw.raw_log
contains analert
andlw.device_type
contains anagent
, thenlw
is inserted twice into thepan_ngfw_ds
dataset as every section is standalone.To eliminate these kind of errors and misunderstandings, it is highly advised to group all rules having the same
vendor
,product
,dataset
, andno_hit
values in a singleINGEST
section.Logs that were discarded by a
drop
stage are considered ingested with a no-match policy. This means they are not kept even ifno_hit = keep
.Keep in mind that all rules inside a group get evaluated independently. This is in contrast to firewall-like rules, which stop evaluating the first rule that is able to make a decision. Therefore, without proper filtering, it is possible to ingest the same log more than once.
You can override the default raw dataset in
INGEST
sections. For more information, see Parsing Rules Raw Dataset.Cortex XDR supports configuring case sensitivity in Parsing Rules only within the INGEST section using the following configuration stage:
config case_sensitive = true | false
You can add a single tagor list of tags to the ingested data as part of the ingestion flow that you can easily query in XQL Search. You can add tags as part of the
INGEST
section or use both theINGEST
andRULE
sections. The following are examples of each.INGEST
section.Adding a single tag.
[INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ] tag add "New Event"
Adding a list of tags.
[INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ] tag add "New Event1", "New Event2", "New Event3"
INGEST
andRULE
sections.Adding a single tag.
[INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop , ingestnull = true ] alter xx = call new_tag_rule;
[RULE:new_tag_rule] tag add "test";
Adding a list of tags.
[INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop , ingestnull = true ] alter xx = call new_tag_rule;
[RULE:new_tag_rule] tag add "test1", "test2", "test3";