What are datasets? - Administrator Guide - Cortex XDR - Cortex

What are datasets? - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Documentation

Product

Cortex XDR

Creation date

2024-03-06

Last date published

2024-10-01

Note

This feature requires a Cortex XDR Pro per GB license.

Cortex XDR runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. If you do not specify a dataset in your query, Cortex XDR runs the query against the default datasets configured, which is by default xdr_data. The xdr_data dataset contains all of the endpoint and network data that Cortex XDR collects. You can always change the default datasets using the set to default option. You can also upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. These uploaded datasets are called lookup datasets.

To query other datasets, you have the following options:

Set a dataset as default, which enables you to query the datasets without specifying them in the query.
Name a specific dataset at the beginning of your query with the dataset stage command.

The type of dataset is based on the method used to upload the data. The possible types include:

Correlation: A dataset containing data saved from a correlation rule.
Lookup: A dataset containing key-value pairs that can be used as a reference to correlate to events. For example, a user list with corresponding access privileges. You can import or create a lookup dataset, and then reference the values for a certain key, run queries and take action. For more information, see Lookup datasets.
Raw: Every dataset where PANW data is ingested out-of-the-box or third-party data is ingested using a configured dedicated collector.
Snapshot: A dataset that contains only the last successful snapshot of the data, such as Workday or ServiceNow CMDB tables.
System: Cortex XDR datasets that are created out-of-the-box.
User: If saved by a query using the target command, the Type can be either User or Lookup.

Important

By default, forensic datasets are not included in XQL query results, unless the dataset query is explicitly defined to use a forensic dataset.

Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following:

The dataset formats supported are dependent on the data retention offerings available in Cortex XDR according to whether you want to query hot storage or cold storage.
- Hot Storage queries are performed on a dataset using the format dataset = <dataset name>. This is the default option.
```
dataset = xdr_data
```
- Cold Storage queries are performed using the format cold_dataset = <dataset name>.
```
cold_dataset = xdr_data
```
The refresh times for datasets, where all Cortex XDR system datasets, which are created out-of-the-box, are continuously ingested in near real-time as the data comes in, except for the following:
- endpoints: Refreshed every hour.
- pan_dss_raw: Refreshed daily.
- Forensics datasets: The Forensics data is not configured to be updated by default. When you enable a collection in the Agent Settings profile, the data is collected only once unless you specify an interval. If you specify an interval, the data is collected every <interval> number of hours with the minimum being 12.
Query against a dataset by selecting it with the dataset command when you create an XQL query. For more information, see Create XQL query.
After you query runs, you can always save your query results as a dataset. You can use the target stage command to save query results as a dataset.

Managing datasets

You can manage your datasets in Cortex XDR from the Settings → Configurations → Data Management → Dataset Management page.

Below are some of the main tasks available for all dataset types by right-clicking a particular dataset listed in the Datasets table. Only tasks that need further explanation are explained below. Datasets can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset, you wouldn't be able to delete the dataset until you removed the dataset view from the XQL query of the Correlation Rule.

Note

For more information on tasks specific to lookup datasets, see Lookup datasets.

Select Set as default to query the dataset without having to specify it in your queries in XQL by typing dataset = <name of dataset>. Once configured, the DEFAULT QUERY TARGET column entry for this dataset is set to Yes. By default, this option is not available when right-clicking the xdr_data dataset as this dataset is the only dataset configured as the DEFAULT QUERY TARGET as it contains all of the endpoint and network data that Cortex XDR collects. Once you Set as default another dataset, you can always remove it by right-clicking the dataset and selecting Remove from defaults. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a join.