What are datasets? - Administrator Guide - Cortex XSIAM - Cortex

What are datasets? - Administrator Guide - Cortex XSIAM - Cortex - Security Operations

Cortex XSIAM Documentation

Product

Cortex XSIAM

Creation date

2024-03-06

Last date published

2024-12-12

Category

Administrator Guide

Abstract

Learn how to import, delete, and interact with custom or third-party datasets in Cortex XSIAM.

Cortex XSIAM runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. If you do not specify a dataset in your query, Cortex XSIAM runs the query against the default datasets configured, which is by default xdr_data for a dataset query. The xdr_data dataset contains all of the endpoint and network data that Cortex XSIAM collects. For a Cortex Data Model (XDM) query, unless specific datasets are specified, a query will run against all mapped datasets. You can always change the default datasets using the set to default option. You can also upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. These uploaded datasets are called lookup datasets.

It's also possible to create dataset views, which provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined. Dataset views enhance data efficiency and security. For example, by segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings. For more information, see Dataset views.

To query other datasets, you have the following options:

Set a dataset as default, which enables you to query the datasets without specifying them in the query.
Name a specific dataset at the beginning of your query with the dataset stage command.

The type of dataset is based on the method used to upload the data. The possible types include:

Correlation: A dataset containing data saved from a correlation rule.
Lookup: A dataset containing key-value pairs that can be used as a reference to correlate to events. For example, a user list with corresponding access privileges. You can import or create a lookup dataset, and then reference the values for a certain key, run queries and take action. For more information, see Lookup datasets.
Raw: Every dataset where PANW data is ingested out-of-the-box or third-party data is ingested using a configured dedicated collector.
Snapshot: A dataset that contains only the last successful snapshot of the data, such as Workday or ServiceNow CMDB tables.
System: Cortex XSIAM datasets that are created out-of-the-box.
User: If saved by a query using the target command, the Type can be either User or Lookup.

Important

By default, forensic datasets are not included in XQL query results, unless the dataset query is explicitly defined to use a forensic dataset.

Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following:

The dataset formats supported are dependent on the data retention offerings available in Cortex XSIAM according to whether you want to query hot storage or cold storage.
- Hot Storage queries are performed on a dataset using the format dataset = <dataset name>. This is the default option.
```
dataset = xdr_data
```
- Cold Storage queries are performed using the format cold_dataset = <dataset name>.
```
cold_dataset = xdr_data
```
The refresh times for datasets, where all Cortex XSIAM system datasets, which are created out-of-the-box, are continuously ingested in near real-time as the data comes in, except for the following:
- endpoints: Refreshed every hour.
- pan_dss_raw: Refreshed daily.
- Forensics datasets: The Forensics data is not configured to be updated by default. When you enable a collection in the Agent Settings profile, the data is collected only once unless you specify an interval. If you specify an interval, the data is collected every <interval> number of hours with the minimum being 12.
Query against a dataset by selecting it with the dataset command when you create an XQL query. For more information, see Create XQL query.
After you query runs, you can always save your query results as a dataset. You can use the target stage command to save query results as a dataset.

Managing datasets and dataset views

You can manage your datasets and dataset views in Cortex XSIAM from the Settings → Configurations → Data Management → Dataset Management page.

Below are some of the main tasks available for all dataset types by right-clicking a particular dataset or dataset view listed in either the Datasets or Dataset Views table. Only tasks that need further explanation are explained below. Datasets and dataset views can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset or dataset view, you wouldn't be able to delete the dataset or dataset view until you removed the dataset view from the XQL query of the Correlation Rule.

Note

For more information on tasks specific to lookup datasets, see Lookup datasets.

Select Set as default to query the dataset without having to specify it in your queries in XQL by typing dataset = <name of dataset>. Once configured, the DEFAULT QUERY TARGET column entry for this dataset is set to Yes in the Datasets table. By default, this option is not available when right-clicking the xdr_data dataset as this dataset is the only dataset configured as the DEFAULT QUERY TARGET as it contains all of the endpoint and network data that Cortex XSIAM collects. Once you Set as default another dataset, you can always remove it by right-clicking the dataset and selecting Remove from defaults. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XSIAM queries the default datasets using a join. This option is only relevant for datasets.