Dataset Management - Learn more about managing your datasets and understanding your overall data storage, period based retention. - Administrator Guide - Cortex XDR - Cortex

Dataset Management - Learn more about managing your datasets and understanding your overall data storage, period based retention. - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Pro Administrator Guide

Product

Cortex XDR

License

Pro

Creation date

2024-07-16

Last date published

2024-12-12

Notice

This feature requires a Cortex XDR Pro license.

The Dataset Management page enables you to manage your datasets and understand your overall data storage duration for different retention periods and datasets based on your hot and cold storage licenses, and retention add-ons that extend your storage. You can view details about your Cortex XDR licenses and retention add-ons by selecting Settings → Cortex XDR License. For more information on license retention and the defaults provided per license, see License Retention.

Important

Cortex XDR enforces retention on all log-type datasets excluding Host Inventory, Vulnerability Assessment, Metrics, and Users.

Hot and Cold Storage

Your current hot and cold storage licenses, including the default license retention and any additonal retention add-ons to extend storage, are listed within the Hot Storage License and Cold Storage License sections of the Dataset Management page. Whenever you extend your license retention, depending on your requirements and license add-ons for both hot storage and cold storage, the add-ons are listed.

Note

Cold storage, in addition to a cold storage license, requires compute units (CU) to run cold storage queries. For more information on CU, see Manage Compute Units Usage. For information on the CU add-on license, see Cortex XDR Pro LicenseCortex XDR Pro License.

Additional Hot Storage

You can expand your license retention to include flexible Hot Storage based retention to help accommodate varying storage requirements for different retention periods and datasets. This add-on license is available to purchase based on your storage requirements for a minimum of 1,000 GB. If this license is purchased, an Additional Storage subheading in the Hot Storage License section is displayed on the Dataset Management page with a bar indicating how much of the storage is used.

Note

Only datasets that are already handled as part of the GB license are supported for this license. In addition, the retention configuration is only available in Cortex XDR, as opposed to the public APIs or configuration from the parent MSSP tenant.

Edit Retention Plan

On any dataset configured to use Additional Hot Storage, you can edit the retention period. This enables you to view the current retention details for hot and cold storage and configure the retention. This includes setting the amount of flexible hot storage-based retention designated for a dataset and the priority for the dataset's hot storage. This is used when the storage limit is exceeded to know the data most critical to preserve.

Select Settings → Configurations → Data Management → Dataset Management.
In the Datasets table, right-click any dataset designated with flexible hot storage, and select Edit Retention Plan.
Set the following parameters:
- Additional hot storage: Set the amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days.
- Hot Storage Priority: Select the priority designated for this dataset's hot storage as either Low, Medium, or High. This is used when the storage limit is exceeded. Data is first deleted from lowest to highest, and then from the oldest to latest timestamp.
Click Save.

Datasets Table

For each dataset listed in the table, the following information is available:

Note

Certain fields are exposed and hidden by default. An asterisk (*) is beside every field that is exposed by default.
Datasets include dataset permission enforcements in the Cortex Query Language(XQL), Query Center, and XQL Widgets. For example, to view or access any of the endpoints and host_inventory datasets, you need role-based access control (RBAC) permissions to the Endpoint Administration and Host Inventory views. Managed Security Services Providers (MSSP) administration permissions are not enforced on child tenants, but only on the MSSP tenant.

Field	Description
*TYPE	Displays the type of dataset based on the method used to upload the data. The possible values include: Correlation, Lookup, Raw, Snapshot, System, and User. For more information on each dataset type, see Manage Datasets.
*LOG UPDATE TYPE	Event logs are updated either continuously (Logs) or the current state is updated periodically (State) as detailed in the LAST UPDATED column.
*LAST UPDATED	Last time the data in the dataset logs were updated. Important This column is updated once a day. Therefore, if the dataset was created or updated by the target or lookup flows, it's possible that the Last Updated value is a day behind when the queries or reports were run as it was before this column was updated.
*ADDITIONAL STORAGE	Amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days.
*TOTAL DAYS STORED	Actual number of days that the data is stored in the Cortex XDR tenant, which is comprised of the HOT RANGE + the COLD RANGE.
*HOT RANGE	Details the exact period of the Hot Storage from the start date to the end date.
*COLD RANGE	Details the exact period of the Cold Storage from the start date to the end date.
*TOTAL SIZE STORED	Actual size of the data that is stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage. For the `xdr_data` dataset, where the first 31 days of storage are included with your license, the first 31 days are not included in the TOTAL SIZE STORED number.
*ADDITIONAL SIZE STORED	Actual size of the additional flexible hot storage data that is stored in the Cortex XDR tenant in GB. This number is dependent on the events stored in the hot storage.
*AVERAGE DAILY SIZE	Average daily amount stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage.
*HOT STORAGE PRIORITY	Indicates the priority set for the dataset's hot storage as either Low, Medium, or High. This is used when the storage limit is exceeded. Data is first deleted from lowest to highest, and then from the oldest to latest timestamp.
*TOTAL EVENTS	Number of total events/logs that are stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage.
*AVERAGE EVENT SIZE	Average size of a single event in the dataset (TOTAL SIZE STORED divided by the TOTAL EVENTS). This number is dependent on the events stored in the hot storage.
*TTL	For lookup datasets, displays the value of the time to live (TTL) configured for when lookup entries expire and are removed automatically from the dataset. Forever: The lookup entries never expire (default). Custom: Enables setting when the lookup entries expire by configuring the number of Days, Hours, and Minutes. The maximum number of days is 99999. For more information, see Set time to live for lookup datasets.
DEFAULT QUERY TARGET	Details whether the dataset is configured to use as your default query target in XQL Search, so when you write your queries you do not need to define a dataset. By default, only the `xdr_data` dataset is configured as the DEFAULT QUERY TARGET and this field is set to Yes. All other datasets have this field set to No. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a `join`.
TOTAL HOT RETENTION	Total hot storage retention configured for the dataset in months, where a month is calculated as 31 days.
TOTAL COLD RETENTION	Total cold storage retention configured for the dataset in months, where a month is calculated as 31 days.