Learn more about managing your datasets and understanding your overall data storage, period-based retention.
Prerequisite
Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.
The Dataset Management page enables you to manage your datasets and understand your overall data storage duration for different retention periods and datasets based on your hot and cold storage licenses, and retention add-ons that extend your storage. You can view details about your Cortex Cloud licenses and retention add-ons by selecting Settings → Cortex Cloud License.
Important
Cortex Cloud enforces retention on all log-type datasets excluding Host Inventory, Vulnerability Assessment, Metrics, and Users.
Your current hot and cold storage licenses, including the default license retention and any additional retention add-ons to extend storage, are listed within the Hot Storage License and Cold Storage License sections of the Dataset Management page. Whenever you extend your license retention, depending on your requirements and license add-ons for both hot storage and cold storage, the add-ons are listed.
Note
Cold storage, in addition to a cold storage license, requires compute units (CU) to run cold storage queries. For more information on CU, see Manage compute units.
You can expand your license retention to include flexible Hot Storage based retention to help accommodate varying storage requirements for different retention periods and datasets. This add-on license is available to purchase based on your storage requirements for a minimum of 1,000 GB. If this license is purchased, an Additional Storage subheading in the Hot Storage License section is displayed on the Dataset Management page with a bar indicating how much of the storage is used.
Note
Only datasets that are already handled as part of the GB license are supported for this license. In addition, the retention configuration is only available in Cortex Cloud, as opposed to the public APIs.
On any dataset configured to use Additional Hot Storage, you can edit the retention period. This enables you to view the current retention details and configure the retention. This includes setting the amount of flexible hot storage-based retention designated for a dataset and the priority for the dataset's hot storage.
Select Settings → Configurations → Data Management → Dataset Management.
In the Datasets table, right-click any dataset designated with flexible hot storage, and select Edit Retention Plan.
Set the following parameters:
Additional hot storage: Set the amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days.
Hot Storage Priority: Select the priority designated for this dataset's hot storage as either Low, Medium, or High.
Click Save.
For each dataset listed in the table, the following information is available:
Note
Certain fields are exposed and hidden by default. An asterisk (*) is beside every field that is exposed by default.
Datasets include dataset permission enforcements in the Cortex Query Language(XQL), Query Center, and XQL Widgets. For example, to view or access any of the
endpointsandhost_inventorydatasets, you need role-based access control (RBAC) permissions to the Endpoint Administration and Host Inventory views. Managed Security Services Providers (MSSP) administration permissions are not enforced on child tenants, but only on the MSSP tenant.
Field | Description |
|---|---|
*TYPE | Displays the type of dataset based on the method used to upload the data. The possible values include: Correlation, Lookup, Raw, Snapshot, System, and User. For more information on each dataset type, see What are datasets?.What are datasets? |
*LOG UPDATE TYPE | Event logs are updated either continuously (Logs) or the current state is updated periodically (State) as detailed in the Last Updated column. |
*LAST UPDATED | Last time the data in the dataset logs were updated. ImportantThis column is updated once a day. Therefore, if the dataset was created or updated by the target or lookup flows, it's possible that the Last Updated value is a day behind when the queries or reports were run as it was before this column was updated. |
*ADDITIONAL STORAGE | Amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days. |
*TOTAL DAYS STORED | Actual number of days that the data is stored in the Cortex Cloud tenant, which is comprised of the HOT RANGE + the COLD RANGE. |
*HOT RANGE | Details the exact period of the Hot Storage from the start date to the end date. |
*COLD RANGE | Details the exact period of the Cold Storage from the start date to the end date. |
*TOTAL SIZE STORED | Actual size of the data that is stored in the Cortex Cloud tenant. This number is dependent on the events stored in the hot storage. For the |
*ADDITIONAL SIZE STORED | Actual size of the additional flexible hot storage data that is stored in the Cortex Cloud tenant in GB. This number is dependent on the events stored in the hot storage. |
*AVERAGE DAILY SIZE | Average daily amount stored in the Cortex Cloud tenant. This number is dependent on the events stored in the hot storage. |
*HOT STORAGE PRIORITY | Indicates the priority set for the dataset's hot storage as either Low, Medium, or High. |
*TOTAL EVENTS | Number of total events/logs that are stored in the Cortex Cloud tenant. This number is dependent on the events stored in the hot storage. |
*AVERAGE EVENT SIZE | Average size of a single event in the dataset (TOTAL SIZE STORED divided by the TOTAL EVENTS). This number is dependent on the events stored in the hot storage. |
*TTL | For lookup datasets, displays the value of the time to live (TTL) configured for when lookup entries expire and are removed automatically from the dataset. The possible values are:
For more information, see Set time to live for lookup datasets. |
DEFAULT QUERY TARGET | Details whether the dataset is configured to use as your default query target in XQL Search, so when you write your queries you do not need to define a dataset. By default, only the |
TOTAL HOT RETENTION | Total hot storage retention configured for the dataset in months, where a month is calculated as 31 days. |
TOTAL COLD RETENTION | Total cold storage retention configured for the dataset in months, where a month is calculated as 31 days. |
Cortex Cloud supports creating dataset views in the Dataset Management page to enhance data efficiency and security. Dataset views provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined, and provide multiple benefits, such as joining datasets into logical subsets through defined queries, manipulating data without altering underlying datasets, and segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings.
Once a dataset view is created, you can edit or delete the dataset view by right-clicking the dataset view in the Dataset Views table. A dataset view can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset view, you wouldn't be able to delete the dataset view until you removed the dataset view from the XQL query of the Correlation Rule.
Cortex Cloud logs entries for events related to creating, editing, and deleting datasets or dataset views. These monitored activities are available to view in the datasets and dataset views audit logs in the Management Audit Logs. For more information, see Monitor datasets and dataset views activity.
When building an XQL query to define a dataset view, the query is built in the same way as creating a query through the Query Builder. Yet, it's important to be aware of the following points that are specific for dataset view queries:
The following features are unsupported in dataset view queries:
RT Correlation Rules
Query Library
Presets
Cold storage queries (
cold_dataset = <dataset name>)
Only the following XQL stages are supported when building a dataset view query:
alterdedupfieldsfilterjoinreplacenullunion
Once the dataset view is created, it is listed as an available
datasetwhen building your XQL queries as long as you have the necessary permissions to access the dataset view in the Role-based access control (RBAC) settings.
Select Settings → Configurations → Data Management → Dataset Management → Dataset Views.
Click New Datset View.
Enter a Name and Description (optional) for the dataset view.
Create your XQL query for the dataset view by typing in the query box.
(Optional) Click Run to view the query results.
The query must contain no errors, including using only supported commands, to run; otherwise, the Run button remain disabled.
Click Save.
Note
You'll only be able to save the dataset view if the query contains no errors; otherwise, the Save button is disabled.
Once the dataset view is created, you can now control user access permissions through Role-based access control (RBAC).
Notice
Managing Roles requires an Account Admin or Instance Administrator role. For more information, see Predefined user roles.
Access permissions for dataset views are configured in the same way that you set dataset access permissions for any dataset through user roles in Cortex Cloud Access Management. Cortex Cloud uses role-based access control (RBAC) to manage roles with specific permissions for controlling user access. RBAC helps manage access to Cortex Cloud components and datasets, so that users, based on their roles, are granted minimal access required to accomplish their tasks. Once the user role is configured to access these dataset views, you can now assign the user role to the designated users or user groups, who you want to access these dataset views.
Select Settings → Configurations → Access Management.
Configure a user role with the dataset views that you want users to access.
Select Roles.
You can perform one of the following:
To create a new role to assign the dataset views, click New Role, and set a Role Name and Description (optional).
To edit an existing user role with these dataset views, right-click the relevant user role, and select Edit Role.
To create a new role based on an existing role, right-click the relevant user role, select Save As New Role, and set a Role Name and Description (optional).
Under Datasets, you have two options for setting the Cortex Query Language (XQL) dataset access permissions for the user role:
Set the user role with access to all XQL datasets by disabling the Enable dataset access management toggle.
Set the user role with limited access to certain XQL datasets by selecting the Enable dataset access management toggle and selecting the datasets under the different dataset category headings.
Scroll down to Dataset View and select the particular dataset views that you want assigned to this user role.
Click Save.
Note
For more information on user roles, see Manage user roles.
Assign the user role with the dataset views configured to the designated users or user groups. For more information, see Assign a user to a role.
For each dataset view listed in the table, information is available. Here are descriptions on the columns that may require further explanation:
Field | Description |
|---|---|
SOURCE QUERY | Displays the query used to create the dataset view. |
IS VALID | Details whether the query for the dataset view is still valid or not. |
RELATED TABLES | Details the other datasets that are related to this dataset view. |