Manage Datasets - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Pro Administrator Guide

Product
Cortex XDR
License
Pro
Creation date
2023-10-31
Last date published
2024-03-18
Category
Administrator Guide
Abstract

Learn how to import, delete, and interact with custom or third-party datasets in Cortex XDR.

Note

This feature requires a Cortex XDR Pro per GB license.

Cortex XDR runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. If you do not specify a dataset in your query, Cortex XDR runs the query against the default datasets configured, which is by default xdr_data. The xdr_data dataset contains all of the endpoint and network data that Cortex XDR collects. You can always change the default datasets using the set to default option. You can also upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. These uploaded datasets are called lookup datasets.

Important

Forensic datasets are not inlcuded by default in XQL Search query results, unless the dataset query is explicitly defined to use a forensic dataset.

To query other datasets, you have two options: you can either set a dataset as default, which enables you to query the datasets without specifying them in the query, or you can name a specific dataset at the beginning of your query with the dataset stage command. You can add to your list of available datasets by uploading a CSV, TSV, or JSON file to Cortex XDR.

Note

You cannot upload a file that contains a byte array (binary data).

Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following.

  • The dataset formats supported are dependent on the data retention offerings available in Cortex XDR according to whether you want to query hot storage or cold storage.

    • Hot Storage queries are performed on a dataset using the format dataset = <dataset name>. This is the default option.

      dataset = xdr_data
    • Cold Storage queries are performed using the format cold_dataset = <dataset name>.

      cold_dataset = xdr_data
  • The refresh times for datasets. All Cortex XDR system datasets, which are created out-of-the-box, are continuously ingested in near real-time as the data comes in, except for the following exceptions.

    • endpoints—Refreshed every hour.

    • pan_dss_raw—Refreshed daily.

    • Forensics datasets—The Forensics data is not configured to be updated by default. When you enable a collection in the Agent Settings profile, the data is collected only once unless you specify an interval. If you specify an interval, the data is collected every <interval> number of hours with the minimum being 12.

Manage datasets from Cortex XDRSettingsConfigurationsData ManagementDataset Management. You can import, view, update, edit, and interact with your available datasets, and delete them when necessary.

  1. Import a file.

    Bring data in from an external file into Cortex XDR manually to create or update lookup datasets.

    1. Select + LookupImport.

    2. Browse to your CSV, TSV, or JSON file, or drag and drop it into the dialog window. You can only upload a TSV file if it contains a .tsv file extension.

      Note

      When uploading a CSV, TSV, or JSON file, ensure that the file meets the following requirements.

      • The maximum size for the total data to be imported into a lookup dataset is 30 MB.

      • Field names can contain characters from different languages, special characters, numbers (0-9), and underscores (_).

      • Field names cannot exceed 128 characters.

      • Field names can't contain duplicate names, white spaces, or carriage returns.

    3. (Optional) Specify a new Name for the target dataset.

      By default, Cortex XDR uses the name of the original file as the dataset name. You can change this name to something that will be more meaningful for your users when they query the dataset. For example, if the original file name is mrkdptusrsnov23.json, you can save the dataset as marketing_dept_users_Nov_2023.

      Dataset names can contain special characters from different languages, numbers (0-9) and underscores (_). You can create dataset names using uppercase characters, but in queries, dataset names are always treated as if they are lowercase.

      Note

      • The name of a dataset created from a tsv file must always include the extension. If the original file name is mrkdptusrsnov23.tsv, you can name save the dataset with the name marketing_dept_users_Nov_2023.tsv.

    4. Replace the existing data in the dataset overwrites the data in an existing Lookup dataset with the contents of the new file.

    5. Add the file as a lookup.

    6. After receiving a notification reporting that the upload succeeded, Refresh (refresh.png) to view it in your list of datasets.

  2. Save query results as a dataset.

    You can use the target stage command to save query results as a dataset. For details about this command, see the XQL Language Reference guide.

  3. Query against a dataset by selecting it with the dataset command when you create an XQL query.Create an XQL Query

  4. Right-click a dataset to view the schema of the dataset, set it as default, delete it, copy it, edit it, and show or hide datasets. In addition, for a dataset with a TYPE set to Lookup, you can also download the JSON file.

    • View Schema to view the schema information for every field found in the dataset result set in the Schema tab of XQL Search. Each system field in the schema is written with an underscore (_) before the name of the field in the FIELD NAME column in the table.

    • Set as default to query the dataset without having to specify it in your queries in XQL Search as dataset = <name of dataset>. Once configured, the DEFAULT QUERY TARGET column entry for this dataset is set to Yes. By default, this option is not available when right-clicking the xdr_data dataset as this dataset is the only dataset configured as the DEFAULT QUERY TARGET as it contains all of the endpoint and network data that Cortex XDR collects. Once you Set as default another dataset, you can always remove it by right-clicking the dataset and selecting Remove from defaults. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a join.

    • Edit the dataset to update the data manually. Available edit actions are below.

      • Add, edit, and delete individual rows.

      • Delete a number of rows.

      • Add, rename, and delete individual columns.

    • To remove a dataset from Cortex XDR, right click it in the dataset list and Delete.

    • Download the JSON file for a dataset with a Type set to Lookup. This option is not available for any other dataset type.

      Note

      When you download a Lookup dataset with field names in a foreign language, the downloaded JSON file displays the fields as COL_<randomstring> as opposed to returning the fields in the foreign language as expected.

    • Copy text to clipboard to copy the name of the dataset to your clipboard.

    • Copy entire row to copy each cell in a row, separated by tabs, to your clipboard.

    • Show rows with ‘<dataset_name>’ to create a filter that displays all datasets with the same name.

    • Hide rows with ‘<dataset_name>’ to create a filter that hides all datasets with the same name.

  5. Filter your available datasets to specify the ones you want to see.

    1. Select Filter.

      An interface for your filter criteria appears.

    2. Select a field, an operator, and a value to match.

    3. Select + AND or + OR to add additional filter expressions.

    4. Save (save-icon.png) your filter to reuse it later.

      After saving, select the three-dot menu (manage-datasets-three-dot-menu.png) to view your filter.

  6. Customize the table.

    Select the three-dot menu (manage-datasets-three-dot-menu.png) and Layout to change the width of rows and columns. You can also select which columns to display. You can always Restore default layout to go back to displaying the default column layout.