Manage Datasets - Administrator Guide - Cortex XDR - Cortex XSIAM - Cortex - Security Operations

Cortex XDR Pro Administrator Guide
Product
Cortex XDR
License
Pro
Creation date
2023-03-23
Last date published
2023-03-23

Note

This feature requires a Cortex XDR Pro per TB license.

Cortex XDR runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. You can upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. If you do not specify a dataset in your query, Cortex XDR runs the query against the default datasets configured, which is by default xdr_data. The xdr_data dataset contains all of the endpoint and network data that Cortex XDR collects. You can always change the default datasets using the set to default option.

To query other datasets, you have two options: you can either set a dataset as default, which enables you to query the datasets without specifying them in the query, or you can name a specific dataset at the beginning of your query with the dataset stage command. You can add to your list of available datasets by uploading a CSV, TSV, or JSON file to Cortex XDR.

Note

You cannot upload a file that contains a byte array (that is, binary data).

Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following.

  • The dataset formats supported are dependent on the data retention offerings available in Cortex XDR according to whether you want to query hot storage or cold storage.

    • Hot Storage queries are performed on a dataset using the format dataset = <dataset name>. This is the default option.

      dataset = xdr_data

    • Cold Storage queries are performed using the format cold_dataset = <dataset name>.

      cold_dataset = xdr_data

  • The refresh times for datasets. All Cortex XDR system datasets, which are created out-of-the-box, are continuously ingested in near real-time as the data comes in, except for the following exceptions.

    • endpoints—Refreshed every hour.

    • pan_dss_raw—Refreshed daily.

    • Forensics datasets—The Forensics data is not configured to be updated by default. When you enable a collection in the Agent Settings profile, the data is collected only once unless you specify an interval. If you specify an interval, the data is collected every <interval> number of hours with the minimum being 12.

Manage datasets from Cortex XDR → Settings → Configurations → Data Management → Dataset Management. On the Dataset Management page, you can import, view, and interact with your available datasets.

  1. Import a dataset.

    1. Select + Lookup.

    2. Browse to your CSV, TSV, or JSON file, or drag and drop it into the dialog window.You can only upload a TSV file that contains a .tsv file extension.

      Note

      When uploading a CSV, TSV, or JSON file, ensure that the file meets the following requirements.

      • Field names are supported using different languages, numbers (0-9), or underscores (_). If you use any other characters, Cortex XDR automatically converts them to underscores (_).

      • Dataset names are supported using different languages. Numbers (0-9) and underscores (_) are supported, but not as the first character of the name.

        You can create dataset names using uppercase characters, but in queries, dataset names are always treated as if they are lowercase.

      • Must start with a letter or underscore. Cannot use prefixes TABLE, FILE, or _PARTITION.

      • Cannot exceed 128 characters.

      • No duplicate names, white spaces, or carriage returns.

    3. (Optional) Rename the file, where only English alphabetical characters are supported.

    4. Add the file as a lookup.

    5. After receiving a notification reporting that the upload succeeded, Refresh (refresh.png) to view it in your list of datasets.

      If the file has the same name as an existing dataset, Cortex XDR will append an underscore and a number to the name to make it unique.

  2. Save query results as a dataset.

    You can use the target stage command to save query results as a dataset. For details about this command, see the XQL Language Reference guide.

  3. Query against a dataset by selecting it with the dataset command when you create an XQL query.Create an XQL Query

  4. Right-click a dataset to view the schema of the dataset, set it as default, delete it, copy it, and show or hide datasets. In addition, for a dataset with a TYPE set to Lookup, you can also download the JSON file.

    • View Schema to view the schema information for every field found in the dataset result set in the Schema tab of XQL Search. Each system field in the schema is written with an underscore (_) before the name of the field in the FIELD NAME column in the table.

    • Set as default to query the dataset without having to specify it in your queries in XQL Search as dataset = <name of dataset>. Once configured, the DEFAULT QUERY TARGET column entry for this dataset is set to Yes. By default, this option is not available when right-clicking the xdr_data dataset as this dataset is the only dataset configured as the DEFAULT QUERY TARGET as it contains all of the endpoint and network data that Cortex XDR collects. Once you Set as default another dataset, you can always remove it by right-clicking the dataset and selecting Remove from defaults. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a join.

    • Delete to remove the dataset from Cortex XDR.

    • Download the JSON file for a dataset with a Type set to Lookup. This option is not available for any other dataset type.

      Note

      When you download a Lookup dataset with field names in a foreign language, the downloaded JSON file displays the fields as COL_<randomstring> as opposed to returning the fields in the foreign language as expected.

    • Copy text to clipboard to copy the name of the dataset to your clipboard.

    • Copy entire row to copy each cell in a row, separated by tabs, to your clipboard.

    • Show rows with ‘<dataset_name>’ to create a filter that displays all datasets with the same name.

    • Hide rows with ‘<dataset_name>’ to create a filter that hides all datasets with the same name.

  5. Filter your available datasets to specify the ones you want to see.

    1. Select Filter.

      An interface for your filter criteria appears.

    2. Select a field, an operator, and a value to match.

    3. Select + AND or + OR to add additional filter expressions.

    4. Save (save-icon.png) your filter to reuse it later.

      After saving, select the three-dot menu (manage-datasets-three-dot-menu.png) to view your filter.

  6. Customize the table.

    Select the three-dot menu (manage-datasets-three-dot-menu.png) and Layout to change the width of rows and columns. You can also select which columns to display. You can always Restore default layout to go back to displaying the default column layout.