Manage datasets in Notebooks - Administrator Guide - Cortex XSIAM - Cortex - Security Operations

Create, edit, and delete datasets directly in Notebooks and use them in rules.

You can create datasets in BigQuery through Notebooks using custom Cortex XSIAM APIs. You can then bring the insights and the data enriched through machine learning into Cortex XSIAM to use them inside rules. For example, you can run a query in Cortex XSIAM that searches an incident and correlates it to a sensitive users list you've created in Notebooks to trigger an alert.

To use the Cortex XSIAM APIs inside Notebooks, in Apps → Notebooks, import them from the Cortex SDK.

from cortex.dataset import define_dataset, create_dataset_from_dataframe, delete_dataset, get_created_datasets. 
from cortex.xql import start_query, get_query_results.

The datasets you create are available for querying in the Query Builder, can be used in defining rules, can be viewed under Dataset Management, and can be selected for access when creating a user role. The creation and deletion of datasets is also recorded in the Management Audit Logs.

To change the schema of a dataset created using the Notebooks API, delete the dataset and create a new dataset with the schema you want.

You can use all the Google BigQuery functions to update the data in a dataset created using the Notebooks API.

The functions that are available for creating and editing datasets in Notebooks are listed below.

define_dataset(table_name: str, client: Optional[Client] = None)

create_dataset_from_dataframe(
    table_name: str,
    dataframe: DataFrame,
    schema: Optional[Sequence[Union[SchemaField, Mapping[str, Any]]]] = None,
    client: Optional[Client] = None,
    bq_client: Optional[BqClient] = None,
)

get_created_datasets(client: Optional[Client] = None)

delete_dataset(dataset_name: str, delete_underlying_bq_table: Optional[bool] = False, client: Optional[Client] = None)