Data storage lifecycle - Administrator Guide - Cortex XSIAM - Cortex - Security Operations

Cortex XSIAM Documentation
Product
Cortex XSIAM
Creation date
2024-03-06
Last date published
2024-10-15
Category
Administrator Guide

Cortex XSIAM data storage is managed in the Cortex XSIAM Data Layer. You receive data storage based on the amount of storage associated with your licenses. Generally, this capacity is determined by factors such as your daily ingestion needs and the number of users in your deployment. All Cortex XSIAM licenses provide you with default retention periods. You can extend your license retention depending on your requirements for hot and cold storage.

To determine the licenses you require for your data storage requirements, it's important that you understand the data storage lifecycle specifically the differences between hot and cold storage options available. The following image shows examples of the differences between hot and cold storage:

Dataset_storage_timeline.png

Data enters Cortex XSIAM via a data stream called the Data Ingestion Pipeline, where all the data manipulation occurs, such as normalization, enrichment, and analytics. Once the data is ready, the data is transferred to any of the following three places, which is dependent on your licenses:

Hot storage

With a regular Cortex XSIAM license, the data is automatically sent to hot storage for the default retention period according to the license, typically one month. If you've purchased additional retention add-ons to extend the hot storage, this is added in monthly increments to the hot storage duration according to the license. For example, a Period-Based Retention - Hot Storage license enables you to extend all the data in hot storage for the number of months designated, while an Additional Hot Storage license provides flexible hot storage so you can extend only the data collected in specific datasets for the number of months designated. During the additional hot storage retention period, data continues to be sent from the Data Ingestion Pipeline. This data is no longer accessible after the hot storage retention period ends and the data is gradually purged.

In the image above, the regular Cortex XSIAM license and additional storage licenses ensure that all the data is accessible from hot storage for two months. After these two months, the data begins to be purged, except for the data in Dataset 2 and Dataset 3. The data in Dataset 2 is accessible for an additional month before it's gradually purged and the data in Dataset 3 is accessible for an additional two months before it's purged.

Cold storage

A regular Cortex XSIAM license doesn't provide any default cold storage retention. Ensure that you understand the following about cold storage:

  • Only after purchasing additional retention with a Period-Based Retention - Cold Storage license is data automatically sent to cold storage from the Data Ingestion Pipeline starting from the purchase date. If you want the cold storage data to align with the hot storage data, you must ensure to purchase your cold storage license at the same time as your regular Cortex XSIAM license.

  • There is no connection between the data in the hot storage and cold storage, so there is no way to add missing data from hot storage to cold storage.

  • The cold storage data is only accessible after the retention period for hot storage is expired. During the hot storage retention period, the cold storage data is collected according to the purchase date. As a result, the retention period for cold storage only begins after the hot storage retention period ends. Once the cold storage retention period ends, the data is no longer accessible and is gradually purged.

  • Requires purchasing a minimum of 6 months of the additional retention.

  • Cold storage, in addition to a cold storage license, requires compute units (CU) to run cold storage queries. For more information on CU, see Manage compute units. For information on the CU add-on license, see Understand the Cortex XSIAM license plan.

In the image above, the cold storage is aligned to the hot storage. The Data Ingestion Pipeline is configured to send data to both hot and cold storage for the first two months but is not accessible in cold storage during the hot storage retention period. After the two months, the Data Ingestion Pipeline continues to send data to cold storage and the data becomes accessible in cold storage for six months except for the data related to Dataset 2 and Dataset 3, which are still within the hot storage retention periods. After the six months of cold storage retention, the data is gradually purged, except for the data related to Dataset 2 and Dataset 3. When Dataset 2 and Dataset 3 finish their allotted hot storage retention periods, the dataset data becomes accessible in cold storage for another six months. Once these six months are over, the dataset data is gradually purged.

Export

A regular Cortex XSIAM license doesn't provide any default export capabilities for Event Forwarding. Only after purchasing an Event Forwarding add-on license is data automatically sent to an intermediate storage location from the Data Ingestion Pipeline starting from the purchase date. This data is no longer accessible after seven days and is gradually purged.

In the image above, the Export data is aligned to the hot and cold storage. The Data Ingestion Pipeline is configured to send data to the intermediate storage location for Event Forwarding, which is accessible for seven days before the data is gradually purged.

For more information on Event Forwarding, see Manage Event Forwarding.

Tip

You can view details about your Cortex XSIAM licenses by selecting Settings → Cortex XSIAM License.