Archive Cortex XSOAR data you no longer need regular access to using Elasticsearch index lifecycle management.
Cortex XSOAR supports archiving of partitioned data. Partitioned data is stored in indices on a monthly basis for easy archiving and accessibility. To free up disk space, you can create Elasticsearch snapshots of the relevant indices and then delete the indices.
Manual
To manually archive older data we recommend deleting all indices for a specific month after creating a snapshot for that month. For example, to delete all January 2020 data, use the following API call:DELETE *dmst-*_202001
.
This can be safely done at any given time without shutting down the Cortex XSOAR service. To restore archived data, follow the restore instructions for Elasticsearch backups.
Automated
Elasticsearch supports index lifecycle management through the ILM API, to automatically manage indices retention and optimize old indices. You can set up an ILM policy through the Elasticsearch API or the Elasticsearch UI. OpenSearch also provides an automatic archiving option, through index state management. You can manage an ISM policy through the OpenSearch API.
You can configure a default ILM/ISM policy for all partitioned indices, or for a specific index type. Cortex XSOAR does not currently support rollover operations.
It is not necessary to define all phases of the policy. If a phase is not defined, the next available phase is processed.
Define ILM (Elasticsearch Clusters)
Create or update the lifecycle policy.
PUT _ilm/policy/<policy_id>
Update the index with the lifecycle name.
Note
The index_name can be a wildcard. For example,
PUT *-dmst-*202203/_settings
defines the policy on all partitioned indices for March 2022. Do NOT use wildcards for non partitioned indices.PUT <index_name>/_settings { "settings": { "index.lifecycle.name": <policy_name> } }
Define ISM (OpenSearch Clusters)
Create or update the lifecycle policy.
Create the lifecycle policy.
PUT _plugins/_ism/policies/<policy_id>
Update the lifecycle policy.
Retrieve the
_seq_no
and_primary_term
for the existing policy:GET _plugins/_ism/policies/<policy_id>
Use the
_seq_no
and_primary_term
values retrieved in the previous step to update the lifecycle policy.PUT _plugins/_ism/policies/policy_1?if_seq_no=[seq_no]&if_primary_term=[primary_term]
Update index settings with policy_id.
Note
The index_name can be a wildcard. For example,
PUT *-dmst-*202203/_settings
defines the policy on all partitioned indices for March 2022. Do NOT use wildcards for non partitioned indices.PUT <index_name>/_settings { "settings": { "plugins": { "index_state_management": { "policy_id": [policy_name] } } } }
Add a policy to an index.
POST _plugins/_ism/add/[index_name] { "policy_id": [policy name] }
Examples
In the following examples, we use hot, warm and delete phases, with the allocate, forcemerge and delete actions.
Each index stays at the hot phase for 90 days, and then moves to the warm stage.
In the warm stage, the number_of_replicas and total_shards_per_node is reduced to 1, and forcemerge is executed.
After 1 year the index is deleted.
Elasticsearch ILM example
PUT _ilm/policy/default_policy { "policy": { "_meta": { "description": "This is a default policy", "project": { "name": "dmst", "department": "dev" } }, "phases": { "warm": { "min_age": "90d", "actions": { "allocate": { "number_of_replicas": 1, "total_shards_per_node": "1" }, "forcemerge": { "max_num_segments": 1 } } }, "delete": { "min_age": "365d", "actions": { "delete": {} } } } } }
OpenSearch ISM example
PUT _plugins/_ism/policies/default_policy { "policy": { "description": "This is a default policy", "default_state": "hot", "states": [ { "name": "hot", "actions": [], "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "90d" } } ] }, { "name": "warm", "actions": [ { "replica_count": { "number_of_replicas": 1 } } ], "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "365d" } } ] }, { "name": "delete", "actions": [ { "delete": {} } ], "transitions": [] } ] } }