Archive Data with Elasticsearch - Administrator Guide - 6.9 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide

Product
Cortex XSOAR
Version
6.9
Creation date
2022-09-29
Last date published
2024-05-22
End_of_Life
EoL
Category
Administrator Guide
Abstract

Archive Cortex XSOAR data you no longer need regular access to, using Elasticsearch index lifecycle management.

Cortex XSOAR supports archiving of partitioned data. Partitioned data is stored in indices on a monthly basis for easy archiving and accessibility. To free up disk space, you can create Elasticsearch snapshots of the relevant indices and then delete the indices.

Manual

To manually archive older data we recommend deleting all indices for a specific month after creating a snapshot for that month. For example, to delete all January 2020 data, use the following API call: DELETE *dmst-*_202001.

This can be safely done at any given time without shutting down the Cortex XSOAR service. To restore archived data, follow the restore instructions for Elasticsearch backups.

Automated

Elasticsearch supports index lifecycle management through the ILM API, to automatically manage indices retention and optimize old indices. You can set up an ILM policy through the Elasticsearch API or the Elasticsearch UI. OpenSearch also provides an automatic archiving option, through index state management. You can manage an ISM policy through the OpenSearch API.

You can configure a default ILM/ISM policy for all partitioned indices, or for a specific index type. Cortex XSOAR does not currently support rollover operations.

It is not necessary to define all phases of the policy. If a phase is not defined, the next available phase is processed.

Define ILM (Elasticsearch clusters)

  1. Create or update the lifecycle policy.

    PUT _ilm/policy/<policy_id>
  2. Update the index with the lifecycle name.

    Note

    The index_name can be a wildcard. For example, PUT *-dmst-*202203/_settings defines the policy on all partitioned indices for March 2022. Do NOT use wildcards for non partitioned indices.

    PUT <index_name>/_settings 
    {
    	 "settings": {
    	   	"index.lifecycle.name": <policy_name>
    	 }
    } 

Define ISM (OpenSearch clusters)

  1. Create or update the lifecycle policy.

    • Create the lifecycle policy.

      PUT _plugins/_ism/policies/<policy_id>
    • Update the lifecycle policy.

      Retrieve the _seq_no and _primary_term for the existing policy:

      GET _plugins/_ism/policies/<policy_id>

      Use the _seq_no and _primary_term values retrieved in the previous step to update the lifecycle policy.

      PUT _plugins/_ism/policies/policy_1?if_seq_no=[seq_no]&if_primary_term=[primary_term]
  2. Update index settings with policy_id.

    Note

    The index_name can be a wildcard. For example, PUT *-dmst-*202203/_settings defines the policy on all partitioned indices for March 2022. Do NOT use wildcards for non partitioned indices.

    PUT <index_name>/_settings
    {
      "settings": {
        "plugins": {
          "index_state_management": {
            "policy_id": [policy_name]    
          }
        }
      }
    }
  3. Add a policy to an index.

    POST _plugins/_ism/add/[index_name]
    {
      "policy_id": [policy name]
    }
    

Examples

In the following examples, we use hot, warm and delete phases, with the allocate, forcemerge and delete actions.

  1. Each index stays at the hot phase for 90 days, and then moves to the warm stage.

  2. In the warm stage, the number_of_replicas and total_shards_per_node is reduced to 1, and forcemerge is executed.

  3. After one year the index is deleted.

Elasticsearch ILM example

PUT _ilm/policy/default_policy 
{
   "policy": {
      "_meta": {
         "description": "This is a default policy",
         "project": {
            "name": "dmst",
            "department": "dev"
         }
      },
      "phases": {
         "warm": {
            "min_age": "90d",
            "actions": {
               "allocate": {
                  "number_of_replicas": 1,
                  "total_shards_per_node": "1"
               },
               "forcemerge": {
                  "max_num_segments": 1
               }
            }
         },
         "delete": {
            "min_age": "365d",
            "actions": {
               "delete": {}
            }
         }
      }
   }
}

OpenSearch ISM example

 PUT _plugins/_ism/policies/default_policy
{
  "policy": {
    "description": "This is a default policy",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_index_age": "90d"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "replica_count": {
              "number_of_replicas": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "365d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ]
  }
}