Free up Disk Space with Data Archiving - Administrator Guide - 6.10 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide

Product
Cortex XSOAR
Version
6.10
Creation date
2022-10-13
Last date published
2024-04-08
End_of_Life
EoL
Category
Administrator Guide
Abstract

Free up disk space by archiving Cortex XSOAR folders to condense the unused data within them.

Archive Incident, Entry, and Indicator Data

To free up disk space, you can archive compressed folders to save unused or old data. It is recommended to archive folders and not delete them permanently.

Cortex XSOAR supports full archiving of incident, entry, and indicator data on a monthly basis. For Elasticsearch deployments, see Archive Data with Elasticsearch. For Hosted Service deployments, contact Customer Support.

Locate the folders that reside in the following location (where Cortex XSOAR is installed), /var/lib/demisto/data/.

Note

Although the folders reside in /var/lib/demisto/data/, Do Not save the backup folders under /var/lib/demisto/.

The instructions below refer to paths for a single server deployment. For multi-tenant, you archive data on a per tenant basis. The following are the equivalent paths for multi-tenant:

Single Server

Multi-Tenant

/var/lib/demisto/

/var/lib/demisto/tenants/acc_{TENANT_NAME}/

/var/lib/demisto/data/

/var/lib/demisto/tenants/acc_{TENANT_NAME}/data/

/var/lib/demisto-archive/

/var/lib/demisto-archive/acc_{TENANT_NAME}/

The following data folder and files can be found in this folder:

  • demisto.db: The database for all playbooks and automation (not anything related to incidents and insights).

  • demistoidx: Indexing of the system.

  • partitionsData: Data of incidents, insights, and entries separated by month resolution.

    If you archive indexes you need to archive the matching partitions, so they are not rebuilt.

The following is an example of how the folders and filenames will appear in your system.

$ tree /var/lib/demisto/data

├── demisto.db
├── demistoidx
│   ├── accounts
│   │   ├── index_meta.json
│   │   └── store
...
│   ├── entries_082017
│   │   ├── index_meta.json
│   │   └── store
│   ├── entries_092017
│   │   ├── index_meta.json
│   │   └── store
│   ├── entries_102017
│   │   ├── index_meta.json
│   │   └── store
│   ├── evidences
│   │   ├── index_meta.json
│   │   └── store
│   ├── incidents_082017
│   │   ├── index_meta.json
│   │   └── store
│   ├── incidents_092017
│   │   ├── index_meta.json
│   │   └── store
│   ├── incidents_102017
│   │   ├── index_meta.json
│   │   └── store
│   ├── investigations_082017
│   │   ├── index_meta.json
│   │   └── store
│   ├── investigations_092017
│   │   ├── index_meta.json
│   │   └── store
│   ├── investigations_102017
│   │   ├── index_meta.json
│   │   └── store
...
│   ├── newInsights_082017
│   │   ├── index_meta.json
│   │   └── store
│   ├── newInsights_092017
│   │   ├── index_meta.json
│   │   └── store
│   ├── newInsights_102017
│   │   ├── index_meta.json
│   │   └── store
│   ├── playbooks
│   │   ├── index_meta.json
│   │   └── store
...
└── partitionsData
    ├── demisto_082017.db
    ├── demisto_092017.db
    └── demisto_102017.db
				

Follow these steps to create data archive folders.

Note

In a distributed database deployment, first stop the app server and then the databases. Then run this procedure on each database that contains incidents. By default, you only need to run this procedure on the secondary nodes. However, in distributed database deployments that were converted from a single server deployment, there is incident data in the main database. Therefore, you also need to run this procedure on the main database.

  1. Stop the Cortex XSOAR service using the following command.

    $ sudo service demisto stop

  2. Create the following directories.

    The following steps use the year 2019 as an example.

    mkdir /var/lib/demisto-archive

    mkdir /var/lib/demisto-archive/archived-2019

  3. Navigate to the /var/lib/demisto-archive/ folder.

    cd /var/lib/demisto-archive/

  4. Move the data you want to archive to the archive directory. The following command moves all folders that have a mmyyyy suffix.

    mv /var/lib/demisto/data/**/*_<date_to_archive>* /var/lib/demisto-archive/archived-2019

    For example:

    mv /var/lib/demisto/data/**/*_092019* /var/lib/demisto-archive/

    Multi-tenant example:

    mv /var/lib/demisto/tenants/acc_{TENANT_NAME}/data/**/*_<date_to_archive>* var/lib/demisto-archive/archived-2019

    Note

    If the mv /var/lib/demisto/data/**/*_<date_to_archive>* /var/lib/demisto-archive/archived-2019 command does not work in your environment, archive the indices and partition separately. First create the following directories:

    mkdir /var/lib/demisto-archive/archived-2019/demistoidx/

    mkdir /var/lib/demisto-archive/archived-2019/partitionsData/

    Then, archive each index and the partition that has a mmyyyy suffix. The following are the commands to archive the current indices and the partition:

    sudo mv /var/lib/demisto/data/demistoidx/entries_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/demistoidx/evidences_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/demistoidx/incidents_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/demistoidx/invTaskIdx_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/demistoidx/investigations_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/demistoidx/newInsights_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/demistoidx/todosTask_082019 /var/lib/demistoarchive/archived-2019/demistoidx/

    sudo mv /var/lib/demisto/data/partitionsData/demisto_082019.db /var/lib/demisto-archive/archived-2019/partitionsData/demisto_082019.db

  5. Create the compressed archive of your selected files and folders using the following tarball command:

    $ tar -cvzf demisto-2019-archive.tar.gz /var/lib/demisto-archive/archived-2019

  6. Start the Cortex XSOAR service.

    $ sudo service demisto start