Free up disk space by archiving Cortex XSOAR folders to condense the unused data within them.
Archive Incident, Entry, and Indicator Data
To free up disk space, you can archive compressed folders to save unused or old data. It is recommended to archive folders and not delete them permanently.
Cortex XSOAR supports full archiving of incident, entry, and indicator data on a monthly basis. For Elasticsearch deployments, see Archive Data with Elasticsearch. For Hosted Service deployments, contact Customer Support.
Locate the folders that reside in the following location (where Cortex XSOAR is installed), /var/lib/demisto/data/.
Note
Although the folders reside in /var/lib/demisto/data/, Do Not save the backup folders under /var/lib/demisto/.
The instructions below refer to paths for a single server deployment. For multi-tenant, you archive data on a per tenant basis. The following are the equivalent paths for multi-tenant:
Single Server | Multi-Tenant |
|---|---|
/var/lib/demisto/ | /var/lib/demisto/tenants/acc_{ |
/var/lib/demisto/data/ | /var/lib/demisto/tenants/acc_{ |
/var/lib/demisto-archive/ | /var/lib/demisto-archive/acc_{ |
The following data folder and files can be found in this folder:
demisto.db: The database for all playbooks and automation (not anything related to incidents and insights).demistoidx: Indexing of the system.partitionsData: Data of incidents, insights, and entries separated by month resolution.If you archive indexes you need to archive the matching partitions, so they are not rebuilt.
The following is an example of how the folders and filenames will appear in your system.
$ tree /var/lib/demisto/data
├── demisto.db
├── demistoidx
│ ├── accounts
│ │ ├── index_meta.json
│ │ └── store
...
│ ├── entries_082017
│ │ ├── index_meta.json
│ │ └── store
│ ├── entries_092017
│ │ ├── index_meta.json
│ │ └── store
│ ├── entries_102017
│ │ ├── index_meta.json
│ │ └── store
│ ├── evidences
│ │ ├── index_meta.json
│ │ └── store
│ ├── incidents_082017
│ │ ├── index_meta.json
│ │ └── store
│ ├── incidents_092017
│ │ ├── index_meta.json
│ │ └── store
│ ├── incidents_102017
│ │ ├── index_meta.json
│ │ └── store
│ ├── investigations_082017
│ │ ├── index_meta.json
│ │ └── store
│ ├── investigations_092017
│ │ ├── index_meta.json
│ │ └── store
│ ├── investigations_102017
│ │ ├── index_meta.json
│ │ └── store
...
│ ├── newInsights_082017
│ │ ├── index_meta.json
│ │ └── store
│ ├── newInsights_092017
│ │ ├── index_meta.json
│ │ └── store
│ ├── newInsights_102017
│ │ ├── index_meta.json
│ │ └── store
│ ├── playbooks
│ │ ├── index_meta.json
│ │ └── store
...
└── partitionsData
├── demisto_082017.db
├── demisto_092017.db
└── demisto_102017.db
Follow these steps to create data archive folders.
Note
In a distributed database deployment, first stop the app server and then the databases. Then run this procedure on each database that contains incidents. By default, you only need to run this procedure on the secondary nodes. However, in distributed database deployments that were converted from a single server deployment, there is incident data in the main database. Therefore, you also need to run this procedure on the main database.
Stop the Cortex XSOAR service using the following command.
$ sudo service demisto stopCreate the following directories.
The following steps use the year 2019 as an example.
mkdir /var/lib/demisto-archivemkdir /var/lib/demisto-archive/archived-2019Navigate to the
/var/lib/demisto-archive/folder.cd /var/lib/demisto-archive/Move the data you want to archive to the archive directory. The following command moves all folders that have a mmyyyy suffix.
mv /var/lib/demisto/data/**/*_<date_to_archive>* /var/lib/demisto-archive/archived-2019For example:
mv /var/lib/demisto/data/**/*_092019* /var/lib/demisto-archive/Multi-tenant example:
mv /var/lib/demisto/tenants/acc_{TENANT_NAME}/data/**/*_<date_to_archive>* var/lib/demisto-archive/archived-2019Note
If the
mv /var/lib/demisto/data/**/*_<date_to_archive>* /var/lib/demisto-archive/archived-2019command does not work in your environment, archive the indices and partition separately. First create the following directories:mkdir /var/lib/demisto-archive/archived-2019/demistoidx/mkdir /var/lib/demisto-archive/archived-2019/partitionsData/Then, archive each index and the partition that has a mmyyyy suffix. The following are the commands to archive the current indices and the partition:
sudo mv /var/lib/demisto/data/demistoidx/entries_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/demistoidx/evidences_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/demistoidx/incidents_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/demistoidx/invTaskIdx_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/demistoidx/investigations_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/demistoidx/newInsights_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/demistoidx/todosTask_082019 /var/lib/demistoarchive/archived-2019/demistoidx/sudo mv /var/lib/demisto/data/partitionsData/demisto_082019.db /var/lib/demisto-archive/archived-2019/partitionsData/demisto_082019.dbCreate the compressed archive of your selected files and folders using the following tarball command:
$ tar -cvzf demisto-2019-archive.tar.gz /var/lib/demisto-archive/archived-2019Start the Cortex XSOAR service.
$ sudo service demisto start