Migrate an Existing Elasticsearch Deployment - Administrator Guide - 6.9 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide

Product
Cortex XSOAR
Version
6.9
Creation date
2022-09-29
Last date published
2024-05-22
End_of_Life
EoL
Category
Administrator Guide
Abstract

Migrate a Cortex XSOAR deployment that already uses Elasticsearch for managing indicators.

Prior to Cortex XSOAR v6.2, Elasticsearch was used to manage indicators only. If you are currently running Cortex XSOAR v6.0 or v6.1, and use Elasticsearch to manage indicators only, you must migrate all of your existing content. It is no longer possible to use Elasticsearch only for indicator management.

In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.

Note

  • If you are working in an environment with remote repositories, you must run the migration separately on each environment. For example, if both your development and production environments are going to be used with an Elasticsearch database, you must migrate each of those environments, and each environment must use a different index prefix.

  • Run all commands from the Cortex XSOAR server machine. To migrate your data, you need to use the migration tool.

  • You cannot run more than one migration tool process at a time.

  • Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident IDs.

  • By default, the migration tool skips over objects larger than 100 megabytes. After the migration process runs, you can view the skipped large objects and determine whether to migrate them. For more information, see Validate the Migration to Elasticsearch.

  1. Upgrade the Cortex XSOAR server to the desired version, but do not start the server.

  2. Download the migration tool by appending downloadName=elasticsearch_migration_tool_6_X_X to the end of the download link that you received, when installing Cortex XSOAR. Replace X_X with the version number.

  3. Copy your database and migrate data from the copy database to Elasticsearch.

    We recommend you copy your data up to the last three months, without any downtime. If you do not want to do this, go to step 4.

    1. Copy the Cortex XSOAR database by doing one of the following:

    2. Copy the demisto.conf file.

    3. Edit the copy of the demisto.conf file, by adding your Elasticsearch configuration.

      Ensure that elasticsearch is the top-level object in the demisto.config file (within the main curly brackets).

    4. Using demisto or sudo permissions, run the following command:

      sudo ./elasticMigrator -config-path <file path-of-copy-of-demisto.conf> -db-path <path-of-the-copy-of-the-demisto-database> -<flags>

      For a full list of the flags, see Migration Tool Flags. For example, to exclude the last three partitions from the migration, add the -only-old-partitions and -partitions-back flags.

      To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the -objects-to-ignore "newInsights" flag. If you already migrated audits in a previous version, you must run the migration with the -objects-to-ignore "newInsights, audits" flag.

      sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -only-old-partitions -partitions-back 3 -objects-to-ignore "newInsights"

      When you run the migration tool, parameter values specified in the demisto.conf file override values supplied for tool flags and default values. If no value exists in the demisto.conf file, values supplied in the tool flags override default values, but do not write the values to the demisto.config file.

    5. Complete steps 1 to 3 in Validate the migration.

  4. After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.

    1. Create a backup copy of the demisto.conf file for your active database.

    2. Edit the demisto.conf as needed.

    3. Stop the Cortex XSOAR server.

      • CentOS: sudo systemctl stop demisto

      • Ubuntu: sudo service demisto stop

    4. Migrate the remaining months from the active database to Elasticsearch. Use the -partitions-back flag to specify the remaining partitions.

      Note

      To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the -objects-to-ignore "newInsights" flag. If you already migrated audits in a previous version, you must run the migration with the -objects-to-ignore "newInsights, audits" flag.

      For example, sudo ./elasticMigrator -partitions-back 3 -objects-to-ignore “newInsights” migrates the last three partitions, which would include the current month and the previous two months, as well as the main partition.

  5. Validate the migration (all steps).