Migrate an Existing Elasticsearch Deployment - Administrator Guide - EoL - 6.5 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide
Product
Cortex XSOAR
Version
6.5
Creation date
2022-09-28
Last date published
2024-07-16
Category
Administrator Guide
End of Life > EoL
Abstract

Migrate a Cortex XSOAR deployment that already uses Elasticsearch for managing indicators.

Prior to Cortex XSOAR v6.2, Elasticsearch was used to manage indicators only. If you are currently running Cortex XSOAR v6.0 or v6.1, and use Elasticsearch to manage indicators only, you must migrate all of your existing content. It is no longer possible to use Elasticsearch only for indicator management.

In the BoltDB, data related to incidents and indicators is stored in partitions by month. To minimize downtime during the migration, we recommend you create a copy of the database, then migrate data that is older than three months from the copy, while continuing to work in your current environment. Once the initial migration is completed, you should then migrate the last three months.

Note

If you are working in an environment with remote repositories, you must run the migration separately on each environment. For example, if both your development and production environments are going to be used with an Elasticsearch database, you must migrate each of those environments, and each environment must use a different index prefix.

Run all commands from the Cortex XSOAR server machine. To migrate your data, you need to use the migration tool.

Note

You cannot run more than one migration tool process at a time.

Note

Always migrate older data before newer data. Migrating partitions out of order can cause duplicate incident ids.

Note

By default, the migration tool skips over objects larger than 100 megabytes. After the migration process runs, you can view the skipped large objects and determine whether to migrate them. For more information, see Validate the Migration to Elasticsearch.

  1. Upgrade the Cortex XSOAR server to 6.5, but do not start the server.

  2. Download the migration tool by appending downloadName=elasticsearch_migration_tool_6_5_0 to the end of the download link that you received, when installing Cortex XSOAR.

  3. Copy your database and migrate data from the copy database to Elasticsearch.

    It is recommended to copy your data up to the last 3 months, without any downtime. If you do not want to do this, go to step Step 4.

    1. Copy the Cortex XSOAR database by doing one of the following:

    2. Copy the demisto.conf file.

    3. Edit the copy of the demisto.conf file, by adding your Elasticsearch configuration.

      Ensure that elasticsearch is the top-level object in the demisto.config file (within the main curly brackets).

    4. Using demisto or sudo permissions, run the following command:

      sudo ./elasticMigrator -config-path <file path-of-copy-of-demisto.conf> -db-path <path-of-the-copy-of-the-demisto-database> -<flags>

      For a full list of the flags, see Migration Tool Flags. For example, to exclude the last 3 partitions from the migration, add the -only-old-partitions and -partitions-back flags.

      To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the -objects-to-ignore "newInsights" flag. If you already migrated audits in a previous version, you must run the migration with the -objects-to-ignore "newInsights, audits" flag.

      sudo ./elasticMigrator -config-path /usr/local/dev/copy_of_demisto.conf -db-path /usr/local/dev/lib_demisto_copy/data -only-old-partitions -partitions-back 3 -objects-to-ignore "newInsights"

      When you run the migration tool, parameter values specified in the demisto.conf file override values supplied for tool flags and default values. If no value exists in the demisto.conf file, values supplied in the tool flags override default values, but do not write the values to the demisto.config file.

    5. Complete steps 1 to 3 in Validate the migration.

  4. After the migration of the data is complete and validated, migrate your data from the active database to Elasticsearch.

    1. Create a backup copy of the demisto.conf file for your active database.

    2. Edit the demisto.conf as needed.

    3. Stop the Cortex XSOAR server.

      • CentOS: sudo systemctl stop demisto

      • Ubuntu: sudo service demisto stop

    4. Migrate the remaining months from the active database to Elasticsearch. Use the -partitions-back flag to specify the remaining partitions.

      Note

      To avoid overwriting indicators that might already exist in Elasticsearch, you must run the migration with the -objects-to-ignore "newInsights" flag. If you already migrated audits in a previous version, you must run the migration with the -objects-to-ignore "newInsights, audits" flag.

      For example, sudo ./elasticMigrator -partitions-back 3 -objects-to-ignore “newInsights” migrates the last three partitions, which would include the current month and the previous two months, as well as the main partition.

  5. Validate the migration (all steps).