Elasticsearch Migration Overview - Administrator Guide - 6.10 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide
Product
Cortex XSOAR
Version
6.10
Creation date
2022-10-13
Last date published
2024-10-07
End_of_Life
EoL
Category
Administrator Guide
Abstract

Migrate Cortex XSOAR data from BoltDB to Elasticsearch.

Before beginning a migration from BoltDB to Elasticsearch, we recommend reviewing the following considerations.

To have Cortex XSOAR objects work with Elasticsearch, you must install Elasticsearch and Cortex SOAR v6.2 and above. First migrate your database using the migration tool, then your Cortex XSOAR environment, and then Upgrade the Cortex XSOAR Server. When you run the migration tool, objects that are not stored in Elasticsearch are migrated to Elasticsearch.

The following are examples of Elasticsearch migration use cases. Note that if an object is already in Elasticsearch from previous versions, it remains in Elasticsearch when you migrate and upgrade.

  • Single-instance and multi-tenant high availability - starting with v6.1, Cortex XSOAR does not support data redundancy using multiple database nodes with BoltDB. To obtain data redundancy, you must migrate to Elasticsearch. In addition, moving to Elasticsearch is a prerequisite for full high availability (meaning, multiple app servers and data redundancy).

  • Performance - Large volume deployments should consider migrating to Elasticsearch. Review the Elasticsearch System Requirements to gain an understanding if Elasticsearch is appropriate for you.

Prerequisites

Before beginning the migration process, ensure the following:

  • The Elasticsearch cluster must be configured. The Elasticsearch cluster must meet the Elasticsearch System Requirements for Cortex XSOAR. Verify network connectivity between application server(s) and Elasticsearch.

  • Latency between each component must be below 100 ms. For optimal performance, we recommend 10 ms or lower. Verify latency between the following components:

    • Application server(s)

    • Elasticsearch cluster

    • (High Availability) Shared file system

  • Elasticsearch must be listening via HTTPS and the Cortex XSOAR servers must trust the certificate being presented by Elasticsearch. You have the option to use the flag -elasticsearch-insecure=true if required for the initial install.

  • (Optional) - Authentication to Elasticsearch can be configured using a username/password or API key with the appropriate permissions.

  • (High Availability) - The Elasticsearch cluster should have at least three data nodes in the cluster. In addition, we recommend having at least three master-eligible nodes.

  • (Optional - High Availability) - You can create and mount the shared file system for the Cortex XSOAR application servers before or after the migration process.

  • (Optional - High Availability) - You can provision the additional Cortex XSOAR application servers before or after the migration process.

Migration Planning

The following should be considered when planning your migration:

  • Data must be migrated from oldest to newest.

  • If older data is not required (for example, data older than a year), it can be excluded using the partitions-to-ignore flags, or you can migrate just the newer partitions using the partitions flag.

  • Do not make changes to server configurations, settings, or integrations until the migration is fully complete, and the Cortex XSOAR application server is running on Elasticsearch.

  • By default, Elasticsearch limits the max content of an HTTP request to 100 megabytes. Due to this limitation, the migration tool skips over objects that are larger than 100 megabytes. After the migration is complete, you can view a list of any large objects that were not migrated, and view them in the _migration-results/large-objects folder. If you need to migrate these large objects, you can configure your Elasticsearch http.max_content_length and run the migration tool with a higher object-max-size.

    Note

    Most cloud providers do not allow users to configure the http.max_content_length.

Migration Process

While it’s possible to migrate all data at the same time, the server must be stopped to perform the migration and it is difficult to estimate required downtime. To minimize downtime and resource consumption, we recommend performing the migration in two steps, described below.

  1. Create a copy of the BoltDB database.

  2. Download the Elasticsearch migration tool.

  3. Migrate data from the copy, excluding the last three months.

  4. Migrate the remaining three monthly partitions using the live data from the server.

  5. Validate the migration.

  6. Update the Cortex XSOAR server to use the Elasticsearch database.

  7. Perform post-migration checks.

  8. (High Availability) - Set up shared file system.

  9. (High Availability) - Set up load balancer.

  10. (High Availability) - Install additional app servers.

Detailed migration instructions are available in the following topics:

(Optional) Test Migration Time

Depending on the amount of data you need to migrate to Elasticsearch, the migration process can take some time. You have the option to test how long the migration will take, by migrating a copy of the last three months of data.

  1. Change the indexPrefix key in the Elasticsearch section of the demisto.conf file. For example, "indexPrefix": "test1".

  2. Run the migration tool on a copy of the last three months of data and note the total time for the migration.

  3. Delete the new indices using the Elasticsearch API before running the live migration.