Migrate Cortex XSOAR data from BoltDB to Elasticsearch.
Before beginning a migration from BoltDB to Elasticsearch, we recommend reviewing the following considerations.
To have Cortex XSOAR objects work with Elasticsearch, you must install Elasticsearch and Cortex SOAR v6.2 and above. First migrate your database using the migration tool, then your Cortex XSOAR environment, and then Upgrade the Server. When you run the migration tool, objects that are not stored in Elasticsearch are migrated to Elasticsearch.
The following are examples of Elasticsearch migration use cases. Note that if an object is already in Elasticsearch from previous versions, it remains in Elasticsearch when you migrate and upgrade.
Single-instance and multi-tenant high availability - starting with v6.1, Cortex XSOAR does not support data redundancy using multiple database nodes with BoltDB. To obtain data redundancy, you must migrate to Elasticsearch. In addition, moving to Elasticsearch is a prerequisite for full high availability (meaning, multiple app servers and data redundancy).
Performance - Large volume deployments should consider migrating to Elasticsearch. Review the Elasticsearch System Requirements to gain an understanding if Elasticsearch is appropriate for you.
Prerequisites
Before beginning the migration process, the following must be in place:
The Elasticsearch cluster must be configured. The Elasticsearch cluster must meet the Elasticsearch System Requirements for Cortex XSOAR. Ensure network connectivity between application server(s) and Elasticsearch.
Latency between each component must be below 100 ms. For optimal performance, we recommend 10 ms or lower. Verify latency between the following components:
Application server(s)
Elasticsearch cluster
(High Availability) Shared file system
Elasticsearch must be listening via HTTPS and the Cortex XSOAR servers must trust the certificate being presented by Elasticsearch. You have the option to use the flag
-elasticsearch-insecure=true
if required for the initial install.(Optional) - Authentication to Elasticsearch can be configured using a username/password or API key with the appropriate permissions.
(High Availability) - The Elasticsearch cluster should have at least three data nodes in the cluster. In addition, we recommend having at least three master-eligible nodes.
(Optional - High Availability) - You can create and mount the shared file system for the Cortex XSOAR Application Servers before or after the migration process.
(Optional - High Availability) - You can provision the additional Cortex XSOAR application servers before or after the migration process.
Migration Planning
The following should be considered when planning your migration:
Data must be migrated from oldest to newest.
If older data is not required (for example, data older than a year), it can be excluded using the
partitions-to-ignore
flags, or you can migrate just the newer partitions using thepartitions
flag.Do not make changes to server configurations, settings, or integrations until the migration is fully complete, and the Cortex XSOAR application server is running on Elasticsearch.
By default, Elasticsearch limits the max content of an HTTP request to 100 megabytes. Due to this limitation, the migration tool skips over objects that are larger than 100 megabytes. After the migration is complete, you can view a list of any large objects that were not migrated, and view them in the
_migration-results/large-objects
folder. If you need to migrate these large objects, you can configure your Elasticsearch http.max_content_length and run the migration tool with a higherobject-max-size
. Note: Most cloud providers do not allow users to configure thehttp.max_content_length
.
Migration Process
While it’s possible to migrate all data at the same time, the server must be stopped to perform the migration and it is difficult to estimate required downtime. To minimize downtime and resource consumption, we recommend performing the migration in two steps, described below.
Create a copy of the BoltDB database.
Download the Elasticsearch migration tool.
Migrate data from the copy, excluding the last three months.
Migrate the remaining three monthly partitions using the live data from the server.
Validate the migration.
Update the Cortex XSOAR server to use the Elasticsearch database.
Perform post-migration checks.
(High Availability) - Set up shared file system.
(High Availability) - Set up load balancer.
(High Availability) - Install additional app servers.
Detailed migration instructions are available in the following topics:
(Optional) Test Migration Time
Depending on the amount of data you need to migrate to Elasticsearch, the migration process can take some time. You have the option to test how long the migration will take, by migrating a copy of the last three months of data.
Change the
indexPrefix
key in the Elasticsearch section of thedemisto.conf
file. For example,"indexPrefix": "test1"
.Run the migration tool on a copy of the last three months of data and note the total time for the migration.
Delete the new indexes using the Elasticsearch API before running the live migration.