Elasticsearch Best Practices - Installation Guide - 6.8 - Cortex XSOAR - Cortex - Security Operations

It is important to utilize Elasticsearch versatile behavior using shards and replicas. If you are using more than 1 node, we recommend that you set the elasticsearch.defaultShardsPerIndex and elasticsearch.defaultReplicasPerIndex to 1 and 2, respectively. You can also use elasticsearch.shards and elasticsearch.replicas to configure specific number of shards and replicas respectively on each index.

For example, setting the following values in demisto.conf sets 2 replicas for each shard for the incidents index:

"elasticsearch": { "shards": { "common-incident": 1 }, "replicas": { "common-incident": 2 } } }

Suggested configuration based on the number of data nodes in the Elasticsearch cluster:

Cortex XSOAR uses internal batch to Elasticsearch to avoid heavy requests and large indexing latency. You can increase the batch size using elasticsearch.innerBatchSize to increase performance and speed, or decrease the value if Elasticsearch is not optimized.

Due to Cortex XSOAR’s high usage of Elasticsearch, we recommend having the lowest latency possible between the Cortex XSOAR server and the Elasticsearch nodes configured. We also recommend having low latency between nodes to avoid slow indexing or possible data loss on outages. Any latency between Cortex XSOAR and Elasticsearch will greatly impact performance.

Elasticsearch uses an inner memory management implemented in JAVA that is called the heap size. It defaults to 1GB and Cortex XSOAR recommends you increase the heap size to 50% of machine's memory, or more if the machine is dedicated to Elasticsearch.

Elasticsearch uses a refreshing mechanism to flush updated documents to the disk every configured seconds. We recommend setting the configuration index to 1 second. You can use the Cortex XSOAR demisto.conf file to set a specific refreshing interval in seconds for each index using elasticsearch.refreshIntervals.

For example, the following sets the configuration index to 1 second:

"elasticsearch": { "refreshIntervals": { "common-configuration": 1s } }

We recommend using the most recent supported version of Elasticsearch, to ensure you have the latest features and all security updates.

Circuit breakers in Elasticsearch are enabled by default. Verify they are enabled and configured correctly, specifically the request circuit breaker and the parent circuit breaker. These circuit breakers will prevent data nodes from crashing on high memory consumption and will also drop certain requests that Cortex XSOAR will retry later.

Elasticsearch requires a large number of open file descriptors. Verify that the limit on the number of open file descriptors is sufficient to handle a high load of requests, to avoid possible data loss or service interruptions.