Elasticsearch Configurations - Administrator Guide - 6.10 - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Administrator Guide

Product
Cortex XSOAR
Version
6.10
Creation date
2022-10-13
Last date published
2024-12-05
End_of_Life
EoL
Category
Administrator Guide
Abstract

Improve Elasticsearch performance. List of all of the supported configuration elements for Elasticsearch in the demisto.conf file.

Cortex XSOAR uses the demisto.conf file to store the Elasticsearch configuration details. When setting up Elasticsearch for a new Cortex XSOAR installation or migrating data from BoltDB to Elasticsearch, you need to add connection information for the Elasticsearch database to the demisto.conf file. You can also add optional parameters including shards, replicas, and refresh intervals for the indices that will be created within Elasticsearch

For more information about Elasticsearch configuration for Cortex XSOAR, we recommend reviewing Elasticsearch Best Practices before editing the configuration file.

Edit the /etc/demisto.conf file to add the new Elasticsearch configuration. At a minimum, you must provide the URL and a username/password or API key. You can also provide an indexPrefix if you want the indices to be created with a different prefix.

Example

{
     "elasticsearch": {
           "username": "elastic",
           "password": "changeme",
           "apiKey": "",
           "url": "https://readyelasticone:9200",
           "enabled": true,
           "insecure": true,
           "indexPrefix": "",
           "responseHeaderTimeoutSeconds": 120,
           "shards": {
               "common-invplaybook": 3,
               "common-entry": 3
           },
           "replicas": {
               "common-invplaybook": 1,
               "common-entry": 1
           },
           "defaultShardsPerIndex": 1,
           "defaultReplicasPerIndex": 2,
           "refreshIntervals": {
               "*": "30s",
               "common-configuration": "1s",
               "common-incident": "1s"
           }
       } 
}

Note

If you move to Elasticsearch without migrating data or do a migration that is only partially successful, you need to add the externalEntities key to the Server key. If you successfully migrate your data, the key is added automatically.

"Server": {
             "HttpsPort": "443",
             "externalEntities": "incident,indicator,audit,configuration"  // store all types of data objects in Elasticsearch
       },

The following table lists all the possible Elasticsearch configurations supported in the demisto.conf file. Use these elements to tweak Elasticsearch performance.

Name

Type

Description

url

string

The url, or urls, comma-separated to an Elasticsearch node. The url includes the schema and port.

Default: http://localhost:9200

Example: “url”: "http://localhost:9200"

enabled

boolean

Set to true to enable the use of Elasticsearch as the database.

Default: false

Example: “enabled”: true

indexPrefix

string

The index prefix to address when creating, writing, and reading from Cortex XSOAR indices.

Default: N/a

Example: “indexPrefix”: “xsoar”

proxy

boolean

Set to true if Elasticsearch is used behind a proxy service.

Default: false

Example: “proxy”: true

username

string

The Elasticsearch username to establish a connection.

* required (unless API key is provided)

Default: N/a

Example: “username”: “elastic”

password

string

A plain, tommed, or encrypted password for Elasticsearch to establish connection.

* required (unless API key is used)

Default: N/a

Example: “password”: “123”

apiKey

string

The Elasticsearch API key to establish connection.

* required (unless a username and password are used)

Default: N/a

Example: “apiKey”: “ani1”

insecure

boolean

Allow an insecure connection to Elasticsearch for unsigned certificates.

Default: false

Example: “insecure”: true

responseHeaderTimeoutSeconds

number

The number of seconds for timeout while awaiting response headers from Elasticsearch.

Default: 0

Example: “responseHeaderTimeoutSeconds”: 60

dialerTimeoutSeconds

number

The number of seconds for timeout while awaiting to establish connection to Elasticsearch.

Default: 30

Example: “dialerTimeoutSeconds”: 60

maxIdleConnections

number

The maximum number of idle connections to Elasticsearch that remain alive.

Default: 100

Example: “maxIdleConnections”: 400

idleConnectionTimeout

number

The number of seconds for an idle connection to be closed, if not used.

Default: 100

Example: “idleConnectionTimeout”: 200

maxRetriesOnConnectivity

number

The amount of maximum retries when connection is refused or not found from Elasticsearch.

Default: 10

Example: “maxRetriesOnConnectivity”: 50

shards

object

An index name for the number of shards for an object to determine shards allocation on each index.

Default: N/a

Example: “shards”: { “common-incident”: 1 }

replicas

object

An index name for the number of replica objects to determine the replicas allocation on each index.

Default: N/a

Example: “replicas”: { “common- incident”: 1 }

defaultShardsPerIndex

number

The default number of shards for each index created using Cortex XSOAR.

Default: N/a

Example: “defaultShardsPerIndex”: 1

defaultReplicasPerIndex

number

The default number of replicas for each index created using Cortex XSOAR.

Default: N/a

Example: “defaultReplicasPerIndex”: 2

totalFields

object

An index name for the number of maximum fields allowed in the index mapping.

Default: { “common-incident”: 2000, “common-indicator”: 2000, “common-evidence”: 2000 }

Example: { “common- incident”: 5000 }

refreshIntervals

object

An index name to number of seconds to perform refresh on the index. Use * for all indices.

Default: N/a

Example: { “common-configuration”: 30, “common-incident”: 1 }

innerBatchSize

number

The amount of maximum objects to send to Elasticsearch when using bulk updates.

Default: 250

Example: “innerBatchSize”: 500

maxRetriesOnVersionConflicts

number

Amount of retries to run on version conflict errors received from Elasticsearch.

Default: 10

Example: “maxRetriesOnVersionConflicts”: 30

maxResultWindow

number

Maximum allowed results to return from Elasticsearch searches.

Default: 10000

Example: “maxResultWindow”: 50000

aggregationMaxSize

number

Maximum allowed results to return from Elasticsearch aggregations.

Default: 1000

Example: “aggregationMaxSize”: 5000

lastFeedFetchLimit

number

Maximum allowed objects to store on the last feed fetch limit.

Default: 300000

Example: “lastFeedFetchLimit”: 100000

disableCreateIncidentForceIndex

boolean

Disable force refresh on new incidents to allow back-to-back searches using pre-processing scripts.

Default: false

Example: “disableCreateIncidentForceIndex”: true

template.patternPrefix

string

Use a custom template name prefix.

Default: indexPrefix

Example: “template.patternPrefix”: “template-name”

role.roles

array

Default cluster roles to provide new users in Elasticsearch created in Cortex XSOAR.

Default: [“manage”, “monitor”, “manage_own_api_key”]

Example: “role.roles”: [“manage”]

api.roles

array

Default cluster roles to provide new API keys in Elasticsearch created in Cortex XSOAR.

Default: [“manage”, “monitor”, “manage_own_api_key”]

Example: “api.roles”: [“manage”]

debug.enableQuery

boolean

Show Elasticsearch built queries before sending when using the log level debug.

Default: false

Example: “debug.enableQuery”: true