Install additional app servers for a Cortex XSOAR high availability configuration.
To implement a full high availability configuration, you should install a load balancer, configure a shared directory, and install additional app servers.
Install a Load Balancer
Cortex XSOAR recommends using a load balancer or round-robin DNS server to ensure proper high availability and to avoid having a single point of failure for certain components, such as engines, communication tasks, and usage of the API.
Note
When connecting through the load balancer, you can validate the app server you are connected to by adding the /remoteServer/appID
to the URL https://<load balancer url>/remoteServer/app
.
In Cortex XSOAR, navigate to
→ → .In the Server Configuration section, enter the external host name in the External Host Name configuration key. This is the Cortex XSOAR address registered in the load balancer or DNS.
Update engines to connect through the load balancer.
Validate Additional App Servers
After adding additional application servers to a high availability environment, validate each server.
After adding application servers to your high availability deployment, we recommend verifying that each application server has the correct configuration.
Verify the shared file system was created and is owned by demisto user.
sudo ls -lh /<shared>/var/lib/demisto
Modify path as needed if you mounted the shared filed system at another location.
Confirm you see folders for artifacts, images, systemTools, etc.
Verify that the temp folder on the new application server contains folders for tempSystemContent, script directories, etc.
ls -lh /var/lib/demisto/temp
Modify path as needed if you choose a different temp folder location.
Temp folder must be owned by
demisto:demisto
Verify that the Cortex XSOAR license file is in the shared file system folder. If not, upload the file.
Applies only to first application server.
Validate the
/etc/demisto.conf
files are the same across all application servers.Only the
AppID
andInClusterHostName
keys should be different per application server.Log in to the new Cortex XSOAR app server
https://<
.app server
>Install the Elasticsearch Monitoring content pack and validate you can view the Elasticsearch Monitoring dashboard to check Elasticsearch indexes for shards, replicas, and cluster status.
Applies only to first application server.
Go to
→ →Confirm that two or more application servers are showing online.
Create a new incident and verify you can view the incident across all application servers.
Configure a Shared Directory
Configure a shared directory, using the network file sharing solution of your choice, on each Cortex XSOAR app server. Migrate the /var/lib/demisto
directory to the shared file system before installing additional application servers.
Note
If you are using a location that is different from the default /var/lib/demisto
, you must install the additional app servers using the -data-dir
flag.
Stop the service on the first Cortex XSOAR application server.
Migrate the
/var/lib/demisto
directory, without the/var/lib/demisto/temp
subdirectory, to the shared file system.Copy the
demisto.lic
file from/usr/local/demisto
to the shared directory.cp /usr/local/demisto/demisto.lic /<share>/var/lib/demisto
Modify
/etc/demisto.conf
to reference the shared file system, using the example below.The
folders.lib
key must point to the location of the shared file system.The
folders.temp
key must be local to the individual app server and not shared. To change the location of the temp directory, edit thefolders.temp
key in thedemisto.conf
file.( Multi-tenant) - For a multi-tenant deployment, the
folders.temp
conf key must be updated per tenant on each host. The tenant conf file can be found at/usr/local/demisto/tenants/acc_{
tenant_name
}/server.confAdd or modify the license key at the end of
/etc/demisto.conf
."folders": { "lib": "/<share>/var/lib/demisto", "temp": "/var/lib/demisto/temp" }, "license": { "file": { "path": "/<shared>/var/lib/demisto" } },
Start the service on the first Cortex XSOAR app server.
Verify that you see the shared folder
/<share>/var/lib/demisto
ls -lh /<share>/var/lib/demisto
Validate Additional App Servers
After adding additional application servers to a high availability environment, validate each server.
After adding application servers to your high availability deployment, we recommend verifying that each application server has the correct configuration.
Verify the shared file system was created and is owned by demisto user.
sudo ls -lh /<shared>/var/lib/demisto
Modify path as needed if you mounted the shared filed system at another location.
Confirm you see folders for artifacts, images, systemTools, etc.
Verify that the temp folder on the new application server contains folders for tempSystemContent, script directories, etc.
ls -lh /var/lib/demisto/temp
Modify path as needed if you choose a different temp folder location.
Temp folder must be owned by
demisto:demisto
Verify that the Cortex XSOAR license file is in the shared file system folder. If not, upload the file.
Applies only to first application server.
Validate the
/etc/demisto.conf
files are the same across all application servers.Only the
AppID
andInClusterHostName
keys should be different per application server.Log in to the new Cortex XSOAR app server
https://<
.app server
>Install the Elasticsearch Monitoring content pack and validate you can view the Elasticsearch Monitoring dashboard to check Elasticsearch indexes for shards, replicas, and cluster status.
Applies only to first application server.
Go to
→ →Confirm that two or more application servers are showing online.
Create a new incident and verify you can view the incident across all application servers.
Install Additional App Servers
Install additional Cortex XSOAR app servers using the following command:
sudo ./demistoserver-X.sh -- -ha -elasticsearch-url=<url>-elasticsearch-username=<elasticUsername> -elasticsearch-password=<elasticUserPassword> -cluster-address=<internal-app-server-address> -do-not-start-server
(Multi-tenant) - To install additional app servers in a multi-tenant deployment, add the -multi-tenant flag to the command.
Parameter
Description
demistoserver-X
The name of the Cortex XSOAR installer, where X is the version and build number.
elasticsearch-url
URL for the Elasticsearch DB (including prefix http/https and port).
elasticsearch-username
Username in Elasticsearch.
elasticsearch-password
Password for the Elasticsearch database user.
cluster-address
The app server’s internal address to be used for in cluster communication. Other app servers use this address to communicate with this specific Cortex XSOAR instance.
temp-folder="filepath/tmp"
In the shared file system for a high availability environment, the temp directory must be local and not shared.
ha
Verifies user has set up data-dir correctly as a shared folder for high availability, before installing Cortex XSOAR. Confirms demisto user and group exist, demisto user is in the demisto group, data-dir (default path: /var/lib/demisto) exists, and data-dir is owned by user demisto and group demisto.
data-dir
(Optional) The directory that is shared by all app servers in the cluster. Default: /var/lib/demisto.
y
(Optional) The flag that completes the installation silently by answering yes to the remaining installation questions. Default settings are applied where applicable, and the EULA is automatically accepted.
Validate that the shared file system is seen as being owned by the
demisto:demisto user
ls -lh /<share>/var/lib/demisto
Modify the
/etc/demisto.conf
file with the additions made to the first app server:Configure the
externalEntities
key, if it does not already exist.Configure the
lib
folder for the shared directory.Add the temp folder path outside of the mounted remote file system for better performance.
Update the license path.
Add the elastic index configuration matching the first server (if applicable).
Confirm Docker images, including any custom Docker images, are available on the new application server. If needed, manually migrate any images that are not available.
Start the server.
sudo service demisto start
Validate Additional App Servers
After adding additional application servers to a high availability environment, validate each server.
After adding application servers to your high availability deployment, we recommend verifying that each application server has the correct configuration.
Verify the shared file system was created and is owned by demisto user.
sudo ls -lh /<shared>/var/lib/demisto
Modify path as needed if you mounted the shared filed system at another location.
Confirm you see folders for artifacts, images, systemTools, etc.
Verify that the temp folder on the new application server contains folders for tempSystemContent, script directories, etc.
ls -lh /var/lib/demisto/temp
Modify path as needed if you choose a different temp folder location.
Temp folder must be owned by
demisto:demisto
Verify that the Cortex XSOAR license file is in the shared file system folder. If not, upload the file.
Applies only to first application server.
Validate the
/etc/demisto.conf
files are the same across all application servers.Only the
AppID
andInClusterHostName
keys should be different per application server.Log in to the new Cortex XSOAR app server
https://<
.app server
>Install the Elasticsearch Monitoring content pack and validate you can view the Elasticsearch Monitoring dashboard to check Elasticsearch indexes for shards, replicas, and cluster status.
Applies only to first application server.
Go to
→ →Confirm that two or more application servers are showing online.
Create a new incident and verify you can view the incident across all application servers.