Engine Installation - User Guide - 2 - Cortex XPANSE - Cortex - Security Operations

Cortex Xpanse Expander User Guide
Product
Cortex XPANSE
Version
2
Creation date
2024-08-29
Last date published
2024-12-26
Category
User Guide
Solution
Cloud
Abstract

Install, deploy and configure Cortex Xpanse engines.

You can install engines on all Linux and Windows machines. Although engines are intended for Linux operating systems, they can be used on Windows, but Windows machines must be configured to run Linux containers. Docker or Podman must be installed before installing an engine. If you are using the Shell installer for an engine, Docker or Podman is installed automatically.

Engine Hardware Requirements

If your hard drive is partitioned, we recommend a minimum of 35 GB for the /var partition.

Component

Dev Environment Minimum

Production Minimum

CPU

8 CPU cores

16 CPU cores

Memory

16 GB RAM

32 GB RAM

Storage

50 GB

50 GB

Operating System Requirements

You can deploy a Cortex Xpanse engine on the following operating systems:

Operating System

Supported Versions

CentOS

7.x

Ubuntu

18.04, 20.04, 22.04

RHEL

7.x, 8.0, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7

Oracle Linux

7.x

Amazon Linux

2

Note

Centos 8.x reached End of Life (EOL) on December 31, 2021, and is no longer a supported operating system.

Engine Required URLs

You need to allow the following URLs for Cortex Xpanse engines to operate properly.

FUNCTION

SERVICE

PORT

DIRECTION

Integrations

Integration-specific ports

Outbound

Engine connectivity

HTTPS

443 (configurable)

Outbound

Engine Installation Types

Cortex Xpanse supports the following file types for installation on the engine machine:

  • Shell: For all Linux deployments, including Ubuntu, SUSE, RHEL, etc, except RHEL 7.x and CentOS 7.x Automatically installs Docker/Podman, downloads Docker/Podman images, enables remote engine upgrade, and allows installation of multiple engines on the same machine. For RHEL 7.x, see Install Docker Distribution for Red Hat on an Engine.

    The installation file is selected for you. Shell installation supports the purge flag, which by default is false.

    Note

    When upgrading a Shell type engine, you can use the Upgrade Engine feature in the Engines page. For CentOS 7, RHEL 7, or Amazon Linux 2 type engines, you need to upgrade these engine types using a zip type engine and not use the Upgrade Engine feature.

    If you use the Shell installer, Docker/Podman is automatically installed. We recommend using Linux and not Windows to be able to use the Shell Installer which installs all dependencies.

  • DEB: For Ubuntu operating systems.

  • RPM: RHEL operating systems.

    Note

    Use DEB and RPM installation when shell installation is not available. You need to install Docker or Podman and any dependencies. You need to install Docker or Podman and any dependencies. If installing on RHEL v7 or CentOS v7 you need to install Mirantis Container Runtime (formerly Docker Engine - Enterprise) or Red Hat's Docker distribution to run specific Docker dependent integrations and scripts.

  • Zip: Used for Windows and CentOS 7 machines.

  • Configuration: Configuration file for download. When you install one of the other options, this configuration file (d1.conf ) is installed on the engine machine.

Important

For DEB/RPM and Windows engines, Python (including 3.x) and containerization platform (Docker/Podman) must be installed and configured. For Docker or Podman to work correctly on an engine, IPv4 forwarding must be enabled.

Install an Engine
Abstract

Install, deploy and configure Cortex Xpanse engines.

When you install the engine, the d1.config is installed on the engine machine, which contains engine properties such as proxy, log level, log files, etc. If Docker/Podman is already installed, the python.engine.docker and powershell.engine.docker key is set to true. If Docker or Podman is not available when the engine is installed, the key is set to false. If so, you need to set the key to true. Verify that python.engine.docker and powershell.engine.docker configuration key is present in the d1.conf file.

For engines running on a Windows machine, add the following keys to the d1.config file:

  • The python.runner.loop.script.path configuration key with the path to the _script_docker_python_loop.py file (located in the engine’s installation folder). The path to the _script_docker_python_loop.py must be taken from WSL installed on the Windows machine (for example, /mnt/c/Users/<customer user>/Desktop/<customer engine folder>/_script_docker_python_loop.py).

  • The powershell.runner.loop.script.path configuration key with the WSL path to the _script_docker_powershell_loop.ps1 file (also located in engine’s installation folder).

After you install and deploy an engine, there are several ways that you can Manage Engines.

Before you begin

f you are using DEB, RPM or Zip installation, install Docker/Podman. To run Docker dependent integrations and scripts on CentOS v7, install Mirantis Container Runtime. If using RHEL v7, follow the steps in Install Docker Distribution for Red Hat on an Engine.Install Docker Distribution for Red Hat on an Engine

  1. Create an engine.

    1. Select Settings → Configurations → Data Collections → Engines → Create New Engine.

    2. In the Engine Name field, add a meaningful name for the engine.

    3. Select one of the installer types from the drop down list.

      • Shell

      • DEB

      • RPM

      • Zip

      • Configuration

      Tip

      For Linux systems it is recommended to use the Shell installer. If using CentOS 7.x, or Amazon Linux 2, use the Zip installer (see step 5).

    4. (Optional) (Shell only) Select the checkbox to enable multiple engines to run on the same machine.

      If you have an existing engine, you did not select the checkbox, and you want to install another engine on the same machine, you need to delete the existing engine.

    5. (Optional) Add any required configuration in JSON format.

    6. Click OK to create the engine.

  2. For Shell installation, do the following:

    1. Move the .sh file to the engine machine using a tool like SSH or PuTTY.

    2. On the engine machine, grant execution permission by running the following command:

      chmod +x /<engine-file-path>

    3. Install the engine by typing one of the following commands:

      With tools: sudo <engine-file-path>

      Without tools: sudo <engine-file-path> -- -tools=false

      If you receive a permissions denied error, it is likely that you do not have permission to access the /tmp directory.

  3. For RPM/DEB installation do the following:

    1. Move the file to the required machine using a tool like SSH or PuTTY.

    2. Type one of the following installation commands:

      Machine Type

      Install Command

      RHEL (RPM)

      sudo rpm -Uvh d1-2.5_15418-1.x86_64.rpm

      Ubuntu (DEB)

      sudo dpkg --install d1_xxx_amd64.deb

    3. Start the engine by running one of the following commands:

      Machine Type

      Start Command

      RHEL (RPM)

      sudo systemctl start d1

      Ubuntu (DEB)

      sudo service d1 restart

  4. For Zip file installation on Windows, do the following.

    1. Move the d1 zip file to the engine machine using a tool like WinSCP.

    2. Unzip the file and move it to any location you require.

    3. Open the file and run the d1_windows_amd64.exe file.

      Every time you want to connect to you need to run the D1 Application file.

  5. For Zip installation on CentOS 7.x or Amazon Linux 2, run the following commands:

    1. Create the engine folder.

      mkdir /usr/local/demisto

    2. Unzip the engine files to the folder created in step 2.

      unzip ./d1.zip -d /usr/local/demisto

    3. Allow the process to bind to low numbered ports.

      setcap CAP_NET_BIND_SERVICE=+eip /usr/local/demisto/d1_linux_amd64

    4. Run the engine process.

      /usr/local/demisto/d1_linux_amd64

  6. When the engine is connected, you can add the engine to a load balancing group by clicking Load-Balancing Group.

    If you want to add the engine to a new group, click Add to new group from the dropdown list.

    When the engine is in the load-balancing group, it cannot be used as an individual engine and does not appear when configuring an engine from the drop down list.

  7. (Optional) After installing the engine, you may want to set up a proxy, set up Docker hardening, configure the number of workers for the engine, etc. For more information, see Configure Engines.

Engine Offline Installation
Abstract

Install a Cortex Xpanse engine offline when you don’t have access to the Internet (tested on RHEL v8).

Use these instructions to install an engine on a machine without internet connectivity.

On a machine that has internet access, you need to download dependencies, docker images, and from the Cortex Xpanse tenant, the engine installation files. You then need to transfer and install the files to the machine without internet access.

Download Dependencies for Offline Installation

Install the following top level dependencies according to you operating system. These dependencies may be dependent upon other OS libraries.

Note

Always verify that your dependencies are updated and take into account that they might change across releases.

Download Docker Images Offline

To download docker images you need to use the download_packs_and_docker_images script to download the docker image according to the content pack integration you want to use, such as AWS-ILM, Cybereason, EWS, etc.

The download_packs_and_docker_images script enables you to download the latest content packs Docker images in a zip folder to your machine. The script is located in the Utils folder in the GIT Content repository. If you do not have access to the GIT Content repository, you can download the script from here. For detailed information and how to download the Docker images, see download packs offline.

Install an Engine Offline

  1. On a machine with internet access, download the following:

    1. Dependencies for your deployment type.

    2. Relevant Docker images.

  2. In the Cortex Xpanse tenant, download the engine installation file.

    1. Select Settings → Configurations → Data Broker → Engines → Create New Engine.

    2. In the Engine Name field, add a meaningful name for the engine.

    3. Select one of the installer types from the dropdown list.

      For Linux systems it is recommended to use the Shell installer.

    4. (Optional) If you want to add the engine to a load balancing group, from the dropdown list, select the group you want to add.

      The dropdown list only appears after you have created and connected an engine and created a load balancing group. To add the engine to a new group, select Add new group from the dropdown list.

      The engine cannot be used as an individual engine and does not appear when configuring an engine from the dropdown list.

    5. (Optional) (Shell only) Select the checkbox to enable multiple engines to run on the same machine.

      If you have an existing engine, you did not select the checkbox, and you want to install another engine on the same machine, you need to delete the existing engine.

    6. (Optional) Add any required configuration in JSON format.

    7. Click Create New Engine.

  3. On the machine you want to install the engine, do the following:

    1. Transfer the files downloaded in steps 1 and 2.

    2. Verify that the required dependencies in step 1a is installed successfully by running one of the following commands.

      • (Red Hat or CentOS) repoquery -a --installed

      • (Ubuntu or Debian) apt list --installed

    3. Install the engine.

      1. Grant execution permission by running the following command:

        chmod +x /<engine-file-path>

      2. Install the engine by running the following command:

        sudo ./d1-<engine-name>-<XSIAM-version>-xxxxxxx.sh -- -tools=false -do-not-start-engine=true

        For example, sudo ./d1-engine1-8.35-318874.sh -- -tools=false -do-not-start-engine=true

        If you receive a permissions denied error, it is likely that you do not have permission to access the /tmp directory.

    4. (Red Hat v8 & above) If you have not already done so, install and configure Podman, by following the steps in Migrate From Docker to Podman (from step 2 onwards).

    5. Load the Docker images that you downloaded in step 1b, by doing one of the following:

      • (Ubuntu, Debian, Red Hat v7 & below, or CentOS v7 & below) Run the following command:

        sudo docker load -i <YOUR_DOCKER_FILE>.zip

      • (Red Hat v8 & above) Do the following:

        1. Ensure that the docker file has demisto:demisto ownership.

        2. Ensure that you are in the root directory (cd /).

        3. Run the following commands:

          sudo -su demisto

          podman load -i <YOUR_DOCKER_FILE>.zip

        4. (Optional) To verify that images are able to run, use the podman images command. You can also run the podman images -q "demisto/python:1.3-alpine" command to validate specific images and identify any issues.

  4. Start the engine, by running the following command:

    sudo systemctl start d1

    Note

    For multiple engines the d1 service name may differ.

  5. (Optional) After installation has completed, do the following:

    1. Confirm that the engine status is active, by running the systemctl status d1 command.

    2. Validate that the engine is connected and running by going to Settings → Configurations → Data Broker → Engines.

    3. Run the engine on a sample integration. For example, go to Settings → Configurations → Data Collection → Automation & Feed Integrations and search for the Hello World (Community Contribution) integration. Add or edit the instance and in the Run on field, select the engine.

    4. In an Alert War Room, run a simple command to test that the engine is working properly using the integration.

      For example, !helloworld-say-hello name"Mamba"