Troubleshoot Engines - User Guide - 2 - Cortex XPANSE - Cortex - Security Operations

Cortex Xpanse Expander User Guide
Product
Cortex XPANSE
Version
2
Creation date
2024-08-29
Last date published
2024-10-07
Category
User Guide
Solution
Cloud
Abstract

Troubleshoot engines by accessing logs and viewing errors.

When troubleshooting engines, access the logs from Settings → Configurations → Engines and select the engine from which you want to download the logs.

Debug Engines

The d1.log field appears whenever an engine is running. The d1.log field contains information necessary for your customer success team to debug any engine related issue. The field displays any error, as well as noting whether the engine is connected.

engine-debug.png

Engine 443 Error

This error might occur when a connection is established between an engine and , because, by default, Linux does not allow processes to listen on low-level ports.

Error Message

listen tcp :443: bind: permission denied

Solution

  • In the d1.conf file, change the port number to a higher one, for example, 8443.

  • Run this command: sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/binary. After running this command the server should be able to bind to low-numbered ports.

Troubleshoot Engine Installation
Abstract

Troubleshoot a failed engine installation.

After installing the engine, check that the engine is connected to the Cortex Xpanse tenant and that it is running.

  1. Go to Settings → Configurations → Data Broker → Engines and verify that the engine is connected.

  2. If the engine is not connected, run the following command on the engine server to check if the engine service is running.

    sudo systemctl status d1

  3. Access the d1 log on the engine server.

    sudo tail -f /var/log/demisto/d1.log

    • If the engine service is not running, and there’s nothing relevant in the log, run journalctl on the engine server to understand why the installation failed.

    • If the engine service is running, review the errors to see if the engine is failing to connect or if there are other issues (ignore all errors related to \d2ws, since this is not the same as d1ws.) Most often, the server address is incorrect and you will see an error like this:

      error Cannot connect to [wss://<mainServerIP/HostName>/d1ws]: wss://<mainServerIP/HostName>/d1ws: dial tcp: lookup localhost: no such host. . Waiting 3 seconds. Will try until…

      In this case, navigate to /usr/local/demisto/d1.conf and change the EngineURLs parameter to an address the engine can reach. Check the addresses at the beginning of the upgrade_engine.sh file and update them to be the same as in the conf file. The addresses should be a comma-separated list.

      Note

      You can ignore the following error: Cannot create folder '/var/lib/demisto'

      The configurations that might affect the upgrade_engine.sh script are the following variables located at the beginning of the script:

      • SERVER_URLS

      • TRUST_ANY_CERT

      If you make a change to the baseURLs configuration, you must apply the change in /usr/local/demisto/d1.conf AND in /usr/local/demisto/upgrade_engine.sh under the SERVER_URLS var.

      If you make a change in the engine.connection.trust_any_certificate configuration, you must apply the change in /usr/local/demisto/upgrade_engine.sh as follows:

      • If the engine.connection.trust_any_certificate configuration was set to true (trust any certificate), set the TRUST_ANY_CERT variable to -k.

      • If the engine.connection.trust_any_certificate configuration was set to false, the TRUST_ANY_CERT variable should be blank (““).

  4. To check the connectivity from the engine to the Cortex Xpanse tenant, see Troubleshoot Engine Connectivity.Troubleshoot Engine Connectivity

  5. If the installation issue remains, open a support case with logs from the engine.

    1. On the engine server, in /usr/local/demisto/d1.conf, set "LogLevel": "debug”.

    2. Restart the d1 service and let it run for a few minutes.

      sudo systemctl restart d1

    3. Capture a journalctl:

      journalctl --since "1 day ago" > engineTroubleshootingJournalctl.log

    4. On the engine server, tar up the log, conf, journalctl, and install log on the engine.

      tar -cvzf engineLogs.tar.gz /var/log/demisto /usr/local/demisto/d1.conf /tmp/demisto_install.log engineTroubleshootingJournalctl.log

Troubleshoot Engine Upgrades
Abstract

Troubleshoot failed engine upgrades. Manually upgrade Cortex Xpanse engines.

During an upgrade, the upgrade file is sent to the engine server. A cron job running on the engine server checks if that file exists. The most common upgrade error is that the job is not running so the new installer does not run.

  1. SSH to the machine.

  2. Check the d1 service status on the engine server. It is possible that it stopped or doesn't exist.

    sudo systemctl status d1

  3. Access the installer log on the engine server and review the error.

    sudo vi /tmp/demisto_install.log

  4. Rerun the installer on the engine using one of the following options. You can open a second window and run watch df -h. If the problem seems to be disk space, you should resolve the disk space issue and then rerun the installer.

    1. Option 1

      1. Download the installer from the user interface and copy it to the engine.

      2. Add the following commands:

        sudo chmod +x installer.sh

        sudo ./installer.sh -- -y

    2. Option 2

      1. Verify that /usr/local/demisto/d1_upgrade.sh exists.

        sudo chmod +x /usr/local/demisto/d1_upgrade.sh

        sudo /usr/local/demisto/d1_upgrade.sh

      2. If d1_upgrade.sh does not exist, check if /usr/local/demisto/archived_d1_upgrade.sh exists and that it was created at the time of the attempted upgrade.

        If the file exists and was created at the time of the attempted upgrade, run the following on the engine server:

        sudo chmod +x /usr/local/demisto/d1_upgrade_archive.sh

        sudo /usr/local/demisto/d1_upgrade_archive.sh

Troubleshoot Engine Connectivity
Abstract

Troubleshoot engine connectivity from the engine to the endpoint, including an Engine 443 error.

The following provides instructions for troubleshooting connectivity issues from the engine to the endpoint.

  1. Follow the instructions in network troubleshooting.

  2. Ensure that the engine can reach the endpoint by running the following command on the server engine.

    sudo curl -kvv <endpointURL>

  3. If the engine could not reach the endpoint, try the IP with curl instruction adding the http(s)//, or try using ping.

    If this works, add the IP to the /etc/hosts file with the hostname and try to reach the endpoint again by running the following command on the engine server

    sudo curl -kvv <endpointURL>

    If this still fails, then this is an issue of connectivity between the engine and endpoint and you need to resolve this with your networking team.

  4. Once connectivity has been confirmed via curl:

    • Try connecting within Docker without passing host networking.

      docker run -it --rm demisto/netutils:1.0.0.6138 curl -kvv <endpointURL>

      If this succeeds but the integration still fails, it could be an integration credentials issue. In that case, open a support case.

    • If without passing host networking fails, run the following:

      docker run -it --rm --network=host demisto/netutils:1.0.0.6138 curl -kvv <endpointURL>

      If this succeeds, add "python.pass.extra.keys": "--network=host" to /usr/local/demisto/d1.conf and retest the integration.

      If you see a Docker or Selinux issue, see Troubleshoot Docker Networking Issues.Troubleshoot Docker Networking Issues

  5. If the installation issue remains, open a support case with logs from the engine.

    1. On the engine server, in /usr/local/demisto/d1.conf, set "LogLevel": "debug”.

    2. Restart the d1 service and let it run for a few minutes.

      sudo systemctl restart d1

    3. Capture a journalctl:

      journalctl --since "1 day ago" > engineTroubleshootingJournalctl.log

    4. On the engine server, tar up the logs, conf, journalctl, and install log on the engine.

      tar -cvzf engineLogs.tar.gz /var/log/demisto /usr/local/demisto/d1.conf /tmp/demisto_install.log engineTroubleshootingJournalctl.log

Engine 443 Error

This error might occur when a connection is established between an engine and the Cortex Xpanse tenant, because, by default, Linux does not allow processes to listen on low-level ports.

Error Message

listen tcp :443: bind: permission denied

Solution

  • In the d1.conf file, change the port number to a higher one, for example, 8443.

  • Run this command: sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/binary. After running this command the server should be able to bind to low-numbered ports.

Troubleshoot Integrations Running on Engines
Abstract

Troubleshoot integrations running on an engine.

The following are common errors that occur when integrations are running on an engine:

Troubleshoot Engine Import Error or Invalid Syntax Error
Abstract

Troubleshoot engine import error or invalid syntax error when running an integration on an engine.

When running an integration on an engine, the most common errors are:

  • Broken Pipe

  • “ ImportError: No module named...

  • Invalid syntax

  • Script failed to run: exec: “python”: executable file not found in $PATH (2603)

These errors could indicate that the engine is not using Docker.

  1. Use SSH to access the engine server.

  2. Make sure Docker is healthy.

    1. Ensure that Docker is installed and is running.

      sudo systemctl status docker

      If the Docker status is not good, restart your Docker.

      sudo systemctl restart docker

    2. Ensure Docker can run a container.

      sudo docker run hello-world

      If this fails, reinstall your Docker.

  3. Access the d1.conf file on the engine server.

    sudo vi /usr/local/demisto/d1.conf

  4. Add the "python.engine.docker": true configuration to the d1.conf file and remove any other configurations related to python and Docker, such as “python.executable.no.docker”.

  5. Restart the system on the engine server.

    sudo systemctl restart d1

  6. Retest the integration from the user interface. This may take a few minutes since it may need to pull the relevant Docker image.

Troubleshoot Permission Denied
Abstract

Troubleshoot engine permission denied.

A common error message you may see when running integrations on engines is something like: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.35/images/json?t.

  1. Determine if you are using a Docker group or Dockerroot group by running one of the following on the server engine:

    • ls -la /var/run/docker.sock

      The output from this command will show what user/group is running docker.sock. For example:

      srw-rw----. 1 root docker 0 Apr 12 20:32 /var/run/docker.sock

      shows that it’s a Docker group and not Dockerroot.

    • cat /etc/group | grep docker

      This command shows if you are running Docker or Dockerroot.

    Note

    Docker CE installations typically run Docker, while Docker EE installations typically run Dockerroot.

  2. To fix a Docker user, run the following commands on the server engine:

    1. sudo groupadd docker

    2. sudo usermod -aG docker demisto

    3. sudo systemctl restart docker

    4. sudo systemctl restart d1

  3. To fix a dockerroot user, run the following commands on the server engine:

    1. sudo groupadd dockerroot

    2. Set the dockerroot group in /etc/docker/daemon.json. For example: { "group": "dockerroot" }.

    3. sudo usermod -aG dockerroot demisto

    4. sudo chcon -Rt svirt_sandbox_file_t /var/lib/demisto/temp

    5. sudo systemctl restart docker

    6. sudo systemctl restart d1