Troubleshoot issues for Podman for Cortex XSOAR.
dbus-daemon Process Leak
Podman version 3.4.1 and lower has a known issue that dbus-daemon processes may leak when running in an environment containing the dbus-x11 OS package. The issue occurs when the dbus-x11 OS package is installed, for example when installing an X11 desktop environment like GNOME desktop on the host machine.If you experience this issue you see a large number of dbus-daemon processes owned by the demisto OS user. To check if you are affected by the issue, run the following command:
ps -fe | grep demisto | grep dbus-daemon
To fix this issue:
Remove the dbus-x11 OS package and dependent packages by running the following command:
sudo yum remove dbus-x11
After removal you can kill the leaked dbus-daemon processes by running the following OS command:
pgrep -u demisto dbus-daemon | xargs sudo kill
Invalid argument error
When Podman fails to run with an “Invalid argument” error, such as:
ERRO[0000] running `/usr/bin/newuidmap 15936 0 1029 1 1 165536 65536 65537 200000 65536`: newuidmap: write to uid_map failed: Invalid argument Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
This can be caused by duplicate lines for Cortex XSOAR in /etc/subuid
and /etc/subgid
.
To fix this issue:
Check if the
/etc/subuid
file contains multiple lines that start with the Cortex XSOAR username (usually demisto). For example:alice:100000:65536 demisto:165536:65536 demisto:200000:65536 splunk:331072:65536
If this is the case, edit the file as root, and remove the extra line(s) for Cortex XSOAR. The line you should keep is the one that ends with 200000:65536. Continuing with the above example, here is the end result:
alice:100000:65536 demisto:200000:65536 splunk:331072:65536
Repeat the above steps for the
/etc/subgid
file.
Verify Podman Installation
When encountering errors in Cortex XSOAR that are Podman related, such as:
failed to run "docker ps". stderr: [], err: [Timeout. Process killed (1400)
Timeout while waiting for pong response [error 'Read timed out (15s)
Error: error joining network namespace of container 06b8aec6eabe2e735128e3a72cb06c8ae2d97ade60a56ab555034442ea4e2a84: error retrieving network namespace at /tmp/podman-run-989/netns/cni-86dca01c-bd84-1aaf-85fb-72b659a8e42a: unknown FS magic on "/tmp/podman-run-993/netns/cni-86dca01c-bd84-1aaf-85fb-72b659a8e42a": 58465342
verify that Podman is running properly and consider whether to delete Podman data directories.
Verify that Podman is running properly with the
demisto
OS user, by performing the following steps:Change the OS user to
demisto
by running the following command:sudo su - -s /bin/bash demisto
Check that your system complies with the minimum requirements, and view general system information such as host architecture, CPU, OS, registries, container storage path, etc., by running the following command:
podman info
Check all active running containers, container names and IDs, by running the following command:
podman ps
Check that Podman is able to run a container, by running the following command:
podman run --rm -t demisto/python3:3.10.4.29342 echo "podman is working"
If any of the Podman commands are not working, try running with the
--log-level=debug
to receive additional details as to why it is failing. For example:podman -log-level=debug ps
podman --log-level=debug ps podman --log-level=debug run --rm -t demisto/python3:3.10.4.29342 echo "podman is working"
Reset the Podman Data Directories.
If the Podman commands in step 1 are failing you should clean the Podman working directories. Sometimes Podman's data directories get corrupted (for example, as a result of insufficient disk space).
Note
This step removes all Podman images including any custom images customers may have created.
Stop the Cortex XSOAR server by running the following command:
sudo systemctl stop demisto
Ensure that all Podman containers of the
demisto
user are stopped by running the following command:ps -fe | grep demisto | grep 'podman run'
If required, kill the running containers.
Delete the following directories (assuming the
demisto
OS user's home directory is at: /home/demisto)sudo rm -rf /home/demisto/.cache/containers/
sudo rm -rf /home/demisto/.local/share/containers/
sudo rm -rf /tmp/podman-run-$(id -u demisto)
sudo rm -rf /tmp/containers-user-$(id -u demisto)
sudo rm -rf /tmp/tmp/run-$(id -u demisto)
Note
$(id -u demisto)
is used to get thedemisto
user ID, which is part of the directory name. For example,/tmp/podman-run-993
Not all the directories above may be present.
Start the Cortex XSOAR server, by running the following command:
sudo systemctl start demisto
Verify that Podman is working properly with the
demisto
OS user by following step 1.
Report a Support Case for Installation issues
If the procedure set out in the Verify Podman Installation topic doesn't solve the Podman issue and you require assistance from Support, do the following:
Include the following files as part of the support case:
/etc/containers/storage.conf
/home/demisto/.config/containers/storage.conf
If the file does not exist, indicate that there is no such file.
/home/demisto/.config/containers/registries.conf
If the file does not exist, indicate that there is no such file.
Include the output of the following commands as the
demisto
user.Note
To change to the
demisto
OS user, run the following command:sudo su - -s /bin/bash demisto
podman info
podman images
podman --log-level=debug ps
podman --log-level=debug run --rm -t demisto/python3:3.10.4.29342 echo "podman is working"
Keyring Quota Exceeded Error
Script failed to run: Docker code runner got container error: [Docker code script is in inconsistent state, ... error: [exit status 126] stderr: [Error: OCI runtime error: crun: create keyring ...: Disk quota exceeded]
By default, Podman creates a keyring to be used by each container. The limit per user on the machine might be low and Podman can hit the limit when running more containers than the keyring limit.
To check the keyring usage, run the following operating system command: sudo cat /proc/key-users
. The command will return the usage for each UID. (To retrieve the demisto user UID, run id demisto
). The fourth column shows the number of keys used out of the total number available. For more information about keys, see https://www.kernel.org/doc/Documentation/security/keys.txt
You can either increase the limit of max keyrings (increasing to 1000 is safe and reasonable) per user as specified by your Linux vendor documentation or you can disable keyring creation by Podman. We recommend disabling keyring creation, unless keyrings are used by Podman in other applications on the machine.
To disable keyring creation by Podman, modify the containers.conf
file and add the option keyring = false
under the "[containers]" section. For more information, see https://github.com/containers/common/blob/main/docs/containers.conf.5.md
Unused Containers Taking Up Resources
In some cases, if the Podman process crashes or is killed abruptly it can leave containers on disk. You might see errors such as error allocating lock for new container: allocation failed; exceeded num_lock
when the maximum number of locks used to manage containers is exhausted due to the unused containers that remain.
To troubleshoot:
Change to the demisto operating system user
sudo su - -s /bin/bash demisto
.Run
podman ps -a -f status=exited
to check for unused containers.Clean up the unused containers
podman container cleanup --rm -a
.Note
When you run
podman container cleanup --rm -a
, you might see a message such as running or paused containers cannot be moved without force. The message can be safely ignored, as it only pertains to current running containers, which are not removed.After clean up, verify there are no remaining unused containers
podman ps -a -f status=exited
.
Permission issues with directories under /run path
When installing a Cortex XSOAR server or engine on a RHEL system (version 8 or later), or when running an integration on such a server or engine, you get a permission error for a path under /run
(for example /run/user/0
or /run/libpod
).
In RHEL 9 only: Make sure the
container-tools
meta-package is installed, by running:yum -y install container-tools
Run the following commands:
cp /etc/containers/storage.conf /home/demisto/.config/containers/storage.conf
chown demisto:demisto /home/demisto/.config/containers/storage.conf
chmod 600 /home/demisto/.config/containers/storage.conf
Edit
/home/demisto/.config/containers/storage.conf
.Under [storage], change
runroot
to some temporary directory that is accessible by user demisto.For example:
runroot = "/tmp/podman-run-xsoar"
Important
The
runroot
must be located under thetmpfs
file system type. This is required to clean Podman's run state on reboot and for performance reasons.Also under [storage], change
graphroot
(which is where container images are stored) to any location that is owned and accessible by user demisto. We recommend using this standard path:graphroot = "/home/demisto/.local/share/containers/storage"
Caution
Unlike the
runroot
, thegraphroot
must NOT be located under thetmpfs
file system type. Usingtmpfs
for thegraphroot
might corrupt container images, causing command executions to fail. It also degrades performance by forcing Podman to needlessly re-pull images.Under [storage.options.overlay], uncomment the following line (remove the # from the start):
mount_program = "/usr/bin/fuse-overlayfs"
Save the file and run the following.
Note
You must switch to user demisto before running the "system migrate" (running it as root will have no effect).
su - demisto
podman system migrate
Also as user demisto, run the following to ensure the path changes were applied:
podman info | grep Root
You should see the correct runRoot and graphRoot settings.
Still as user demisto, verify the issue is resolved by running:
podman run hello-world
If the issue persists, purge Podman's database by running the following.
Note
The "system migrate" must be done by user demisto.
rm -rf /home/demisto/.local/share/containers/*
podman system migrate