Troubleshoot Podman installation issues, including Keyring Quota Exceeded error and unused containers taking up resources.
Script failed to run: Docker code runner got container error: [Docker code script is in inconsistent state, ... error: [exit status 126] stderr: [Error: OCI runtime error: crun: create keyring ...: Disk quota exceeded]
By default, Podman creates a keyring
that is used by each container. The limit per user on the machine might be low and Podman can reach the limit when running more containers than the keyring
limit. To check the keyring
usage, run the sudo cat /proc/key-users
operating system command.
The command returns the usage for each UID (to retrieve the demisto user UID, run id demisto
). The fourth column shows the number of keys used out of the total number available. For more information about keys, see Kernel Key Retention Service.
You can either increase the limit of max keyrings (increasing to 1000 is safe and reasonable) per user as specified by your Linux vendor documentation or you can disable keyring creation by Podman. We recommend disabling keyring creation, unless keyrings are used by Podman in other applications on the machine. To disable keyring creation by Podman, modify the containers.conf
file and add the option keyring = false
under the "[containers]"
section. For more information, see the Containers Engine Configuration File.
In some cases, if the Podman process crashes or is killed abruptly it can leave containers on disk. You might see errors such as error allocating lock for new container: allocation failed; exceeded num_lock
when the maximum number of locks used to manage containers is exhausted due to the unused containers that remain.
Change to the demisto operating system user
sudo su - -s /bin/bash demisto
.Run
podman ps -a -f status=exited
to check for unused containers.Clean up the unused containers
podman container cleanup --rm -a
.Note
When you run
podman container cleanup --rm -a
, you might see a message such asrunning or paused containers cannot be moved without force
. The message can be safely ignored, as it only pertains to current running containers, which are not removed.After clean up, verify there are no remaining unused containers
podman ps -a -f status=exited
.
When installing a Cortex XSIAM engine on a RHEL system (version 8 or later), or when running an integration on such an engine, you get a permission error for a path under /run
(for example /run/user/0
or /run/libpod
).
In RHEL 9 only: Make sure the
container-tools
meta-package is installed, by running:yum -y install container-tools
Run the following commands:
cp /etc/containers/storage.conf /home/demisto/.config/containers/storage.conf
chown demisto:demisto /home/demisto/.config/containers/storage.conf
chmod 600 /home/demisto/.config/containers/storage.conf
Edit
/home/demisto/.config/containers/storage.conf
.Under [storage], change
runroot
to some temporary directory that is accessible by user demisto.For example:
runroot = "/tmp/podman-run-xsoar"
Important
The
runroot
must be located under thetmpfs
file system type. This is required to clean Podman's run state on reboot and for performance reasons.Also under [storage], change
graphroot
(which is where container images are stored) to any location that is owned and accessible by user demisto. We recommend using this standard path:graphroot = "/home/demisto/.local/share/containers/storage"
Caution
Unlike the
runroot
, thegraphroot
must NOT be located under thetmpfs
file system type. Usingtmpfs
for thegraphroot
might corrupt container images, causing command executions to fail. It also degrades performance by forcing Podman to needlessly re-pull images.Under [storage.options.overlay], uncomment the following line (remove the # from the start):
mount_program = "/usr/bin/fuse-overlayfs"
Save the file and run the following.
Note
You must switch to user demisto before running the "system migrate" (running it as root will have no effect).
su - demisto
podman system migrate
Also as user demisto, run the following to ensure the path changes were applied:
podman info | grep Root
You should see the correct runRoot and graphRoot settings.
Still as user demisto, verify the issue is resolved by running:
podman run hello-world
If the issue persists, purge Podman's database by running the following:
Note
The "system migrate" must be done by user demisto.
rm -rf /home/demisto/.local/share/containers/*
podman system migrate