Learn more about how to verify the Broker VM applet application, connectivity, and processing errors and troubleshoot.
You can monitor the Broker VM applet status from the Broker VM page in the Apps column. For all Broker VM applets in both the Brokers and Clusters tabs, a status indicator icon indicates whether the applet is connected or has an error. Certain Broker VM applets provide more information to help you troubleshoot by providing specific errors and warnings. For each of these errors and warnings, a different root cause is provided with a recommended action so that you can easily resolve the problem. In addition, you can always monitor your Broker VM application, connectivity, and processing errors for supported applets using the Cortex Query Language (XQL) and the collection_auditing dataset, and by creating correlation rules to trigger collection health alerts.
On the Broker VM page, Broker VM applets with an error status display an error icon in the Apps column in both the Brokers and Clusters tabs. In distinct supported cases, detailed error descriptions are provided for some applets:
Green (Connected): Indicates the applet has no issues.
Orange (Warning): Indicates the applet has minor issues, such as processing errors.
Red (Error): Indicates the applet has application or connectivity errors.
In addition, you can also left-click the applet, to get a brief summary of the applet error or warning.
Each status change of a supported Broker VM applet is logged in the collection_auditing dataset. Querying this dataset can help you see all the connectivity changes of an instance over time, the escalation or recovery of the connectivity status, and the error, warning, and informational messages related to status changes.
You can use the collection_auditing dataset to monitor your supported Broker VMs applets. For example, by creating correlation rules to trigger collection health alerts, you can ensure that you are notified whenever an applet receives an error or warning. You can query the collection_auditing dataset to understand what error was thrown, and then use the troubleshooting table provided in Understand how to troubleshoot to resolve the problem. Once the problem is resolved, you can ensure the Broker VM applet is active again by querying the collection_auditing dataset, or by observing the Connected green status of the applet in the Broker VM page in the Apps column. The example below explains the different status changes on the Local Agent Setting that can be used to help you troubleshoot the applet's connectivity issues.
Note
For more information on creating correlation rules to trigger collection health alerts, see How can I set up correlation rules to trigger collection health alerts?.
This example searches for status changes of the Local Agent Settings applet, where the _broker_device_id is LVVTKM6S:
dataset = collection_auditing |filter collector_type = "Local Agent Settings" and _broker_device_id = "LVVTKM6S"
Output results:
The results indicate that the Local Agent Settings applet was active on Jan 10th 2025 16:25:21. On Jan 18th 2025 11:15:26, this applet had a connectivity error relating to an inaccessible URL, and this issue was resolved on Jan 18th 2025 16:27:26 when the applet was back to an active status.
COLLECTOR_TYPE | INSTANCE | CLASSIFICATION | DESCRIPTION | _BROKER_DEVICE_ID | _BROKER_DEVICE_NAME | _BROKER_IP_ADDRESS | _TIME |
|---|---|---|---|---|---|---|---|
Local Agent Settings | LVVTKM6S | INFORMATION | The applet is back to an active status | LVVTKM6S | LVVTKM6S | 155.11.23.22 | Jan 18th 2025 16:27:26 |
Local Agent Settings | LVVTKM6S | ERROR | There is at least one inaccessible url. The broker VM needs to communicate with the same URLs that the agents communicate with. | LVVTKM6S | LVVTKM6S | 155.11.23.22 | Jan 18th 2025 11:15:26 |
Local Agent Settings | LVVTKM6S | INFORMATION | The applet is active | LVVTKM6S | LVVTKM6S | 155.11.23.22 | Jan 10th 2025 16:25:21 |
You can create correlation rules that are based on the fields in the collection_auditing dataset, so you are notified whenever a Broker VM applet status changes to an error and warning.
In this example, a correlation rule triggers an alert if the Local Agent Settings collector changes to an error status.
Example XQL:
dataset = collection_auditing |filter classification = "Error" and collector_type = "Local Agent Settings" and _broker_device_name = "LVVTKM6S"
Additional fields to specify in the correlation rule:
Field | Value |
|---|---|
Time Schedule | Hourly |
Query time frame | 1 Hour |
Alert Suppression | Select Enable alert suppression. |
Action | Select Generate alert. |
Alert Domain | Health |
Severity | Medium |
Type | Collection |
To help you troubleshoot your supported Broker VM applets, the table below lists the different possible warning and error event types, including the applicable error or warning that is displayed when you left-click the applet on the Broker VM page, the description displayed in the collection_auditing dataset, the root cause of the problem, and the recommended action to resolve the problem. We recommend that you use this table as a first resource to troubleshoot your application, connectivity, and processing errors.
Applet | Event Type | Broker VM page message | Description in the collection_auditing dataset | Root Cause | Recommended Action |
|---|---|---|---|---|---|
Database Collector | Warning | The query has been running for over <X> minutes. driver: <driver>, server: <server>, port: <port>, database: <database>, query title: <query title>. | One of the queries has been running for too long. | A long-running query was detected. This may indicate inefficiencies in the query logic or large data retrieval. |
|
Database Collector | Warning | Could not parse the initial value. driver: <driver>, server: <server>, port: <port>, database: <database>, query title: <query title>, initial value: <initial_value>. | Could not parse the initial value for a query. | The rising column initial value failed to parse when initializing the query parameters from the configuration. |
|
Database Collector | Warning | The system couldn't run the new snapshot query because the results from your previous query are still uploading. | The system couldn't run the new snapshot query because the results from your previous query are still uploading. | The query kept running for a long time even when a new execution was supposed to start. | First, on the server side, confirm that the query is taking a long time. If it is, increase the collection interval to prevent the next run from overlapping with the current query. |
Database Collector | Warning | The snapshot query and data upload were canceled because they either exceeded the <X> hour time limit or encountered an internal error. | The snapshot query and data upload were canceled because they took too long, or encountered an internal error. | The query and data upload were canceled because they exceeded the time limit. | On the server, confirm the query is taking a long time. |
Database Collector | Error | Could not connect to the database. driver: <driver>, server: <server>, port: <port>, database: <database>. | Could not connect to the database. | An issue occurred when the collector tried to establish an initial connection to the specified database. |
|
Database Collector | Error | Could not run the query. driver: <driver>, server: <server>, port: <port>, database: <database>, query title: <query title>. | Could not run one of the configured queries. | An issue occurred while executing one of the configured queries, possibly due to syntax or permissions. |
|
Database Collector | Error | Could not write the state to server cache. | Could not write data to the server cache. | The collector encountered an error while trying to write data to the server's cache. | This issue can occur intermittently. If the problem persists, contact support for further assistance. |
Database Collector | Error | Could not stream the collector results to the server. | Could not stream results to the server. | An issue occurred where results could not be streamed to the tenant due to an internal component failure. | This issue can occur intermittently. If the problem persists, contact support for further assistance. |
Files and Folders Collector | Warning | The total file size in the folder <folder_path> for file type <file_type> has exceeded the allowed 30 MB limit in replace mode | The total folder size exceeded the allowed 30MB limit in replace mode. | An issue that occurs when the total folder size exceeds the allowed 30MB limit in Replace mode for the Storage Method as defined in the Data Source Mapping. For more information, see Activate Files and Folders Collector. | This issue occurs when the total folder size exceeds the allowed 30MB limit in Replace mode for the Storage Method as defined in the Data Source Mapping. For more information, see Activate Files and Folders Collector. |
Files and Folders Collector | Error | Failed to stream collector results to server | Could not write data to server cache. | An issue occurred where the collector failed to stream data to an internal component, preventing it from reaching the tenant. | This issue can occur intermittently. If the problem persists, contact support for further assistance. |
Files and Folders Collector | Error | Invalid mount detected at path <folder_path> | Invalid mount point detected. | An issue that occurs when an invalid mount point is detected. |
|
FTP Collector | Error | Failed to load SSH key for server <server> at path <folder_path> | Failed to load SSH key. | The SSH key file for SFTP connections could not be loaded or parsed. | Verify the SSH key configuration. If the problem continues, contact support for further assistance. |
FTP Collector | Error | Failed to login to server <server> at path <folder_path> due to an SSL Certification error. | Failed to login to the server due to an SSL Certification error. | SSL/TLS certificate validation failed during secure FTP connection. | Verify the server's certificates. Ensure the Certificate Authority (CA) is valid and trusted. |
FTP Collector | Error | Login failed for server <server> at path <folder_path> with user <username>. | Login failed for the server. | FTP authentication failed due to incorrect username/password. | Check the configured login details and permissions. Ensure the correct username and password are used. |
FTP Collector | Error | Failed to login to server <server> at path <folder_path> with user <username> as the connection timed out. | Failed to login to the server as the connection timed out. | Connection attempt to the FTP server timed out. | Check the network connection and confirm the FTP server is accessible from the Broker VM. |
FTP Collector | Error | Failed to connect to server <server> at path <folder_path>. | Failed to connect to server. | Unable to establish a connection to the FTP server. | Confirm the server can be accessed and there are no network restrictions. If the problem continues, contact support for further assistance. |
FTP Collector | Error | Connection to the FTP server <server> at path <folder_path> timed out while trying to list the directory content. This probably happened as the server is using Active mode, but only Passive mode is supported. | Connection to the FTP server timed out while trying to list the directory content. This probably happened as the server is using Active mode, but only Passive mode is supported. | FTP server failed to list directory contents, likely due to Active/Passive mode misconfiguration. | Configure the FTP server to use Passive mode, and verify directory listing permissions. If the problem continues, contact support for further assistance. |
FTP Collector | Error | Failed to access the path <folder_path> on server <server>. The path doesn't exist, is unavailable, or access was denied due to permissions. | The specified path couldn't be accessed as either the path doesn't exist, is unavailable, or access was denied due to permissions. | The configured path does not exist, is unavailable, or access was denied due to permissions. | Make sure the configured path is correct and that there are no permissions issues. If the problem continues, contact support for further assistance. |
Kafka Collector | Warning | Failed to parse a log from the configured server list: <bootstrap_server_list>. The log does not match the expected format <expected_type>. Consider using the 'RAW' type to ingest logs in their current format. | Log parsing failed due to an unexpected format for Apache Kafka servers. | The user configured the port to receive logs from type CEF or LEEF, but the logs are either of a different type or contain errors. | Verify the logs in the Apache Kafka topics are in the correct format. If the format doesn't match, consider using the 'RAW' log type in the collector configuration. |
Kafka Collector | Error | Could not create an Apache Kafka dialer for the configured server list: <bootstrap_server_list> | Could not create an Apache Kafka dialer for the Kafka client. | Attempt to create the SSL/SASL dialer configuration failed during initialization. | Verify the Apache Kafka configuration, including SSL/SASL settings, and ensure that the certificates are valid and correctly configured. If the problem continues, contact support for further assistance. |
Kafka Collector | Error | Failed to retrieve a topic list from the configured server list: <bootstrap_server_list> | Failed to retrieve a topic list from the Apache Kafka servers. | The collector failed to retrieve a topic list from the Apache Kafka servers. | Check for any network connectivity issues and the status of the Apache Kafka server. If the problem continues for an extended period, contact support for further assistance. |
Kafka Collector | Error | Failed to create an Apache Kafka consumer for the configured server list: <bootstrap_server_list> | Failed to create an Apache Kafka consumer. | The collector failed to create a consumer for any of the configured topics. | Check for any network connectivity issues and the status of the Apache Kafka server. If the problem continues for an extended period, contact support for further assistance. |
Kafka Collector | Error | Failed to read a message using the Apache Kafka consumer from the configured server list: <bootstrap_server_list> | Failed to read a message using the Apache Kafka consumer. | An error occurred while the collector was reading messages using the Apache Kafka consumer. | Review the collector's logs for more specific error details. If the problem continues, contact support for further assistance. |
Kafka Collector | Error | Couldn't retrieve the number of partitions for the topic from the configured server list: <bootstrap_server_list> | Couldn't retrieve the number of partitions for the topic. | The collector failed to retrieve the partition list from the Apache Kafka server. | Check for any network connectivity issues and the status of the Apache Kafka server. If the problem continues for an extended period, contact support for further assistance. |
Kafka Collector | Error | There are no configured topics available on the configured server list: <bootstrap_server_list> | There are no configured topics available on the Apache Kafka servers. | The topic patterns configured don't match any existing topics on the Kafka Collector applets. The system will automatically try again to find them. | This is mainly an informational message. The system will automatically try again to find the topics. If the topics are expected to exist, verify the topic names and patterns in the collector configuration. |
Local Agent Settings | Warning | Failed to download and store <package_type> (version: <version>, os: <os>) inside the Broker VM for agent package caching | Failed to download and store content/installer for agent package caching. | An issue that occurs while downloading the content/installer and extracting it. | If the problem persists, contact support for further assistance. |
Local Agent Settings | Warning | The Broker VM failed to process request of agent package cache update <package_type>. | Failed to process request of agent package cache update. | Broker VM failed to get the list of content/installer from the tenant. | If the problem persists, contact support for further assistance. |
Local Agent Settings | Warning | The disk space allocated for agent installer and content caching exceeds 90% (<number> GB). | The disk space allocated for agent installer and content caching exceeds 90%. | Agent installer and content caching exceeds 90% of the allocated disk space for. | If you intend to use the Broker VM for agent installer and content caching, you must use extra disk space. For more information, Increase Broker VM storage allocated for data caching. |
Local Agent Settings | Error | Inaccessible url:<url> The Broker VM needs to communicate with the same URLs as the Agents. | There is at least one inaccessible URL. The Broker VM needs to communicate with the same URLs that the agents communicate with. | An issue with the accessibility to a URL as the Broker VM needs to communicate with the same URLs as the Agents. |
|
Local Agent Settings | Error | No more disk space available for agent installer and content caching. | No more disk space available for agent installer and content caching. | Agent installer and content caching exceeds 100% of the allocated disk space for. | If you intend to use the Broker VM for agent installer and content caching, you must use extra disk space. For more information, Increase Broker VM storage allocated for data caching. |
NetFlow Collector | Warning | Received a NetFlow packet with an unsupported protocol version on port <dest_port>. The supported versions are <supported_versions>. | Received a NetFlow packet with an unsupported protocol version. | The NetFlow Collector received packets using an unsupported protocol version by the current implementation. | Ensure the NetFlow exporter is configured to use one of the supported NetFlow versions. Verify that only NetFlow traffic is being sent to the configured port. If the problem continues, contact support for further assistance. |
Network Mapper | Warning | Part of the IP range configured has no hosts. Check the Broker VM logs for more details. | Part of the IP range configured has no hosts. Check the Broker VM logs for more details. | The Network Mapper wasn't able to detect any active hosts in part of the configured IP range during the scanning process. | Verify the configured IP range and ensure hosts are active and reachable. If the problem continues for an extended period, contact support for further assistance. |
Syslog Collector | Warning | First log over the connection doesn't have the expected type. Log format: <received format>, expected format: <expected format>. The connection has been closed by the collector. vendor: <vendor>, product: <product>, source: <source ip>, port: <destination port>. | The initial received log over the connection does not match the expected log format. | The Syslog collector applet was configured to receive logs of a specific type, and the logs that were received through the configured port are either of a different type or contain errors. | Check the logs transmitted over the connection and ensure they match the log type specified in the configuration. |
Syslog Collector | Warning | Log size exceeded 65 KB for <source ip>, on port: <dest port> (vendor: <vendor>, product: <product>). This may happen if a single log entry is too large or if logs lack end-of-line delimiters, causing them to merge and exceed the size limit. | A log entry exceeded the maximum allowed size. | The issue is caused by either a log exceeding 65 KB or logs missing an end-of-line delimiter, which causes them to aggregate as 'partial logs' and hit the 65 KB limit. | Ensure the logs are within the size limit and include an end-of-line delimiter. |
Syslog Collector | Warning | Connection rejected from <source ip>. Port <dest port> is restricted to specific source IPs, and the provided IP is not on the allowed list. | Connection rejected due to a source IP restriction. | The configuration includes specific networks/IPs to send logs to a specific port of the Syslog collector applet. The Syslog collector applet detected a | This is an informational event and does not impact the data collector applet's operation, except for rejecting the SRC IP connection. Examine the |
Syslog Collector | Warning | Failed to parse log from <source ip> on port <dest port> (vendor: <vendor>, product: <product>). The log does not match the expected format 'CEF', as configured for this port. Consider using the 'RAW' type to ingest logs in their current format. | Log parsing failed due to an unexpected format. | The port was configured to receive logs in a CEF format, and the logs that were sent from the log exporter were either of a different type or contain errors. | Ensure that the logs are in the correct CEF format. If it does not match the format type, consider using 'RAW' type logs in the configuration. |
Syslog Collector | Warning | Failed to parse log from <source ip> on port <dest port> (vendor: <vendor>, product: <product>). The log does not match the expected format 'LEEF', as configured for this port. Consider using the 'RAW' type to ingest logs in their current format. | Log parsing failed due to an unexpected format. | The port was configured to receive logs in a LEEF format, and the logs that were sent from the log exporter were either of a different type or contain errors. | Ensure that the logs are in the correct LEEF format. If it does not match the type, consider using 'RAW' type logs in the configuration. |
Syslog Collector | Warning | Unable to automatically detect the initial log over the connection for vendor: <vendor>, product: <product>, source: <source ip>, port: <dest port>. the log will be discarded. | Unable to automatically detect the initial log over the connection, the log will be discarded. | The configuration is set to 'auto-detect' for the log type, and the Syslog collector applet can't detect the first log over the connection from the four known log types: CEF, LEEF, Corelight, and Cisco. The Syslog collector applet does attempt a second try on the second log over the connection. If this attempt fails as well, then the Syslog collector uses the 'RAW' type log for all the logs over this specific connection. | Ensure that the logs being sent have one of the four supported log types: CEF, LEEF, Cisco, or Corelight. If it does not match any of the types, consider using 'RAW' type logs in the configuration. |
Syslog Collector | Warning | Failed to parse octet-framing packet from <source ip> on port <port> (vendor: <vendor>, product: <product>). Please verify the packet structure and format. | Failed to process the packet due to a missing length header. The packet does not conform to the expected octet-framing format. | The log exporter is sending the logs using a octet-framing method, and the Syslog collector is experiencing problems with the structure/format that is being sent. | Ensure that the logs are being sent correctly using the octet-framing method. |
Syslog Collector | Error | Failed loading client key from the configuration on port <dest port>. Please verify the certificate and key settings. | Failed to load a client key from the configuration. | An issue with the certificates added by the user to the Syslog collector's configuration. | Verify the settings for the certificates and keys. |
Syslog Collector | Error | Failed to process CA certificate from the | Failed to parse the CA certificate from the provided configuration. | An issue with the certificates added by the user to the Collector's configuration. | Verify the settings for the certificates and keys. |
WEC | Error | Failed starting WEC applet due to an internal error. The applet would require a restart to restore functionality. | Failed starting WEC applet due to an internal error. | A rare error that can occur due to an internal program issue. | Deactivate the WEC, and then reactivate it. The WEC configuration will be saved, and the collector will restart. If the error persists, contact support for further assistance. |
Database Collector/ Files and Folders Collector/ FTP Collector/ Kafka Collector/ NetFlow Collector/ Network Mapper/ WEC / Syslog Collector | Error | The collector could not connect to the third-party persistence service. This issue must be resolved to restore functionality. | Failed to establish an initial connection to the persistence service. | The persistence service for the data collector applet, is not functioning properly. As a result, the applet is unable to operate since this service is essential for its functionality. | If the problem persists, contact support for further assistance. |
Database Collector/ Files and Folders Collector/ FTP Collector/ Kafka Collector/ NetFlow Collector/ Network Mapper/ WEC / Syslog Collector | Error | The collector is unable to read from the persistence service. This issue must be resolved to restore functionality. | Unable to read from the persistence service. | The persistence service for the applets, is not functioning properly. As a result, the applet is unable to operate since this service is essential for its functionality. | The data collector applet is unable to read from the persistence service. This issue must be resolved to restore functionality. |
Database Collector/ Files and Folders Collector/ FTP Collector/ Kafka Collector/ NetFlow Collector/ Network Mapper/ WEC / Syslog Collector | Error | The collector could not reach the receptor (Tenant) to transmit data. This issue must be resolved to restore functionality. | Unable to connect to the receptor for data transmission. | The applet can't send logs to the tenant due to some network problem. | Review the network setup and check for any firewalls, proxies, or other network components that could potentially cause interruptions. |
Files and Folders Collector/ FTP Collector | Warning | Failed to parse log from <folder_path> (vendor: <vendor>, product: <product>. The log does not match the expected format <expected_type>, as configured for this folder path. Consider using the 'RAW' type to ingest logs in their current format. | Log parsing failed due to an unexpected format. | The user configured the port to receive logs from type CEF or LEEF, but the logs are either of a different type or contain errors. | Verify the logs in the configured folder path and ensure they match the expected format. As a workaround, consider using the 'RAW' log type in the collector configuration. |