Learn more about how to verify the Broker VM applet application, connectivity, and processing errors and troubleshoot.
You can monitor the Broker VM applet status from the Broker VM page in the Apps column. For all Broker VM applets in both the Brokers and Clusters tabs, a status indicator icon indicates whether the applet is connected or has an error. Certain Broker VM applets provide more information to help you troubleshoot by providing specific errors and warnings. For each of these errors and warnings, a different root cause is provided with a recommended action so that you can easily resolve the problem. In addition, you can always monitor your Broker VM application, connectivity, and processing errors for supported applets using the Cortex Query Language (XQL) and the collection_auditing dataset, and by creating correlation rules to trigger collection health alerts.
On the Broker VM page, Broker VM applets with an error status display an error icon in the Apps column in both the Brokers and Clusters tabs. In distinct supported cases, detailed error descriptions are provided for some applets:
Green (Connected): Indicates the applet has no issues.
Orange (Warning): Indicates the applet has minor issues, such as processing errors.
Red (Error): Indicates the applet has application or connectivity errors.
In addition, you can also left-click the applet, to get a brief summary of the applet error or warning.
Each status change of a supported Broker VM applet is logged in the collection_auditing dataset. Querying this dataset can help you see all the connectivity changes of an instance over time, the escalation or recovery of the connectivity status, and the error, warning, and informational messages related to status changes.
You can use the collection_auditing dataset to monitor your supported Broker VMs applets. For example, by creating correlation rules to trigger collection health alerts, you can ensure that you are notified whenever an applet receives an error or warning. You can query the collection_auditing dataset to understand what error was thrown, and then use the troubleshooting table provided in Understand how to troubleshoot to resolve the problem. Once the problem is resolved, you can ensure the Broker VM applet is active again by querying the collection_auditing dataset, or by observing the Connected green status of the applet in the Broker VM page in the Apps column. The example below explains the different status changes on the Local Agent Setting that can be used to help you troubleshoot the applet's connectivity issues.
Note
For more information on creating correlation rules to trigger collection health alerts, see How can I set up correlation rules to trigger collection health alerts?.
This example searches for status changes of the Local Agent Settings applet, where the _broker_device_id is LVVTKM6S:
dataset = collection_auditing |filter collector_type = "Local Agent Settings" and _broker_device_id = "LVVTKM6S"
Output results:
The results indicate that the Local Agent Settings applet was active on Jan 10th 2025 16:25:21. On Jan 18th 2025 11:15:26, this applet had a connectivity error relating to an inaccessible URL, and this issue was resolved on Jan 18th 2025 16:27:26 when the applet was back to an active status.
COLLECTOR_TYPE | INSTANCE | CLASSIFICATION | DESCRIPTION | _BROKER_DEVICE_ID | _BROKER_DEVICE_NAME | _BROKER_IP_ADDRESS | _TIME |
|---|---|---|---|---|---|---|---|
Local Agent Settings | LVVTKM6S | INFORMATION | The applet is back to an active status | LVVTKM6S | LVVTKM6S | 155.11.23.22 | Jan 18th 2025 16:27:26 |
Local Agent Settings | LVVTKM6S | ERROR | There is at least one inaccessible url. The broker VM needs to communicate with the same URLs that the agents communicate with. | LVVTKM6S | LVVTKM6S | 155.11.23.22 | Jan 18th 2025 11:15:26 |
Local Agent Settings | LVVTKM6S | INFORMATION | The applet is active | LVVTKM6S | LVVTKM6S | 155.11.23.22 | Jan 10th 2025 16:25:21 |
You can create correlation rules that are based on the fields in the collection_auditing dataset, so you are notified whenever a Broker VM applet status changes to an error and warning.
In this example, a correlation rule triggers an alert if the Local Agent Settings collector changes to an error status.
Example XQL:
dataset = collection_auditing |filter classification = "Error" and collector_type = "Local Agent Settings" and _broker_device_name = "LVVTKM6S"
Additional fields to specify in the correlation rule:
Field | Value |
|---|---|
Time Schedule | Hourly |
Query time frame | 1 Hour |
Alert Suppression | Select Enable alert suppression. |
Action | Select Generate alert. |
Alert Domain | Health |
Severity | Medium |
Type | Collection |
To help you troubleshoot your supported Broker VM applets, the table below lists the different possible warning and error event types, including the applicable error or warning that is displayed when you left-click the applet on the Broker VM page, the description displayed in the collection_auditing dataset, the root cause of the problem, and the recommended action to resolve the problem. We recommend that you use this table as a first resource to troubleshoot your application, connectivity, and processing errors.
Applet | Event Type | Broker VM page message | Description in the collection_auditing dataset | Root Cause | Recommended Action |
|---|---|---|---|---|---|
Local Agent Settings | Warning | Failed to download and store <package_type> (version: <version>, os: <os>) inside the Broker VM for agent package caching | Failed to download and store content/installer for agent package caching. | An issue that occurs while downloading the content/installer and extracting it. | If the problem persists, contact support for further assistance. |
Local Agent Settings | Warning | The Broker VM failed to process request of agent package cache update <package_type>. | Failed to process request of agent package cache update. | Broker VM failed to get the list of content/installer from the tenant. | If the problem persists, contact support for further assistance. |
Local Agent Settings | Error | The Broker VM failed to get urls for <package_type>. | The Broker VM failed to get URLs for agent package caching. | An issue that occurs when there is connectivity issue between the Broker VM to the tenant. | If the problem persists, contact support for further assistance. |
Local Agent Settings | Error | Inaccessible url:<url> The Broker VM needs to communicate with the same URLs as the Agents. | There is at least one inaccessible URL. The Broker VM needs to communicate with the same URLs that the agents communicate with. | An issue with the accessibility to a URL as the Broker VM needs to communicate with the same URLs as the Agents. |
|
Syslog Collector | Warning | First log over the connection doesn't have the expected type. Log format: <received format>, expected format: <expected format>. The connection has been closed by the collector. vendor: <vendor>, product: <product>, source: <source ip>, port: <destination port>. | The initial received log over the connection does not match the expected log format. | The Syslog collector applet was configured to receive logs of a specific type, and the logs that were received through the configured port are either of a different type or contain errors. | Check the logs transmitted over the connection and ensure they match the log type specified in the configuration. |
Syslog Collector | Warning | Log size exceeded 65 KB for <source ip>, on port: <dest port> (vendor: <vendor>, product: <product>). This may happen if a single log entry is too large or if logs lack end-of-line delimiters, causing them to merge and exceed the size limit. | A log entry exceeded the maximum allowed size. | The issue is caused by either a log exceeding 65 KB or logs missing an end-of-line delimiter, which causes them to aggregate as 'partial logs' and hit the 65 KB limit. | Ensure the logs are within the size limit and include an end-of-line delimiter. |
Syslog Collector | Warning | Connection rejected from <source ip>. Port <dest port> is restricted to specific source IPs, and the provided IP is not on the allowed list. | Connection rejected due to a source IP restriction. | The configuration includes specific networks/IPs to send logs to a specific port of the Syslog collector applet. The Syslog collector applet detected a | This is an informational event and does not impact the data collector applet's operation, except for rejecting the SRC IP connection. Examine the |
Syslog Collector | Warning | Failed to parse log from <source ip> on port <dest port> (vendor: <vendor>, product: <product>). The log does not match the expected format 'CEF', as configured for this port. Consider using the 'RAW' type to ingest logs in their current format. | Log parsing failed due to an unexpected format. | The port was configured to receive logs in a CEF format, and the logs that were sent from the log exporter were either of a different type or contain errors. | Ensure that the logs are in the correct CEF format. If it does not match the format type, consider using 'RAW' type logs in the configuration. |
Syslog Collector | Warning | Failed to parse log from <source ip> on port <dest port> (vendor: <vendor>, product: <product>). The log does not match the expected format 'LEEF', as configured for this port. Consider using the 'RAW' type to ingest logs in their current format. | Log parsing failed due to an unexpected format. | The port was configured to receive logs in a LEEF format, and the logs that were sent from the log exporter were either of a different type or contain errors. | Ensure that the logs are in the correct LEEF format. If it does not match the type, consider using 'RAW' type logs in the configuration. |
Syslog Collector | Warning | Unable to automatically detect the initial log over the connection for vendor: <vendor>, product: <product>, source: <source ip>, port: <dest port>. the log will be discarded. | Unable to automatically detect the initial log over the connection, the log will be discarded. | The configuration is set to 'auto-detect' for the log type, and the Syslog collector applet can't detect the first log over the connection from the four known log types: CEF, LEEF, Corelight, and Cisco. The Syslog collector applet does attempt a second try on the second log over the connection. If this attempt fails as well, then the Syslog collector uses the 'RAW' type log for all the logs over this specific connection. | Ensure that the logs being sent have one of the four supported log types: CEF, LEEF, Cisco, or Corelight. If it does not match any of the types, consider using 'RAW' type logs in the configuration. |
Syslog Collector | Warning | Failed to parse octet-framing packet from <source ip> on port <port> (vendor: <vendor>, product: <product>). Please verify the packet structure and format. | Failed to process the packet due to a missing length header. The packet does not conform to the expected octet-framing format. | The log exporter is sending the logs using a octet-framing method, and the Syslog collector is experiencing problems with the structure/format that is being sent. | Ensure that the logs are being sent correctly using the octet-framing method. |
Syslog Collector | Error | Failed loading client key from the configuration on port <dest port>. Please verify the certificate and key settings. | Failed to load a client key from the configuration. | An issue with the certificates added by the user to the Syslog collector's configuration. | Verify the settings for the certificates and keys. |
Syslog Collector | Error | Failed to process CA certificate from the | Failed to parse the CA certificate from the provided configuration. | An issue with the certificates added by the user to the Collector's configuration. | Verify the settings for the certificates and keys. |
WEC | Error | Failed starting WEC applet due to an internal error. The applet would require a restart to restore functionality. | Failed starting WEC applet due to an internal error. | A rare error that can occur due to an internal program issue. | Deactivate the WEC, and then reactivate it. The WEC configuration will be saved, and the collector will restart. If the error persists, contact support for further assistance. |
WEC | Error | Unfamiliar request arrived from the Domain Controller: <source>. Request: <request>. | Unfamiliar request arrived from the Domain Controller. | An issue that occurs when a Domain Controller sends an unsupported request to the WEC. | Identify the reason for this request on the Domain Controller and take steps to prevent it from recurring. |
WEC | Error | Received subscription termination from the domain controller <source>. Error: <error>. | Subscription terminated from the Domain Controller due to an error. | An issue that occurs when the Domain Controller closes the connection due to an error. | Open the Event Viewer on the Domain Controller and search for errors. Resolve the issue indicated by the error, which can be related to certificates, permissions, or collector configuration. |
WEC | Error | Received a Cancel request from Domain Controller: <source>. Go to the Domain Controller, open 'Event-Log forwarding Plugin' in Event Viewer and search for errors. | Received a Cancel request from Domain Controller. | An issue that occurs when the Domain Controller requests to cancel the connection. | Open the Event Viewer on the Domain Controller and search for errors. Resolve the issue indicated by the error, which can be related to certificates, permissions, or collector configuration. |
WEC | Error | Failed handling WEF request from Domain Controller: <source> due to an internal error. | Failed handling WEF request from Domain Controller due to an internal error. | A rare error that can occur due to an internal program issue. | Contact support for further assistance |
WEC / Syslog Collector | Error | The collector could not connect to the third-party persistence service. This issue must be resolved to restore functionality. | Failed to establish an initial connection to the persistence service. | The persistence service for the data collector applet, is not functioning properly. As a result, the applet is unable to operate since this service is essential for its functionality. | If the problem persists, contact support for further assistance. |
WEC / Syslog Collector | Error | The collector is unable to read from the persistence service. This issue must be resolved to restore functionality. | Unable to read from the persistence service. | The persistence service for the applets, is not functioning properly. As a result, the applet is unable to operate since this service is essential for its functionality. | The data collector applet is unable to read from the persistence service. This issue must be resolved to restore functionality. |
WEC / Syslog Collector | Error | The collector could not reach the receptor (Tenant) to transmit data. This issue must be resolved to restore functionality. | Unable to connect to the receptor for data transmission. | The applet can't send logs to the tenant due to some network problem. | Review the network setup and check for any firewalls, proxies, or other network components that could potentially cause interruptions. |