Learn more about the data ingestion health metrics in the metrics_source dataset and the metrics_view preset.
The data ingestion metrics are calculated in 5-minute aggregation periods and saved to the metrics_source dataset and metrics_view preset. These metrics measure the amount, size, and rate in which logs are ingested by a data source:
Metric | Description |
|---|---|
total_size_bytes | Total size (in bytes) of the logs collected during the aggregation period. |
total_size_rate | Average size (in bytes per second) of the logs collected during the aggregation period. |
total_event_count | Total number of logs collected during the aggregation period |
total_event_rate | Average number (in count per second) of logs collected during the aggregation period. |
In the metrics_source dataset the data ingestion metrics are saved alongside additional fields that describe the data source associated with the metrics. Only entries with ingestion metric values greater than zero are saved in the dataset. Entries with zero values are not saved in this dataset. metrics_view is a preset for data in the metrics_source dataset. The preset also simulates completion of entries with zero values in data ingestion metrics at runtime, which allows effective use of metrics. Therefore, when investigating disruptions in data collection, we recommend using the metrics_view preset in XQL queries and correlation rules.
The built-in data ingestion monitoring and alerts mechanism uses the data ingestion metrics to identify disruption in the data ingestion pipeline. When Cortex XSIAM identifies a significant deviation from the normal pattern of log collection, ingestion alerts are triggered. You can see all ingestion alerts on the Data Ingestion Health page. To troubleshoot or investigate an alert, right click an alert and click Investigate in XQL query to see the metrics that caused the alert to be triggered. For more information, see Viewing ingestion and collection alerts.
Alternatively, you can create your own custom logic for data ingestion health monitoring by setting up correlation rules that monitor the data ingestion metrics. For more information, see Creating Correlation Rules to Monitor Data Ingestion Health.
The following table describes all the fields in the metrics_source dataset and metrics_view preset: