Learn more about the data ingestion health metrics in the metrics_source dataset and the metrics_view preset.
The data ingestion metrics are calculated in 5-minute aggregation periods and saved to the metrics_source dataset and metrics_view preset. These metrics measure the amount, size, and rate in which logs are ingested by a data source:
Metric | Description |
|---|---|
total_size_bytes | Total size (in bytes) of the logs collected during the aggregation period. |
total_size_rate | Average size (in bytes per second) of the logs collected during the aggregation period. |
total_event_count | Total number of logs collected during the aggregation period |
total_event_rate | Average number (in count per second) of logs collected during the aggregation period. |
In the metrics_source dataset the data ingestion metrics are saved alongside additional fields that describe the data source associated with the metrics. Only entries with ingestion metric values greater than zero are saved in the dataset. Entries with zero values are not saved in this dataset. metrics_view is a preset for data in the metrics_source dataset. The preset also simulates completion of entries with zero values in data ingestion metrics at runtime, which allows effective use of metrics. Therefore, when investigating disruptions in data collection, we recommend using the metrics_view preset in XQL queries and correlation rules.
Cortex XSIAM's built-in data ingestion monitoring and alerts mechanism uses the data ingestion metrics to identify disruptions in the data ingestion pipeline. Using analytical logic, Cortex XSIAM creates an ingestion baseline for each data source that reflects the routine pattern of log collection. If a data source isn't ingesting logs or there is a significant deviation from the baseline, ingestion alerts are triggered. You can see all ingestion alerts on the Health Alerts page. To troubleshoot or investigate an alert, right click an alert and click Investigate in XQL query. For more information, see Investigate and resolve health alerts.
In addition, you can create your own custom logic for data ingestion health monitoring by setting up correlation rules that monitor the data ingestion metrics. For more information, see Creating correlation rules to monitor data ingestion health.
The following table describes all the fields in the metrics_source dataset and metrics_view preset: