Learn more about the data ingestion health metrics in the metrics_source dataset and the metrics_view preset.
Prerequisite
For Cortex XDR to monitor data ingestion health and create health issues, you must enable the following settings under Configurations:
Cortex - Analytics: Go to → . For more information, see Enable the Analytics Engine and Identity Analytics.
Data Ingestion Monitoring: Go to → → → . For more information, see Set up your environment.
The data ingestion metrics are calculated in 5-minute aggregation periods and saved to the metrics_source dataset and metrics_view preset. These metrics measure the amount, size, and rate in which logs are ingested by a data source:
Metric | Description |
|---|---|
total_size_bytes | Total size (in bytes) of the logs collected during the aggregation period. |
total_size_rate | Average size (in bytes per second) of the logs collected during the aggregation period. |
total_event_count | Total number of logs collected during the aggregation period |
total_event_rate | Average number (in count per second) of logs collected during the aggregation period. |
In the metrics_source dataset, the data ingestion metrics are saved alongside additional fields that describe the data source associated with the metrics. Only entries with ingestion metric values greater than zero are saved in the dataset. Entries with zero values are not saved in this dataset.
metrics_view is a preset for data in the metrics_source dataset. The preset also simulates completion of entries with zero values in data ingestion metrics at runtime, which allows effective use of metrics. Therefore, when investigating disruptions in data collection, we recommend using the metrics_view preset in XQL queries and correlation rules.
In addition, you can create your own custom logic for data ingestion health monitoring by setting up correlation rules that monitor the data ingestion metrics. For more information, see Creating correlation rules to monitor data ingestion health.
The following table describes all the fields in the metrics_source dataset and metrics_view preset: