Ingest Logs from Elasticsearch Filebeat - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Pro Administrator Guide

Product
Cortex XDR
License
Pro
Creation date
2023-10-31
Last date published
2024-03-19
Category
Administrator Guide
Abstract

Cortex XDR can ingest logs from Elasticsearch Filebeat, a file system logger that logs file activity on your endpoints and servers.

Note

Ingesting logs and data requires a Cortex XDR Pro per GB license.

If you want to ingest logs about file activity on your endpoints and servers and do not use the Cortex XDR agent, you can install Elasticsearch Filebeat as a system logger and then forward those logs to Cortex XDR. To facilitate log ingestion, Cortex XDR supports the same protocols that Filebeat and Elasticsearch use to communicate. Cortex XDR supports using Filebeat up to version 8.2 with the Filebeat data collector. Cortex XDR also supports logs in single line format or multiline format. For more information on handling messages that span multiple lines of text in Elasticsearch Filebeat, see Manage Multiline Messages.

Cortex XDR supports all sections in the filebeat.yml configuration file, such as support for Filebeat fields and tags. As a result, this enables you to use the add_fields processor to identify the product/vendor for the data collected by Filebeat so the collected events go through the ingestion flow (Parsing Rules). To configure the product/vendor ensure that you use the default fields attribute, as opposed to the target attribute, as shown in the following example.

processors:
  - add_fields:
      fields:
        vendor: <Vendor>
        product: <Product>

To provide additional context during investigations, Cortex XDR automatically creates a new Cortex Query Language (XQL) dataset from your Filebeat logs. You can then use the XQL dataset to search across the logs Cortex XDR received from Filebeat.

To receive logs, you configure collection settings for Filebeat in Cortex XDR and output settings in your Filebeat installations. As soon as Cortex XDR begins receiving logs, the data is visible in XQL Search queries.

  1. In Cortex XDR , set up Data Collection.

    1. Select SettingsConfigurationsData CollectionCustom Collectors.

    2. In the Filebeat configuration, click Add Instance.

    3. Specify a descriptive Name for your Filebeat log collection configuration.

    4. Specify the Vendor and Product for the type of logs you are ingesting.

      The vendor and product are used to define the name of your XQL dataset (<vendor>_<product>_raw). If you do not define a vendor or product, Cortex XDR examines the log header to identify the type and uses that to define the vendor and product in the dataset. For example, if the type is Acme and you opt to let Cortex XDR determine the values, the dataset name would be acme_acme_raw.

    5. Save & Generate Token.

      Click the copy icon next to the key and record it somewhere safe. You will need to provide this key when you set up output settings on your Filebeat instance. If you forget to record the key and close the window you will need to generate a new key and repeat this process.

  2. Set up Filebeat to forward logs.

    After installing the Filebeat agent, configure an Elasticsearch output:

    1. Under the output.elasticsearch section, configure the following entities:

      filebeat-setup.png
      • hosts—Copy the API URL from your Filebeat configuration and paste it in this field.

      • compression_level—5 (recommended)

      • bulk_max_size—1000 (recommended)

      • api_key—Paste the key you created in when you configured Filebeat Log Collection in Cortex XDR.

      • proxy_url—(Optional) <server_ip>:<port_number>. You can specify your own <server_ip> or use the Broker VM to proxy Filebeat communication using the format <Broker_VM_ip>:<port_number>. When using the Broker VM, ensure that you activate the Local Agent Settings applet with the Agent Proxy enabled.

    2. Save the changes to your output file.

    After Cortex XDR begins receiving logs from Filebeat, they will be available in XQL Search queries.

  3. (Optional) Monitor your Filebeat integration.

    You can return to the SettingsConfigurationsData CollectionCustom Collectors page to monitor the status of your Filebeat configuration. For each instance, Cortex XDR displays the number of logs received in the last hour, day, and week. You can also use the Data Ingestion Dashboard to view general statistics about your data ingestion configurations.

  4. (Optional) Set up alert notifications to monitor the following events.

    • A Filebeat agent status changes to disconnected.

    • A Filebeat module has stopped sending logs.