Ingest logs from Microsoft Azure Event Hub - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Documentation
Product
Cortex XDR
Creation date
2024-03-06
Last date published
2024-10-10
Category
Administrator Guide
Abstract

Ingest logs from Microsoft Azure Event Hub with an option to ingest audit logs to use in Cortex XDR authentication stories.

Cortex XDR can ingest different types of data from Microsoft Azure Event Hub using the Microsoft Azure Event Hub data collector. To receive logs from Azure Event Hub, you must configure the settings in Cortex XDR based on your Microsoft Azure Event Hub configuration. After you set up data collection, Cortex XDR begins receiving new logs and data from the source.

When Cortex XDR begins receiving logs, the app creates a new dataset (MSFT_Azure_raw) that you can use to initiate XQL Search queries. For example, queries refer to the in-app XQL Library. For enhanced cloud protection, you can also configure Cortex XDR to normalize Azure Event Hub audit logs, including Azure Kubernetes Service (AKS) audit logs, with other Cortex XDR authentication stories across all cloud providers using the same format, which you can query with XQL Search using the cloud_audit_logs dataset. For logs that you do not configure Cortex XDR to normalize, you can change the default dataset. Cortex XDR can also raise Cortex XDR alerts (Analytics, IOC, BIOC, and Correlation Rules) when relevant from Azure Event Hub logs. While Correlation Rules alerts are raised on non-normalized and normalized logs, Analytics, IOC, and BIOC alerts are only raised on normalized logs.

Enhanced cloud protection provides:

  • Normalization of cloud logs

  • Cloud logs stitching

  • Enrichment with cloud data

  • Detection based on cloud analytics

  • Cloud-tailored investigations

The following table provides a brief description of the different types of Azure audit logs you can collect.

Note

For more information on Azure Event Hub audit logs, see Overview of Azure platform logs.

Type of data

Description

Activity logs

Retrieves events related to the operations on each Azure resource in the subscription from the outside in addition to updates on Service Health events.

Note

These logs are from the management plane.

Azure Active Directory (AD) Activity logs and Azure Sign-in logs

Contain the history of sign-in activity and audit trail of changes made in Azure AD for a particular tenant.

Note

Even though you can collect Azure AD Activity logs and Azure Sign-in logs using the Azure Event Hub data collector, we recommend using the Microsoft 365 data collector, because it is easier to configure. In addition, ensure that you don't configure both collectors to collect the same types of logs, because if you do so, you will be creating duplicate data in Cortex XDR.

Resource logs, including AKS audit logs

Retrieves events related to operations that were performed within an Azure resource.

Note

These logs are from the data plane.

Danger

Ensure that you do the following tasks before you begin configuring data collection from Azure Event Hub.

Configure the Azure Event Hub collection in Cortex XDR:

  1. In the Microsoft Azure console, open the Event Hubs page, and select the Azure Event Hub that you created for collection in Cortex XDR.

  2. Record the following parameters from your configured event hub, which you will need when configuring data collection in Cortex XDR.

    • Your event hub’s consumer group.

      1. Select Entities → Event Hubs, and select your event hub.

      2. Select Entities → Consumer groups, and select your event hub.

      3. In the Consumer group table, copy the applicable value listed in the Name column for your Cortex XDR data collection configuration.

    • Your event hub’s connection string for the designated policy.

      1. Select Settings → Shared access policies.

      2. In the Shared access policies table, select the applicable policy.

      3. Copy the Connection string-primary key.

    • Your storage account connection string required for partitions lease management and checkpointing in Cortex XDR.

      1. Open the Storage accounts page, and either create a new storage account or select an existing one, which will contain the storage account connection string.

      2. Select Security + networking → Access keys, and click Show keys.

      3. Copy the applicable Connection string.

  3. Configure diagnostic settings for the relevant log types you want to collect and then direct these diagnostic settings to the designated Azure Event Hub.

    1. Open the Microsoft Azure console.

    2. Your navigation is dependent on the type of logs you want to configure.

      Log type

      Navigation path

      Activity logs

      Select Azure services → Activity log → Export Activity Logs, and +Add diagnostic setting.

      Azure AD Activity logs and Azure Sign-in logs

      1. Select Azure services → Azure Active Directory.

      2. Select Monitoring → Diagnostic settings, and +Add diagnostic setting.

      Resource logs, including AKS audit logs

      1. Search for Monitor, and select Settings → Diagnostic settings.

      2. From your list of available resources, select the resource that you want to configure for log collection, and then select +Add diagnostic setting.

        Note

        For every resource that you want to confiure, you'll have to repeat this step, or use Azure policy for a general configuration.

    3. Set the following parameters:

      • Diagnostic setting name: Specify a name for your Diagnostic setting.

      • Logs Categories/Metrics: The options listed are dependent on the type of logs you want to configure. For Activity logs and Azure AD logs and Azure Sign-in logs, the option is called Logs Categories, and for Resource logs it's called Metrics.

        Log type

        Log categories/metrics

        Activity logs

        Select from the list of applicable Activity log categories, the ones that you want to configure your designated resource to collect. We recommend selecting all of the options.

        • Administrative

        • Security

        • ServiceHealth

        • Alert

        • Recommendation

        • Policy

        • Autoscale

        • ResourceHealth

        Azure AD Activity logs and Azure Sign-in logs

        Select from the list of applicable Azure AD Activity and Azure Sign-in Logs Categories, the ones that you want to configure your designated resource to collect. You can select any of the following categories to collect these types of Azure logs.

        • Azure AD Activity logs:

          • AuditLogs

        • Azure Sign-in logs:

          • SignInLogs

          • NonInteractiveUserSignInLogs

          • ServicePrincipalSignInLogs

          • ManagedIdentitySignInLogs

          • ADFSSignInLogs

        Note

        There are additional log categories displayed. We recommend selecting all the available options.

        Resource logs, including AKS audit logs

        The list displayed is dependent on the resource that you selected. We recommend selecting all the options available for the resource.

      • Destination details: Select Stream to event hub, where additional parameters are displayed that you need to configure. Ensure that you set the following parameters using the same settings for the Azure Event Hub that you created for the collection.

        • Subscription: Select the applicable Subscription for the Azure Event Hub.

        • Event hub namespace: Select the applicable Subscription for the Azure Event Hub.

        • (Optional) Event hub name: Specify the name of your Azure Event Hub.

        • Event hub policy: Select the applicable Event hub policy for your Azure Event Hub.

    4. Save your settings.

  4. Configure the Azure Event Hub collection in Cortex XDR.

    1. Select Settings → Configurations → Data Collection → Collection Integrations.

    2. In the Azure Event Hub configuration, click Add Instance to begin a new configuration.

    3. Set these parameters.

      • Name: Specify a descriptive name for your log collection configuration.

      • Event Hub Connection String: Specify your event hub’s connection string for the designated policy.

      • Storage Account Connection String: Specify your event hub’s connection string for the designated policy.

      • Consumer Group: Specify your event hub’s consumer group.

      • Log Format: Select the log format for the logs collected from the Azure Event Hub as Raw, JSON, CEF, LEEF, Cisco-asa, or Corelight.

        Note

        When you Normalize and enrich audit logs, the log format is automatically configured. As a result, the Log Format option is removed and is no longer available to configure (default).

        • CEF or LEEF: The Vendor and Product defaults to Auto-Detect.

          Note

          For a Log Format set to CEF or LEEF, Cortex XDR reads events row by row to look for the Vendor and Product configured in the logs. When the values are populated in the event log row, Cortex XDR uses these values even if you specified a value in the Vendor and Product fields in the Azure Event Hub data collector settings. Yet, when the values are blank in the event log row, Cortex XDR uses the Vendor and Product that you specified in the Azure Event Hub data collector settings. If you did not specify a Vendor or Product in the Azure Event Hub data collector settings, and the values are blank in the event log row, the values for both fields are set to unknown.

        • Cisco-asa: The following fields are automatically set and not configurable.

          • Vendor: Cisco

          • Product: ASA

          Cisco data can be queried in XQL Search using the cisco_asa_raw dataset.

        • Corelight: The following fields are automatically set and not configurable.

          • Vendor: Corelight

          • Product: Zeek

          Corelight data can be queried in XQL Search using the corelight_zeek_raw dataset.

        • Raw or JSON: The following fields are automatically set and are configurable.

          • Vendor: Msft

          • Product: Azure

          Raw or JSON data can be queried in XQL Search using the msft_azure_raw dataset.

      • Vendor and Product: Specify the Vendor and Product for the type of logs you are ingesting.

        The Vendor and Product are used to define the name of your Cortex Query Language (XQL) dataset (<vendor>_<product>_raw). The Vendor and Product values vary depending on the Log Format selected. To uniquely identify the log source, consider changing the values if the values are configurable.

        Note

        When you Normalize and enrich audit logs, the Vendor and Product fields are automatically configured, so these fields are removed as available options (default).

      • Normalize and enrich audit logs: (Optional) For enhanced cloud protection, you can Normalize and enrich audit logs by selecting the checkbox (default). If selected, Cortex XDR normalizes and enriches Azure Event Hub audit logs with other Cortex XDR authentication stories across all cloud providers using the same format. You can query this normalized data with XQL Search using the cloud_audit_logs dataset.

    4. Click Test to validate access, and then click Enable.

      When events start to come in, a green check mark appears underneath the Azure Event Hub configuration with the amount of data received.