Ingest logs from Amazon CloudWatch - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Documentation

Product
Cortex XDR
Creation date
2024-03-06
Last date published
2024-10-10
Category
Administrator Guide
Abstract

Take advantage of Cortex XDR investigation capabilities and set up generic or EKS log ingestion for your Amazon CloudWatch logs.

You can forward generic and Elastic Kubernetes Service (EKS) logs to Cortex XDR from Amazon CloudWatch. When forwarding EKS logs, the following log types are included:

  • API Server: Logs pertaining to API requests to the cluster.

  • Audit: Logs pertaining to cluster access via the Kubernetes API.

  • Authenticator: Logs pertaining to authentication requests into the cluster.

  • Scheduler: Logs pertaining to scheduling decisions.

  • Controller Manager: Logs pertaining to the state of cluster controllers.

You can ingest generic logs of the raw data or in a JSON format from Amazon Kinesis Firehose. EKS logs are automatically ingested in a JSON format from Amazon Kinesis Firehose. To enable log forwarding, you set up Amazon Kinesis Firehose and then add that to your Amazon CloudWatch configuration. After you complete the set up process, logs from the respective service are then searchable in Cortex XDR to provide additional information and context to your investigations.

As soon as Cortex XDR begins receiving logs, the application automatically creates one of the following Cortex Query Language (XQL) datasets depending on the type of logs you've configured:

  • Generic: <Vendor>_<Product>_raw

  • EKS: amazon_eks_raw

These datasets enable you to search the logs in XQL Search. For example, queries refer to the in-app XQL Library. For enhanced cloud protection, you can also configure Cortex XDR to normalize EKS audit logs, which you can query with XQL Search using the cloud_audit_logs dataset. Cortex XDR can also raise Cortex XDR alerts (Analytics, IOC, BIOC, and Correlation Rules) when relevant from AWS logs. While Correlation Rules alerts are raised on non-normalized and normalized logs, Analytics, IOC, and BIOC alerts are only raised on normalized logs.

Enhanced cloud protection provides the following:

  • Normalization of cloud logs

  • Cloud logs stitching

  • Enrichment with cloud data

  • Detection based on cloud analytics

  • Cloud-tailored investigations

To set up Amazon CloudWatch integration, you require certain permissions in AWS. You need a role that enables access to configuring Amazon Kinesis Firehose.

  1. Set up the Amazon CloudWatch integration in Cortex XDR.

    1. Select Settings → Configurations → Data Collection → Collection Integrations.

    2. In the Amazon CloudWatch configuration, click Add Instance.

    3. Specify a descriptive Name for your log collection configuration.

    4. Select the Log Type as one of the following, where your selection changes the options displayed:

      • Generic: When selecting this log type, you can configure the following settings:

        • Log Format: Choose the format of the data input source (CloudWatch) that you'll export to Cortex XDR , either JSON or Raw.

        • Specify the Vendor and Product for the type of generic logs you are ingesting.

          The vendor and product are used to define the name of your XQL dataset (<Vendor>_<Product>_raw). If you do not define a vendor or product, Cortex XDR uses the default values of Amazon and AWS with the resulting dataset name as amazon_aws_raw. To uniquely identify the log source, consider changing the values.

      • EKS: When selecting this log type, the following options are displayed:

        • The Vendor is automatically set to Amazon and Product to EKS , and is non-configurable. This means that all data for the EKS logs, whether it's normalized or not, can be queried in XQL Search using the amazon_eks_raw dataset.

        • (Optional) You can decide whether to Normalize and enrich audit logs as part of the enhanced cloud protection by selecting the checkbox (default). If selected, Cortex XDR is configured to normalize EKS audit logs, which you can query with XQL Search using the cloud_audit_logs dataset.

    5. Save & Generate Token.

      Click the copy icon next to the key and record it somewhere safe. You will need to provide this key when you set up output settings in AWS Kinesis Firehose. If you forget to record the key and close the window you will need to generate a new key and repeat this process.

    6. Select Done to close the window.

  2. Create a Kinesis Data Firehose delivery stream to your chosen destination.

    1. Log in to the AWS Management Console, and open the Kinesis console.

    2. Select Data Firehose → Create delivery stream.

      aws-kinesis-firehose-create-delivery-system.png
    3. Define the name and source for your stream.

      • Delivery stream name: Enter a descriptive name for your stream configuration.

      • Source: Select Direct PUT or other sources.

      • Server-side encryption for source records in the delivery stream: Ensure this option is disabled.

      Click Next to proceed to the process record configuration.

    4. Define the process records.

      • Transform source records with AWS Lambda: Set the Data Transformation as Disabled.

      • Convert record format: Set Record format conversion as Disabled.

      Click Next to proceed to the destination configuration.

    5. Choose a destination for the logs.

      Choose HTTP Endpoint as the destination and configure the HTTP endpoint configuration settings:

      • HTTP endpoint name: Specify the name you used to identify your AWS log collection configuration in Cortex XDR.

      • HTTP endpoint URL: Copy the API URL associated with your log collection from the Cortex XDR management console. The URL will include your tenant name (https://api-<tenant external URL>/logs/v1/aws).

      • Access key: Paste in the token key you recorded earlier during the configuration of your Cortex XDR log collection settings.

      • Content encoding: Select GZIP. Disabling content encoding may result in high egress costs.

      • Retry duration: Enter 300 seconds.

      • S3 bucket: Set the S3 backup mode as Failed data only. For the S3 bucket, we recommend that you create a dedicated bucket for Cortex XDR integration.

      Click Next to proceed to the settings configuration.

    6. Configure additional settings.

      • HTTP endpoint buffer conditions: Set the Buffer size as 1 MiB and the Buffer interval as 60 seconds.

      • S3 buffer conditions: Use the default settings for Buffer size as 5 MiB and Buffer interval as 300 seconds unless you have alternative sizing preferences.

      • S3 compression and encryption: Choose your desired compression and encryption settings.

      • Error logging: Select Enabled.

      • Permissions: Create or update IAM role option.

      Select Next.

    7. Review your configuration and Create delivery stream.

      When your delivery stream is ready, the status changes from Creating to Active.

  3. To begin forwarding logs, add the Kinesis Firehose instance to your Amazon CloudWatch configuration.

    To do this, add a subscription filter for Amazon Kinesis Firehose.

  4. Verify the status of the integration.

    Return to the Integrations page and view the statistics for the log collection configuration.