Configure Data Collection from Amazon S3 Manually - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Pro Administrator Guide

Product
Cortex XDR
License
Pro
Creation date
2023-10-31
Last date published
2024-03-18
Category
Administrator Guide
Abstract

Set up network flow log ingestion for your Amazon S3 logs manually (without a script).

Note

Ingesting logs and data requires a Cortex XDR Pro per GB license.

There are various reasons why you may need to configure data collection from Amazon S3 manually, as opposed to using the CloudFormation Script provided in Cortex XDR. For example, if your organization does not use CloudFormation scripts, you will need to follow the instructions below, which explain at a high-level how to perform these steps manually with a link to the relevant topic in the Amazon S3 documentation with the detailed steps to follow.

As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 Cortex Query Language (XQL) dataset (aws_s3_raw). This enables you to search the logs with XQL Search using the dataset. For example queries, refer to the in-app XQL Library. For enhanced cloud protection, you can also configure Cortex XDR to ingest network flow logs as Cortex XDR network connection stories, which you can query with XQL Search using the xdr_dataset dataset with the preset called network_story. Cortex XDR can also raise Cortex XDR alerts (Analytics, Correlations, IOC, and BIOC) when relevant from Amazon S3 logs. While Correlation Rules alerts are raised on non-normalized and normalized logs, Analytics, IOC, and BIOC alerts are only raised on normalized logs.

Enhanced cloud protection provides:

  • Normalization of cloud logs

  • Cloud logs stitching

  • Enrichment with cloud data

  • Detection based on cloud analytics

  • Cloud-tailored investigations

Be sure you do the following tasks before you begin configuring data collection manually from Amazon CloudWatch to Amazon S3.

Note

If you already have an Amazon S3 bucket configured with VPC flow logs that you want to use for this configuration, you do not need to perform the prerequisite steps detailed in the first two bullets.

  • Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS).

    • Amazon S3 bucketGetObject

    • SQSChangeMessageVisibility, ReceiveMessage, and DeleteMessage.

  • Create a dedicated Amazon S3 bucket for collecting network flow logs with the default settings. For more information, see Creating a bucket using the Amazon S3 Console.

    Note

    It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a Lifecycle rule in the Management tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.

  • Ensure that you can access your Amazon Virtual Private Cloud (VPC) and have the necessary permissions to create flow logs.

  • Determine how you want to provide access to Cortex XDR to your logs and perform API operations. You have the following options.

    • Designate an AWS IAM user, where you will need to know the Account ID for the user and have the relevant permissions to create an access key/id for the relevant IAM user. This is the default option as explained in Configure the Amazon S3 collection by selecting Access Key.

    • Create an assumed role in AWS to delegate permissions to a Cortex XDR AWS service. This role grants Cortex XDR access to your flow logs. For more information, see Creating a role to delegate permissions to an AWS service. This is the Assumed Role option as described in the Configure the Amazon S3 collection. For more information on creating an assumed role for Cortex XDR , see Create an Assumed Role.

    To collect Amazon S3 logs that use server-side encryption (SSE), the user role must have an IAM policy that states that Cortex XDR has kms:Decrypt permissions. With this permission, Amazon S3 automatically detects if a bucket is encrypted and decrypts it. If you want to collect encrypted logs from different accounts, you must have the decrypt permissions for the user role also in the key policy for the master account Key Management Service (KMS). For more information, see Allowing users in other accounts to use a KMS key.

Configure Cortex XDR to receive network flow logs from Amazon S3 manually.

  1. Log in to the AWS Management Console.

  2. From the menu bar, ensure that you have selected the correct region for your configuration.

  3. Configure your Amazon Virtual Private Cloud (VPC) with flow logs. For more information, see AWS VPC Flow Logs.

    Note

    If you already have an Amazon S3 bucket configured with VPC flow logs, skip this step and go to Configure an Amazon Simple Queue Service (SQS).

  4. Configure an Amazon Simple Queue Service (SQS). For more information, see Configuring Amazon SQS queues (console).

    Note

    Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.

  5. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket. For more information, see Amazon S3 Event Notifications.

  6. Configure access keys for the AWS IAM user that Cortex XDR uses for API operations. For more information, see Managing access keys for IAM users.

    Note

    • It is the responsibility of the customer’s organization to ensure that the user who performs this task of creating the access key is designated with the relevant permissions. Otherwise, this can cause the process to fail with errors.

    • Skip this step if you are using an Assumed Role for Cortex XDR.

  7. Update the Access Policy of your SQS queue and grant the required permissions mentioned above to the relevant IAM user. For more information, see Granting permissions to publish event notification messages to a destination.

    Note

    Skip this step if you are using an Assumed Role for Cortex XDR.

  8. Configure the Amazon S3 collection in Cortex XDR.

    1. Select SettingsConfigurationsData CollectionCollection Integrations.

    2. In the Amazon S3 configuration, click Add Instance to begin a new configuration.

    3. Set these parameters, where the parameters change depending on whether you configured an Access Key or Assumed Role.

      • To provide access to Cortex XDR to your logs and perform API operations using a designated AWS IAM user, leave the Access Key option selected. Otherwise, select Assumed Role, and ensure that you Create an Assumed Role for Cortex XDR before continuing with these instructions. In addition, when you create an Assumed Role for Cortex XDR, ensure that you edit the policy that defines the permissions for the role with the Amazon S3 Bucket ARN and SQS ARN.

      • SQS URL—Specify the SQS URL, which is the ARN of the Amazon SQS that you configured in the AWS Management Console. For more information on how to retrieve your Amazon SQS ARN, see the Specify SQS queue field when you configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket.

      • Name—Specify a descriptive name for your log collection configuration.

      • When setting an Access Key, set these parameters.

        • AWS Client ID—Specify the Access key ID, which you received when you created access keys for the AWS IAM user in AWS.

        • AWS Client Secret—Specify the Secret access key you received when you created access keys for the AWS IAM user in AWS.

      • When setting an Assumed Role, set these parameters.

        • Role ARN—Specify the Role ARN for the Assumed Role for Cortex XDR in AWS.

        • External Id—Specify the External Id for the Assumed Role for Cortex XDR in AWS.

      • Log Type—Select Flow Logs to configure your log collection to receive network flow logs from Amazon S3. When configuring network flow log collection, the following additional field is displayed for Enhanced Cloud Protection.

        You can Normalize and enrich flow logs by selecting the checkbox. When selected, Cortex XDR ingests the network flow logs as Cortex XDR network connection stories, which you can query using XQL Search from the xdr_dataset dataset using the preset called network_story.

    4. Click Test to validate access, and then click Enable.

      Once events start to come in, a green check mark appears underneath the Amazon S3 configuration with the number of logs received.