Ingest generic logs from Amazon S3 - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Documentation

Product
Cortex XDR
Creation date
2024-03-06
Last date published
2024-11-07
Category
Administrator Guide
Abstract

Take advantage of Cortex XDR investigation capabilities and set up generic log ingestion for your Amazon S3 logs.

You can forward generic logs for the relative service to Cortex XDR from Amazon S3.

To receive generic data from Amazon Simple Storage Service (Amazon S3), you must first configure data collection from Amazon S3. You can then configure the Collection Integrations settings in Cortex XDR for Amazon S3. After you set up collection integration, Cortex XDR begins receiving new logs and data from the source.

Note

For more information on configuring data collection from Amazon S3, see the Amazon S3 Documentation.

As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 Cortex Query Language (XQL) dataset (<Vendor>_<Product>_raw). This enables you to search the logs using XQL Search with the dataset. For example queries, refer to the in-app XQL Library. Cortex XDR can also raise Cortex XDR alerts (Correlation Rules only) when relevant from Amazon S3 logs.

Note

You need to set up an Amazon S3 data collector to receive generic logs when collecting logs from BeyondTrust Privilege Management Cloud. For more information, see Ingest logs from BeyondTrust Privilege Management Cloud.

Prerequisites

Perform the following tasks before you begin configuring data collection from Amazon S3:

  • Create a dedicated Amazon S3 bucket, which collects the generic logs that you want capture. For more information, see Creating a bucket using the Amazon S3 Console.

    Note

    It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a Lifecycle rule in the Management tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.

  • The logs collected by your dedicated Amazon S3 bucket must adhere to the following guidelines.

    • Each log file must use the 1 log per line format as multi-line format is not supported.

    • The log format must be compressed as gzip or uncompressed.

    • For best performance, we recommend limiting each file size to up to 50 MB (compressed).

  • Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS).

    • Amazon S3 bucket: GetObject

    • SQS: ChangeMessageVisibility, ReceiveMessage, and DeleteMessage.

  • Determine how you want to provide access to Cortex XDR to your logs and perform API operations. You have the following options:

    • Designate an AWS IAM user, where you will need to know the Account ID for the user and have the relevant permissions to create an access key/id for the relevant IAM user.

    • Create an assumed role in AWS to delegate permissions to a Cortex XDR AWS service. This role grants Cortex XDR access to your flow logs. For more information, see Creating a role to delegate permissions to an AWS service. This is the Assumed Role option described in the configure the Amazon S3 collection in Cortex XDR. For more information on creating an assumed role for Cortex XDR, see Create an assumed role.

    To collect Amazon S3 logs that use server-side encryption (SSE), the user role must have an IAM policy that states that Cortex XDR has kms:Decrypt permissions. With this permission, Amazon S3 automatically detects if a bucket is encrypted and decrypts it. If you want to collect encrypted logs from different accounts, you must have the decrypt permissions for the user role also in the key policy for the master account Key Management Service (KMS). For more information, see Allowing users in other accounts to use a KMS key.

Configure Cortex XDR to receive generic logs from Amazon S3:

  1. Log in to the AWS Management Console.

  2. From the menu bar, ensure that you have selected the correct region for your configuration.

  3. Configure an Amazon Simple Queue Service (SQS).

    Note

    Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.

    1. In the Amazon SQS Console, click Create Queue.

    2. Configure the following settings, where the default settings should be configured unless otherwise indicated.

      • Type: Select Standard queue (default).

      • Name: Specify a descriptive name for your SQS queue.

      • Configuration section: Leave the default settings for the various fields.

      • Access policyChoose method: Select Advanced and update the Access policy code in the editor window to enable your Amazon S3 bucket to publish event notification messages to your SQS queue. Use this sample code as a guide for defining the “Statement” with the following definitions.

        -“Resource”: Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format “arn:sns:Region:account-id:topic-name”.

        You can retrieve your bucket’s ARN by opening the Amazon S3 Console in a browser window. In the Buckets section, select the bucket that you created for collecting the Amazon S3 flow logs, click Copy ARN, and paste the ARN in the field.

        bucket-copy-arn.png

        Note

        For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.

        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "s3.amazonaws.com"
              },
              "Action": "SQS:SendMessage",
              "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]",
              "Condition": {
                "ArnLike": {
                  "aws:SourceArn": "[ARN of your Amazon S3 bucket]"
                }
              }
            }
          ]
        }
      • Dead-letter queue section: We recommend that you configure a queue for sending undeliverable messages by selecting Enabled, and then in the Choose queue field selecting the queue to send the messages. You may need to create a new queue for this, if you do not already have one set up. For more information, see Amazon SQS dead-letter queues.

    3. Click Create queue.

      Once the SQS is created, a message indicating that the queue was successfully configured is displayed at the top of the page.

  4. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket.

    1. Open the Amazon S3 Console and in the Properties tab of your Amazon S3 bucket, scroll down to the Event notifications section, and click Create event notification.

    2. Configure the following settings:

      • Event name: Specify a descriptive name for your event notification containing up to 255 characters.

      • Prefix: Do not set a prefix as the Amazon S3 bucket is meant to be a dedicated bucket for collecting only network flow logs.

      • Event types: Select All object create events for the type of event notifications that you want to receive.

      • Destination: Select SQS queue to send notifications to an SQS queue to be read by a server.

      • Specify SQS queue: You can either select Choose from your SQS queues and then select the SQS queue, or select Enter SQS queue ARN and specify the ARN in the SQS queue field.

        You can retrieve your SQS queue ARN by opening another instance of the AWS Management Console in a browser window, and opening the Amazon SQS Console, and selecting the Amazon SQS that you created. In the Details section, under ARN, click the copy icon (copy-icon.png)), and paste the ARN in the field.

        sqs-arn2.png
    3. Click Save changes.

      Once the event notification is created, a message indicating that the event notification was successfully created is displayed at the top of the page.

      Note

      If your receive an error when trying to save your changes, you should ensure that the permissions are set up correctly.

  5. Configure access keys for the AWS IAM user.

    Note

    • It is the responsibility of your organization to ensure that the user who performs this task of creating the access key is assigned the relevant permissions. Otherwise, this can cause the process to fail with errors.

    • Skip this step if you are using an Assumed Role for Cortex XDR.

    1. Open the AWS IAM Console, and in the navigation pane, select Access managementUsers.

    2. Select the User name of the AWS IAM user.

    3. Select the Security credentials tab, and scroll down to the Access keys section, and click Create access key.

    4. Click the copy icon () next to the Access key ID and Secret access key keys, where you must click Show secret access key to see the secret key, and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the AWS Client ID and AWS Client Secret in Cortex XDR. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.

      Note

      For more information, see Managing access keys for IAM users.

  6. Update the Access policy of your Amazon SQS queue.

    Note

    Skip this step if you are using an Assumed Role for Cortex XDR.

    1. In the Amazon SQS Console, select the SQS queue that you created when you configured an Amazon Simple Queue Service (SQS).

    2. Select the Access policy tab, and Edit the Access policy code in the editor window to enable the IAM user to perform operations on the Amazon SQS with permissions to SQS:ChangeMessageVisibility, SQS:DeleteMessage, and SQS:ReceiveMessage. Use this sample code as a guide for defining the “Sid”: “__receiver_statement” with the following definitions.

      • “aws:SourceArn”: Specify the ARN of the AWS IAM user. You can retrieve the User ARN from the Security credentials tab, which you accessed when you configured access keys for the AWS API user.

      • “Resource”: Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format “arn:sns:Region:account-id:topic-name”.

        Note

        For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.

        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "s3.amazonaws.com"
              },
              "Action": "SQS:SendMessage",
              "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]",
              "Condition": {
                "ArnLike": {
                  "aws:SourceArn": "[ARN of your Amazon S3 bucket]"
                }
              }
            },
           {
              "Sid": "__receiver_statement",
              "Effect": "Allow",
              "Principal": {
                "AWS": "[Add the ARN for the AWS IAM user]"
              },
              "Action": [
                "SQS:ChangeMessageVisibility",
                "SQS:DeleteMessage",
                "SQS:ReceiveMessage"
              ],
              "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]"
            }
          ]
        }
  7. Configure the Amazon S3 collection in Cortex XDR.

    1. Select SettingsConfigurationsData CollectionCollection Integrations.

    2. In the Amazon S3 configuration, click Add Instance.

    3. Set these parameters, where the parameters change depending on whether you configured an Access Key or Assumed Role.

      • To provide access to Cortex XDR to your logs and perform API operations using a designated AWS IAM user, leave the Access Key option selected. Otherwise, select Assumed Role, and ensure that you create an Assumed Role for Cortex XDR before continuing with these instructions. In addition, when you create an Assumed Role for Cortex XDR, ensure that you edit the policy that defines the permissions for the role with the Amazon S3 Bucket ARN and SQS ARN.

      • SQS URL: Specify the SQS URL, which is the ARN of the Amazon SQS that you configured in the AWS Management Console.

      • Name: Specify a descriptive name for your log collection configuration.

      • When setting an Access Key, set these parameters.

        • AWS Client ID: Specify the Access key ID, which you received when you configured access keys for the AWS IAM user in AWS.

        • AWS Client Secret: Specify the Secret access key you received when you configured access keys for the AWS IAM user in AWS.

      • When setting an Assumed Role, set these parameters.

        • Role ARN: Specify the Role ARN for the Assumed Role you created for Cortex XDR in AWS.

        • External Id: Specify the External Id for the Assumed Role you created for Cortex XDR in AWS.

      • Log Type: Select Generic to configure your log collection to receive generic logs from Amazon S3, which can include different types of data, such as file and metadata. When selecting this option, the following additional fields are displayed.

        • Log Format: Select the log format type as Raw, JSON, CEF, LEEF, Cisco, Corelight, or Beyondtrust Cloud ECS.

          Note

          -The Vendor and Product defaults to Auto-Detect when the Log Format is set to CEF or LEEF.

          -For a Log Format set to CEF or LEEF, Cortex XDR reads events row by row to look for the Vendor and Product configured in the logs. When the values are populated in the event log row, Cortex XDR uses these values even if you specified a value in the Vendor and Product fields in the Amazon S3 data collector settings. Yet, when the values are blank in the event log row, Cortex XDR uses the Vendor and Product that you specified in these fields in the Amazon S3 data collector settings. If you did not specify a Vendor or Product in the Amazon S3 data collector settings, and the values are blank in the event log row, the values for both fields are set to unknown.

          For a Log Format set to Beyondtrust Cloud ECS, the following fields are automatically set and are not configurable:

          -Vendor: Beyondtrust

          -Product: Privilege Management

          -Compression: Uncompressed

          For more information, see Ingest logs from BeyondTrust Privilege Management Cloud.

          For a Log Format set to Cisco, the following fields are automatically set and not configurable.

          -Vendor: Cisco

          -Product: ASA

          For a Log Format set to Corelight, the following fields are automatically set and not configurable:

          -Vendor: Corelight

          -Product: Zeek

          For a Log Format set to Raw or JSON, the following fields are automatically set and are configurable.

          -Vendor: AMAZON

          -Product: AWS

          Cortex XDR supports logs in single line format or multiline format. For a JSON format, multiline logs are collected automatically when the Log Format is configured as JSON. When configuring a Raw format, you must also define the Multiline Parsing Regex as explained below.

        • Vendor: (Optional) Specify a particular vendor name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset <Vendor>_<Product>_raw that Cortex XDR creates as soon as it begins receiving logs.

        • Product: (Optional) Specify a particular product name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset name <Vendor>_<Product>_raw that Cortex XDR creates as soon as it begins receiving logs.

        • Compression: Select whether the logs are compressed into a gzip file or are uncompressed.

        • Multiline Parsing Regex: (Optional) This option is only displayed when the Log Format is set to Raw, where you can set the regular expression that identifies when the multiline event starts in logs with multilines. It is assumed that when a new event begins, the previous one has ended.

    4. Click Test to validate access, and then click Enable.

      Once events start to come in, a green check mark appears underneath the Amazon S3 configuration with the number of logs received.