Ingest Logs and Data from a GCP Pub/Sub - Administrator Guide - Cortex XSIAM - Cortex - Security Operations

Cortex XSIAM Documentation

Product
Cortex XSIAM
Creation date
2024-03-06
Last date published
2025-02-11
Category
Administrator Guide
Abstract

If you use the Pub/Sub messaging service from Global Cloud Platform (GCP), you can send logs and data from GCP to Cortex XSIAM.

If you use the Pub/Sub messaging service from Global Cloud Platform (GCP), you can send logs and data from your GCP instance to Cortex XSIAM. Data from GCP is then searchable in Cortex XSIAM to provide additional information and context to your investigations using the GCP Cortex Query Language (XQL) dataset, which is dependent on the type of GCP logs collected. For example queries, refer to the in-app XQL Library. You can configure a Google Cloud Platform collector to receive generic, flow, audit, or Google Cloud DNS logs. When configuring generic logs, you can receive logs in a Raw, JSON, CEF, LEEF, Cisco, or Corelight format.

You can also configure Cortex XSIAM to normalize different GCP logs as part of the enhanced cloud protection, which you can query with XQL Search using the applicable dataset. Cortex XSIAM can also raise Cortex XSIAM alerts (Analytics, IOC, BIOC, and Correlation Rules) when relevant from GCP logs. While Correlation Rules alerts are raised on non-normalized and normalized logs, Analytics, IOC, and BIOC alerts are only raised on normalized logs.

Enhanced cloud protection provides the following:

  • Normalization of cloud logs

  • Cloud logs stitching

  • Enrichment with cloud data

  • Detection based on cloud analytics

  • Cloud-tailored investigations

The following table lists the various GCP log types the XQL datasets you can use to query in XQL Search:

GCP log type

Dataset

Dataset with normalized data

Audit logs, including Google Kubernetes Engine (GKE) audit logs

google_cloud_logging_raw

cloud_audit_logs

Generic logs

Log Format types:

  • CEF or LEEF: Automatically detected from either the logs or the user's input in the User Interface.

  • Cisco: cisco_asa_raw

  • Corelight: corelight_zeek_raw

  • JSON or Raw: google_cloud_logging_raw

N/A

Google Cloud DNS logs

google_dns_raw

xdr_data: Once configured, Cortex XSIAM ingests Google Cloud DNS logs as XDR network connection stories, which you can query with XQL Search using the xdr_data dataset with the preset called network_story.

Network flow logs

google_cloud_logging_raw

xdr_data: Once configured, Cortex XSIAM ingests network flow logs as XDR network connection stories, which you can query with XQL Search using the xdr_data dataset with the preset called network_story.

Note

When collecting flow logs, we recommend that you include GKE annotations in your logs, which enable you to view the names of the containers that communicated with each other. GKE annotations are only included in logs if appended manually using the custom metadata configuration in GCP. For more information, see VPC Flow Logs Overview. In addition, to customize metadata fields, you must use the gcloud command-line interface or the API. For more information, see Using VPC Flow Logs.

To receive logs and data from GCP, you must first set up log forwarding using a Pub/Sub topic in GCP. You can configure GCP settings using either the GCP web interface or a GCP cloud shell terminal. After you set up your service account in GCP, you configure the Data Collection settings in Cortex XSIAM. The setup process requires the subscription name and authentication key from your GCP instance.

After you set up log collection, Cortex XSIAM immediately begins receiving new logs and data from GCP.

  1. Log in to your GCP account.

  2. Set up log forwarding from GCP to Cortex XSIAM.

    1. Select LoggingLogs Router.

    2. Select Create SinkCloud Pub/Sub topic, and then click Next.

    3. To filter only specific types of data, select the filter or desired resource.

    4. In the Edit Sink configuration, define a descriptive Sink Name.

    5. Select Sink DestinationCreate new Cloud Pub/Sub topic.

    6. Enter a descriptive Name that identifies the sink purpose for Cortex XSIAM, and then Create.

    7. Create Sink and then Close when finished.

  3. Create a subscription for your Pub/Sub topic.

    1. Select the hamburger menu in G Cloud and then select Pub/SubTopics.

    2. Select the name of the topic you created in the previous steps. Use the filters if necessary.

    3. Create SubscriptionCreate subscription.

    4. Enter a unique Subscription ID.

    5. Choose Pull as the Delivery Type.

    6. Create the subscription.

      After the subscription is set up, G Cloud displays statistics and settings for the service.

    7. In the subscription details, identify and note your Subscription Name.

      Optionally, use the copy button to copy the name to the clipboard. You will need the name when you configure Collection in Cortex XSIAM.

  4. Create a service account and authentication key.

    You will use the key to enable Cortex XSIAM to authenticate with the subscription service.

    1. Select the menu icon, and then select IAM & AdminService Accounts.

    2. Create Service Account.

    3. Enter a Service account name and then Create.

    4. Select a role for the account: Pub/SubPub/Sub Subscriber.

    5. Click ContinueDone.

    6. Locate the service account by name, using the filters to refine the results, if needed.

    7. Click the Actions menu identified by the three dots in the row for the service account and then Create Key.

    8. Select JSON as the key type, and then Create.

      After you create the service account key, G Cloud automatically downloads it.

  1. Launch the GCP cloud shell terminal or use your preferred shell with gcloud installed.

    gcp-cli.png
  2. Define your project ID.

    gcloud config set project <PROJECT_ID>
                         
  3. Create a Pub/Sub topic.

    gcloud pubsub topics create <TOPIC_NAME>
                         
  4. Create a subscription for this topic.

    gcloud pubsub subscriptions create <SUBSCRIPTION_NAME> --topic=<TOPIC_NAME>
                         

    Note the subscription name you define in this step as you will need it to set up log ingestion from Cortex XSIAM.

  5. Create a logging sink.

    During the logging sink creation, you can also define additional log filters to exclude specific logs. To filter logs, supply the optional parameter --log-filter=<LOG_FILTER>

    gcloud logging sinks create <SINK_NAME> pubsub.googleapis.com/projects/<PROJECT_ID>/topics/<TOPIC_NAME> --log-filter=<LOG_FILTER>
                         

    If setup is successful, the console displays a summary of your log sink settings:

    Created [https://logging.googleapis.com/v2/projects/PROJECT_ID/sinks/SINK_NAME]. Please remember to grant `serviceAccount:LOGS_SINK_SERVICE_ACCOUNT` \ the Pub/Sub Publisher role on the topic. More information about sinks can be found at /logging/docs/export/configure_export
  6. Grant log sink service account to publish to the new topic.

    Note the serviceAccount name from the previous step and use it to define the service for which you want to grant publish access.

    gcloud pubsub topics add-iam-policy-binding <TOPIC_NAME> --member serviceAccount:<LOGS_SINK_SERVICE_ACCOUNT> --role=roles/pubsub.publisher
  7. Create a service account.

    For example, use cortex-xdr-sa as the service account name and Cortex XSIAM Service Account as the display name.

    gcloud iam service-accounts create <SERVICE_ACCOUNT> --description="<DESCRIPTION>" --display-name="<DISPLAY_NAME>"
  8. Grant the IAM role to the service account.

    gcloud pubsub subscriptions add-iam-policy-binding <SUBSCRIPTION_NAME> --member serviceAccount:<SERVICE_ACCOUNT>@<PROJECT_ID>.iam.gserviceaccount.com --role=roles/pubsub.subscriber
  9. Create a JSON key for the service account.

    You will need the JSON file to enable Cortex XSIAM to authenticate with the GCP service. Specify the file destination and filename using a .json extension.

    gcloud iam service-accounts keys create <OUTPUT_FILE> --iam-account <SERVICE_ACCOUNT>@<PROJECT_ID>.iam.gserviceaccount.com
  10. After Cortex XSIAM begins receiving information from the GCP Pub/Sub service, you can use the XQL Query language to search for specific data.