Monitor Broker VM using Prometheus - Administrator Guide - Cortex XDR - Cortex - Security Operations

Cortex XDR Documentation
Product
Cortex XDR
Creation date
2024-03-06
Last date published
2024-10-01
Category
Administrator Guide
Abstract

Learn more on monitoring the Broker VM using Prometheus.

You can enable local monitoring of the Broker VM to provide usage statistics in a Prometheus metrics format. You can tap in and export data by navigating to http://<broker_vm_address>:9100/metrics/. By default, monitoring is disabled.

Prerequisite

To monitor the Broker VM using Prometheus, ensure that you enable monitoring on the Broker VM. This is performed after configuring and registering your Broker VM, when you can edit existing configurations and define additional settings in the Broker VMs page.

  1. Select Settings → Configurations → Data Broker → Broker VMs.

  2. In the Broker VMs table, locate your Broker VM, right-click, and select Configure.

    Note

    For all Broker VM nodes added to a HA cluster, you can also Configure the Broker VM nodes from the Clusters tab.

  3. In the Broker VM Configurations page, select Monitoring from the left pane.

  4. Clear the Use Default (Disabled) checkbox.

  5. In the Montoring menu, select Enabled.

  6. Click Save.

How to set up Prometheus and Grafana to monitor the Broker VM

Below is an example of how to set up Prometheus and Grafana to monitor the Broker VM. This is set up using a docker compose on an Ubuntu machine to monitor the CPU usage.

Perform the following procedures in the order listed below.

Task 1. Install Docker and Docker Compose

  1. Update your Ubuntu system:

    sudo apt update

  2. Install Docker:

    Note

    For more information on Docker, see the Docker website.

    sudo apt install docker.io

  3. Start the Docker service:

    sudo systemctl start docker

  4. Enable Docker to start on boot:

    sudo systemctl enable docker

  5. Install Docker Compose:

    sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
    sudo chmod +x /usr/local/bin/docker-compose
Task 2. Create a Docker Compose file

This task includes setting up Prometheus and Grafana.

  1. Create a file named docker-compose.yml, and open it for editing:

    vim docker-compose.yml

  2. Add the following content to the file:

    version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
     - ./prometheus.yml:/etc/prometheus/prometheus.yml
     - prometheus_data:/prometheus
   command:
     - '--config.file=/etc/prometheus/prometheus.yml'
     - '--storage.tsdb.path=/prometheus'
     - '--web.console.libraries=/etc/prometheus/console_libraries'
     - '--web.console.templates=/etc/prometheus/consoles'
     - '--web.enable-lifecycle'
     - '--log.level=debug'
   ports:
     - '9090:9090'
 grafana:
   image: grafana/grafana-enterprise
   container_name: grafana
   restart: unless-stopped
   ports:
    - '3000:3000'
   volumes:
     - grafana_data:/var/lib/grafana
volumes:
  grafana_data: {}
  prometheus_data: {}

  3. Save and close the file.

Task 3. Create a Prometheus configuration file

You need to configure Prometheus to scrape the Broker VM metrics by creating a Prometheus configuration file.

  1. Create a Prometheus configuration file named prometheus.yml in the same directory as the docker-compose.yml file that you created above.

  2. Open the prometheus.yml file for editing:

    vim prometheus.yml

  3. Add the following content to the file:

    global:
  scrape_interval: 15s
  scrape_timeout: 10s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['<your server IP address>:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['<Broker VM IP address>:9100']

  4. Save and close the file.

Task 4. Run Docker Compose

  1. In the terminal, run the following command from the project directory:

    docker-compose up -d

  2. Verify that Prometheus is running correctly:

    docker-compose logs -f prometheus
Task 5. Access Grafana and Set Up Prometheus as a Data Source

  1. Open a web browser and go to http://<your server>:3000.

  2. Log in to Grafana using the default credentials.

    • Username: admin

    • Password: admin

  3. Set up Prometheus as a data source:

    1. In the left pane, select Administation → Data sources.

    2. Click Add data source, and select Prometheus.

    3. Under HTTP, set the URL to http://<your server IP address>:9090.

    4. To verify the connection, click Save & Test.

Task 6. Create Dashboards in Grafana

You can now create dashboards in Grafana to visualize the data from Prometheus.

  1. In Grafana, on the left pane, click Dashboards.

  2. Select New and create a new dashboard.

  3. Add a panel to the dashboard and configure the dashboard to display the Prometheus metrics that you want.

  4. To monitor CPU usage, use the following metric:

    100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)