Set up a local development environment - Developer Guide - Cortex XSIAM - Cortex

Cortex XSIAM

Deploy Cortex XSIAM. See Setup overview for more details.
Review Before you start and Contribution requirements.Before You Start

GitHub

Go to GitHub and sign in or sign up for an account.

Docker

Make sure Docker is installed on your system and is working correctly by running the hello-world container:

sb@dddd:~/demisto$ docker run --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

[... output omitted for brevity ...]

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

sb@dddd:~/demisto$

Note

If you are using Windows with WSL2, you can still use Docker Desktop from WSL. See here for details.

Log in to GitHub, navigate to the Content Repo, and click Fork.
Once the fork is complete, copy the URL.
This is the fork where you will commit your code and, once ready, create the Pull Request to submit your contribution back to the Cortex XSIAM content repository. Do not work on the master or main branch.

In the shell, create a folder (for example, ~/demisto) and clone your fork of the content repository using git clone [your_fork_url], where [your_fork_url] is the URL you copied from GitHub after forking.

sb@dddd:~$ mkdir demisto
sb@dddd:~$ cd demisto
sb@dddd:~/demisto$ git clone https://github.com/[omitted]/content.git
Cloning into 'content'...
remote: Enumerating objects: 108, done.
remote: Counting objects: 100% (108/108), done.
remote: Compressing objects: 100% (90/90), done.
remote: Total 101143 (delta 50), reused 53 (delta 18), pack-reused 101035
Receiving objects: 100% (101143/101143), 110.65 MiB | 11.04 MiB/s, done.
Resolving deltas: 100% (73634/73634), done.
Checking out files: 100% (4522/4522), done.
sb@dddd:~/demisto$

Note

You must clone your fork of the repository, as you will need to be able to write into it. Do not clone demisto/content, because you won't be able to push commits.

Set up a remote environment

Follow these steps to set up a fully configured remote environment with VS Code Dev Containers.

For Windows users, we recommend using Windows Subsystem for Linux (WSL) for better performance.

Install WSL.
Open WSL in VS Code.
Enable Docker support.
Verify WSL 2 is installed by running:
```
wsl --list --verbose
```
Verify the installed distribution is running WSL 2.
To change versions, use the command:
```
wsl --set-version <distro name> 2
```
Replace <distro name> with the name of the Linux distribution that you want to update. For example, wsl --set-version Ubuntu 2 sets your Ubuntu distribution to use WSL 2.

Set up a local environment

You need Python 3 installed on your system. We recommend using pyenv.

Verfiy pyenv in installed and the eval "$(pyenv init -)" expression is placed in your shell configuration (~/.bashrc or ~/.zshrc).
```
sb@dddd:~/demisto$ eval "$(pyenv init -)"
sb@dddd:~/demisto$ pyenv -v
pyenv 1.2.15
sb@dddd:~/demisto$~/demisto$
```
See more more details about pyenv installation. Either Homebrew for MacOS or the automatic installer on Linux/WSL work.

If the required version of Python is missing, you need to install it. Because pyenv compiles CPython, you may need some libraries. See more troubleshooting info.

For example, install Python 3.10.5.

sb@dddd:~/demisto$ pyenv install 3.10.5
Downloading Python-3.10.5.tar.xz...
-> https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tar.xz
Installing Python-3.10.5...
Installed Python-3.10.5 to /home/sb/.pyenv/versions/3.10.5

sb@dddd:~/demisto$ pyenv versions
  3.10.5
sb@dddd:~/demisto$

Before running the bootstrap script that creates the virtual environment, set up pyenv to work correctly in the content folder you just cloned.

Initially, no local python interpreter has been set via pyenv.

sb@dddd:~/demisto$ cd content
sb@dddd:~/demisto/content$ pyenv local
pyenv: no local version configured for this directory

Set pyenv to use the latest version Python 3 you previously installed and verify that everything is set correctly.

sb@dddd:~/demisto/content$ pyenv local 3.10.5

sb@dddd:~/demisto/content$ pyenv local
3.10.5

sb@dddd:~/demisto/content$ which python3
/home/sb/.pyenv/shims/python3

sb@dddd:~/demisto/content$ python3 -V
Ptyhon 3.10.5

Run the bootstrap script.

The script sets up a pre-commit hook that validates your modified files before committing. It also sets up a Python virtual environment for development with the package requirements for Python3. Run the script from the root directory of the source tree: .hooks/bootstrap. After the script completes, activate the newly created virtual environment: poetry shell.

sb@dddd:~/demisto/content$ .hooks/bootstrap
Installing 'pre-commit' hooks
=======================
Configure poetry to install virtual environment in the project repo (will be available in (.venv)
Check if poetry files are valid
All set!
Installing dependencies...
Detected local env.
Installing dependencies from lock file

No dependencies to install or update
==========================
Done setting up virtualenv with poetry
Activate the venv by running: poetry shell
Deactivate by running: deactivate
=======================
Running: npm install ...

up to date, audited 230 packages in 1s

Note

If you are using WSL and you see errors about "python.exe" getting called, disable it in the App Execution Alias (see more details).

Cortex XSIAM content ships with a HelloWorld integration that provides basic functionality and is useful to understand how to create integrations.

It's located in the Packs/HelloWorld/Integrations/HelloWorld folder. demisto-sdk runs the linting and unit testing to verify the dev environment is working (including Python and Docker).

Verify you are running inside the Poetry virtual environment.

sb@dddd:~/demisto/content$ poetry shell
(.venv) sb@dddd:~/demisto/content$

Run the demisto-sdk lint command on the Packs/HelloWorld/Integrations/HelloWorld folder using the -i option, or if you want to run against all the committed files in your branch you can use demisto-sdk lint -g. It will run both the linters and pytest (unit testing).

Note

The tests run within a Docker container, so if the tests pass, it means that your development environment is up and running correctly.

(venv) sb@dddd:~/demisto/content$ demisto-sdk lint -i Packs/HelloWorld/Integrations/HelloWorld
Execute lint and test on 1/1 packages
HelloWorld - Facts - Using yaml file /home/sb/dev/demisto/content/Packs/HelloWorld/Integrations/HelloWorld/HelloWorld.yml
HelloWorld - Facts - Pulling docker images, can take up to 1-2 minutes if not exists locally 
HelloWorld - Facts - demisto/python3:3.8.2.6981 - Python 3.8
HelloWorld - Facts - Tests found
HelloWorld - Facts - Lint file /home/sb/dev/demisto/content/Packs/HelloWorld/Integrations/HelloWorld/HelloWorld_test.py
HelloWorld - Facts - Lint file /home/sb/dev/demisto/content/Packs/HelloWorld/Integrations/HelloWorld/HelloWorld.py
HelloWorld - Flake8 - Start
HelloWorld - Flake8 - Successfully finished
HelloWorld - Bandit - Start
HelloWorld - Bandit - Successfully finished
HelloWorld - Mypy - Start
HelloWorld - Mypy - Successfully finished
HelloWorld - Vulture - Start
HelloWorld - Vulture - Successfully finished
HelloWorld - Flake8 - Start
HelloWorld - Flake8 - Successfully finished
HelloWorld - Image create - Trying to pull existing image devtestdemisto/python3:3.8.2.6981-02b43abe979132c89892e089d5b8254d
HelloWorld - Image create - Found existing image devtestdemisto/python3:3.8.2.6981-02b43abe979132c89892e089d5b8254d
HelloWorld - Image create - Copy pack dir to image devtestdemisto/python3:3.8.2.6981-02b43abe979132c89892e089d5b8254d
HelloWorld - Image create - Image sha256:ba9f6ede55 created successfully
HelloWorld - Pylint - Image sha256:ba9f6ede55 - Start
HelloWorld - Pylint - Image sha256:ba9f6ede55 - exit-code: 0
HelloWorld - Pylint - Image sha256:ba9f6ede55 - Successfully finished
HelloWorld - Pytest - Image sha256:ba9f6ede55 - Start
        ============================= test session starts ==============================
        platform linux -- Python 3.8.2, pytest-5.0.1, py-1.8.1, pluggy-0.13.1
        rootdir: /devwork
        plugins: json-0.4.0, forked-1.1.3, mock-2.0.0, asyncio-0.10.0, datadir-ng-1.1.1, requests-mock-1.7.0, xdist-1.31.0
        collected 10 items

        HelloWorld_test.py ..........                                            [100%]

        -------------- generated json report: /devwork/report_pytest.json --------------
        ========================== 10 passed in 0.43 seconds ===========================
HelloWorld - Pytest - Image sha256:ba9f6ede55 - exit-code: 0
HelloWorld - Pytest - Image sha256:ba9f6ede55 - Successfully finished
Flake8       - [PASS]
Bandit       - [PASS]
Mypy         - [PASS]
Vulture      - [PASS]
Pytest       - [PASS]
Pylint       - [PASS]
Pwsh analyze - [SKIPPED]
Pwsh test    - [SKIPPED]

Passed Unit-tests:
  - Package: HelloWorld
      - Image: demisto/python3:3.8.2.6981
         - HelloWorld_test.py::test_say_hello
         - HelloWorld_test.py::test_start_scan
         - HelloWorld_test.py::test_status_scan
         - HelloWorld_test.py::test_scan_results
         - HelloWorld_test.py::test_search_alerts
         - HelloWorld_test.py::test_get_alert
         - HelloWorld_test.py::test_update_alert_status
         - HelloWorld_test.py::test_ip
         - HelloWorld_test.py::test_domain
         - HelloWorld_test.py::test_fetch_incidents

#########
 Summary 
#########
Packages: 1
Packages PASS: 1
Packages FAIL: 0

The Git flow requires creating a branch with your new code that you will later use to submit a Pull Request.

To create a branch, use the git checkout -b [branch_name] command, where the [branch_name] corresponds to your integration.

(venv) sb@dddd:~/demisto/content$ git checkout -b my_integration_name
Switched to a new branch 'my_integration_name'

Create a directory under Packs/<Your pack name>, named after your product where you will put all your content files later, and add it to the staged changes in Git. Use PascalCase in the directory name (for example, MyIntegration). See Content Pack Structure for more information.

You can create a pack and an integration directory using the demisto-sdk init command. The following is an example of creating a pack called MyNewPack, with an integration called MyIntegration, with the metadata file created automatically:

➜  content-docs2 git:(add-pack-and-sdk-docs) ✗ demisto-sdk init --pack 
Please input the name of the initialized pack: MyNewPack
Successfully created the pack test in: MyIntegration

Do you want to fill pack's metadata file? Y/N y

Display name of the pack: MyNewPack

Description of the pack: A description for my newly created pack.

Support type of the pack: 
[1] demisto
[2] partner
[3] developer
[4] community

Enter option: 2

Server min version: 5.0.0

Author of the pack: Partner name 

The url of support, should represent your GitHub account (optional): https://github.com/<PartnerGitAccount>

The email in which you can be contacted in: partner@partner.com

Pack category options: 
[1] Analytics & SIEM
[2] Utilities
[3] Messaging
[4] Endpoint
[5] Network Security
[6] Vulnerability Management
[7] Case Management
[8] Forensics & Malware Analysis
[9] IT Services
[10] Data Enrichment & Threat Intelligence
[11] Authentication
[12] Database
[13] Deception
[14] Email Gateway

Enter option: 1

Tags of the pack, comma separated values: 
Created pack metadata at path : MyNewPack/metadata.json

Do you want to create an integration in the pack? Y/N y
Please input the name of the initialized integration: test
Do you want to use the directory name as an ID for the integration? Y/N y
Finished creating integration: MyNewPack/Integrations/test.

The last step is to commit your changes and push them to the origin to verify that the pre-commit checks pass.

You can also run the hooks locally using Demisto SDK by running the following commands. See this README for more information about Demisto SDK commands and formats.

demisto-sdk format - Auto-corrects formatting for the validation to pass.
demisto-sdk validate -g - Validates the integrity of the YAML files, and verifies they follow the pre-set roles.
demisto-sdk lint -i <The path to your changed/newly added content entity> - Runs lint and pytest on the changed Python files.

Run git commit -m '[some commit message]', which automatically runs the pre-validation checks.

(venv) sb@dddd:~/demisto/content$ git commit -m 'Initial commit of MyIntegration'
Validating files...
Starting validating files structure
Using git
Running validation on branch my_integration_name
Validates only committed files
Starting validation against origin/master
The files are valid
Starting secrets detection
Finished validating secrets, no secrets were found.

Skipping running dev tasks (flake8, mypy, pylint, pytest). If you want to run this as part of the precommit hook
set CONTENT_PRECOMMIT_RUN_DEV_TASKS=1. You can add the following line to ~/.zshrc:
echo "export CONTENT_PRECOMMIT_RUN_DEV_TASKS=1" >> ~/.zshrc

Or if you want to manually run dev tasks: ./Tests/scripts/pkg_dev_test_tasks.py -d <integration/script dir>
Example: ./Tests/scripts/pkg_dev_test_tasks.py -d Scripts/ParseEmailFiles

On branch my_integration_name
Untracked files:
        .python-version

nothing added to commit but untracked files present

Note

Ignore the .python-version file warning, that is generated by pyenv and is not added to the repository.
Since there are no files yet in the directory you created (for example, Integrations/MyIntegration), it will not show up in your branch after the commit. However, you can verify that all the components are in place.

Push to your branch with the command git push origin [branch_name]. You will be prompted for your GitHub credentials.

(venv) sb@dddd:~/demisto/content$ git push origin my_integration_name
Username for 'https://github.com': [omitted]
Password for 'https://[omitted]@github.com':
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'my_integration_name' on GitHub by visiting:
remote:      https://github.com/[omitted]/content/pull/new/my_integration_name
remote:
To https://github.com/[omitted]/content
 * [new branch]          my_integration_name -> my_integration_name
(venv) sb@dddd:~/demisto/content$

Go back to GitHub and under your fork you should be able to see that there is a new branch with the name you provided (for example, my_integration_name).
The development environment for Cortex XSIAM is complete, with a fully configured virtual environment where you can run different validation and utility scripts. You can now start writing your code.

Set up a local development environment - Developer Guide - Cortex XSIAM - Cortex - Security Operations

Cortex XSIAM Developer Guide

Cortex XSIAM

GitHub

Docker

Note

Note

Set up a remote environment

VS Code

Docker

Open the repository in VS Code

Open the Dev Container

Docker performance issues

Errors opening the dev container

Set up a local environment

Note

Note

Note

Note