Set up a local development environment ↗

You can write code directly in the UI, but to contribute a full integration, you need a full development environment external to Cortex XSIAM. An external development environment enables linting your code, running unit tests with pytest, creating documentation, submitting your changes via git and more. There are three options for your external development environment, a traditional local environment described below, a GitHub Codespace environment, or a containerized environment.

Setting up the local development environment involves the following steps:

Verify prerequisites

Cortex XSIAM

GitHub

Go to GitHub and sign in or sign up for an account.

Docker

Make sure Docker is installed on your system and is working correctly by running the hello-world container:

sb@dddd:~/demisto$ docker run --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

[... output omitted for brevity ...]

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

sb@dddd:~/demisto$

Note

If you are using Windows with WSL2, you can still use Docker Desktop from WSL. See here for details.

Fork the GitHub repo

  1. Log in to GitHub, navigate to the Content Repo, and click Fork.

  2. Once the fork is complete, copy the URL.

    This is the fork where you will commit your code and, once ready, create the Pull Request to submit your contribution back to the Cortex XSIAM content repository. Do not work on the master or main branch.

    xsiam-clone-repo.png

Clone the GitHub fork locally

In the shell, create a folder (for example, ~/demisto) and clone your fork of the content repository using git clone [your_fork_url], where [your_fork_url] is the URL you copied from GitHub after forking.

sb@dddd:~$ mkdir demisto
sb@dddd:~$ cd demisto
sb@dddd:~/demisto$ git clone https://github.com/[omitted]/content.git
Cloning into 'content'...
remote: Enumerating objects: 108, done.
remote: Counting objects: 100% (108/108), done.
remote: Compressing objects: 100% (90/90), done.
remote: Total 101143 (delta 50), reused 53 (delta 18), pack-reused 101035
Receiving objects: 100% (101143/101143), 110.65 MiB | 11.04 MiB/s, done.
Resolving deltas: 100% (73634/73634), done.
Checking out files: 100% (4522/4522), done.
sb@dddd:~/demisto$

Note

You must clone your fork of the repository, as you will need to be able to write into it. Do not clone demisto/content, because you won't be able to push commits.

Set up environments

Set up a remote environment

Follow these steps to set up a fully configured remote environment with VS Code Dev Containers.

Verify prerequisites

VS Code

  1. Download and install VS Code.
  2. Install the Remote Development pack.

If you are not familiar with VS Code, follow the VS Code getting started guide.

Docker

Follow the instructions here to install Docker to your operating system.

  • Windows: Docker Desktop 2.0+ on Windows 10 Pro/Enterprise. Windows 10 Home (2004+) requires Docker Desktop 2.3+ and the WSL 2 back-end.
  • macOS: Docker Desktop 2.0+.
  • Linux: Docker CE/EE 18.06+ and Docker Compose 1.21+

Install and enable WSL and Docker

For Windows users, we recommend using Windows Subsystem for Linux (WSL) for better performance.

  1. Install WSL.
  2. Open WSL in VS Code.
  3. Enable Docker support.

  4. Verify WSL 2 is installed by running:

    wsl --list --verbose

    Verify the installed distribution is running WSL 2.

    To change versions, use the command:

    wsl --set-version <distro name> 2

    Replace <distro name> with the name of the Linux distribution that you want to update. For example, wsl --set-version Ubuntu 2 sets your Ubuntu distribution to use WSL 2.

Clone the repository

You can clone the terminal and work directly with VS Code. To work with Github in VS Code, follow the instructions here.

Open the repository in VS Code

If you already cloned the repository with VS Code, go directly to open the dev container.

  1. In VS Code, go to File+Open Folder.
  2. Select your GitHub repository.

Open the Dev Container

  1. Click the green button.

    xsiam-green-button.png

  2. Click Reopen in Container.
  3. It may take a few minutes until the Dev container is ready.

Troubleshooting

Docker performance issues

  • For Windows, use WSL2.
  • Update Docker.
  • Disable Autosave in VS Code.

Since Docker is not native for Mac or Windows, there may be performance issues.

Errors opening the dev container

  • Update Docker.

  • Clean up Docker.

      docker system prune -a --volumes

Set up a local environment

Verify your operating system folder structure

If you are using Windows with WSL and your code resides in a shared folder on the Windows tree (/mnt/c/code/demisto), verify the folder is set to be case sensitive.

Install pyenv and Python

You need Python 3 installed on your system. We recommend using pyenv.

  1. Verfiy pyenv in installed and the eval "$(pyenv init -)" expression is placed in your shell configuration (~/.bashrc or ~/.zshrc).

    sb@dddd:~/demisto$ eval "$(pyenv init -)"
sb@dddd:~/demisto$ pyenv -v
pyenv 1.2.15
sb@dddd:~/demisto$~/demisto$

    See more more details about pyenv installation. Either Homebrew for MacOS or the automatic installer on Linux/WSL work.

  2. If the required version of Python is missing, you need to install it. Because pyenv compiles CPython, you may need some libraries. See more troubleshooting info.

    For example, install Python 3.10.5.

    sb@dddd:~/demisto$ pyenv install 3.10.5
Downloading Python-3.10.5.tar.xz...
-> https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tar.xz
Installing Python-3.10.5...
Installed Python-3.10.5 to /home/sb/.pyenv/versions/3.10.5

sb@dddd:~/demisto$ pyenv versions
  3.10.5
sb@dddd:~/demisto$

Install Poetry

  1. Install Poetry.

  2. Once you set up your development environment, you can activate Poetry with the poetry shell command. Include (.venv) in front of the prompt. Note that the shell command is available in a plugin.

    sb@dddd:~/demisto/content$ poetry shell
(.venv) sb@dddd:~/demisto/content$

    Note

    You can leave the Poetry virtual environment using the deactivate command:

    (.venv) sb@dddd:~/demisto/content$ deactivate
sb@dddd:~/demisto/content$

Install the nvm package manager

  1. Install the nvm package manager.
  2. Run nvm install node.

Install pipx

Pipx is a package that enables you to install and run the Python application globally in an isolated Python environment.

To install Pipx, run the following commands:

pip install --user pipx
pipx ensurepath

Install demisto-sdk

Demisto SDK is a tool that assists in the contribution process. It helps you to generate a content pack, maintain your files, and validate them before committing to the branch. Install Demisto SDK using the pipx command pipx install demisto-sdk --force. To verify you have the latest version of the SDK, run: demisto-sdk --version.

Run the bootstrap script

Before running the bootstrap script that creates the virtual environment, set up pyenv to work correctly in the content folder you just cloned.

Initially, no local python interpreter has been set via pyenv.

sb@dddd:~/demisto$ cd content
sb@dddd:~/demisto/content$ pyenv local
pyenv: no local version configured for this directory

  1. Set pyenv to use the latest version Python 3 you previously installed and verify that everything is set correctly.

    sb@dddd:~/demisto/content$ pyenv local 3.10.5

sb@dddd:~/demisto/content$ pyenv local
3.10.5

sb@dddd:~/demisto/content$ which python3
/home/sb/.pyenv/shims/python3

sb@dddd:~/demisto/content$ python3 -V
Ptyhon 3.10.5

  2. Run the bootstrap script.

    The bootstrap script sets up a pre-commit hook that validates your modified files before committing. It also sets up a Python virtual environment for development with the package requirements for Python3.

    Note

    If you are using WSL and you see errors about "python.exe" getting called, disable it in the App Execution Alias (see more details).

    1. Run the script from the root directory of the source tree: .hooks/bootstrap.

      sb@dddd:~/demisto/content$ .hooks/bootstrap
Installing 'pre-commit' hooks
=======================
Configure poetry to install virtual environment in the project repo (will be available in (.venv)
Check if poetry files are valid
All set!
Installing dependencies...
Detected local env.
Installing dependencies from lock file

No dependencies to install or update
==========================
Done setting up virtualenv with poetry
Activate the venv by running: poetry shell
Deactivate by running: deactivate
=======================
Running: npm install ...

up to date, audited 230 packages in 1s
    2. After the bootstrap script completes, install an extra plugin to get poetry shell to work by running poetry self add poetry-plugin-shell.
    3. After the plugin script completes, activate the newly created virtual environment by running poetry shell.

Run linters and unit tests

Cortex XSIAM content ships with a HelloWorld integration that provides basic functionality and is useful to understand how to create integrations.

It's located in the Packs/HelloWorld/Integrations/HelloWorld folder. demisto-sdk runs the linting and unit testing to verify the dev environment is working (including Python and Docker).

  1. Verify you are running inside the Poetry virtual environment.

    sb@dddd:~/demisto/content$ poetry shell
(.venv) sb@dddd:~/demisto/content$

  2. Run the demisto-sdk pre-commit command on the Packs/HelloWorld/Integrations/HelloWorld folder using the -i option, or if you want to run against all the committed files in your branch you can use demisto-sdk pre-commit -g. It will run both the linters and pytest (unit testing).

    Note

    The tests run within a Docker container, so if the tests pass, it means that your development environment is up and running correctly.

    (venv) sb@dddd:~/demisto/content$ demisto-sdk pre-commit -i Packs/HelloWorld/Integrations/HelloWorld
Running pre-commit using template /Users/sfainberg/dev/demisto/content/.pre-commit-config_template.yaml
Running pre-commit with Python 3.11 on:
Packs/HelloWorld/Integrations/HelloWorld/HelloWorld.py
Packs/HelloWorld/Integrations/HelloWorld/HelloWorld.yml
Packs/HelloWorld/Integrations/HelloWorld/HelloWorld_description.md
Packs/HelloWorld/Integrations/HelloWorld/HelloWorld_image.png
Packs/HelloWorld/Integrations/HelloWorld/HelloWorld_test.py
Packs/HelloWorld/Integrations/HelloWorld/README.md
Packs/HelloWorld/Integrations/HelloWorld/command_examples
Packs/HelloWorld/Integrations/HelloWorld/test_data/get_alert.json
Packs/HelloWorld/Integrations/HelloWorld/test_data/incident_note_list_command.json
Packs/HelloWorld/Integrations/HelloWorld/test_data/ip_reputation.json
DockerHook - Unable to find image docker.io/devtestdemisto/python3:3.11.10.115186-12dd7198e064c21c217cc72c87ddadd5. Creating image based on docker.io/demisto/python3:3.11.10.115186 - Could take 2-3 minutes at first
check json...............................................................Passed
check yaml...............................................................Passed
check python ast.........................................................Passed
check for merge conflicts................................................Passed
debug statements (python)................................................Passed
python tests naming......................................................Passed
check for added large files..............................................Passed
check for case conflicts.................................................Passed
poetry-check.........................................(no files to check)Skipped
pycln....................................................................Passed
ruff-py3.11..............................................................Passed
autopep8.................................................................Passed
mypy-py3.11..............................................................Passed
xsoar-lint...............................................................Passed
pylint-in-docker-demisto/python3:3.11.10.115186..........................Passed
pytest-in-docker-demisto/python3:3.11.10.115186..........................Passed
validate-deleted-files...................................................Passed
validate-content-paths...................................................Passed
validate-conf-json...................................(no files to check)Skipped
validate.................................................................Passed
secrets..................................................................Passed
merge-pytest-reports.....................................................Passed
coverage-pytest-analyze..................................................Passed

Create a branch

The Git flow requires creating a branch with your new code that you will later use to submit a Pull Request.

To create a branch, use the git checkout -b [branch_name] command, where the [branch_name] corresponds to your integration.

(venv) sb@dddd:~/demisto/content$ git checkout -b my_integration_name
Switched to a new branch 'my_integration_name'

Create your integration directory

Create a directory under Packs/<Your pack name>, named after your product where you will put all your content files later, and add it to the staged changes in Git. Use PascalCase in the directory name (for example, MyIntegration). See Content Pack Structure for more information.

You can create a pack and an integration directory using the demisto-sdk init command. The following is an example of creating a pack called MyNewPack, with an integration called MyIntegration, with the metadata file created automatically:

➜  content-docs2 git:(add-pack-and-sdk-docs) ✗ demisto-sdk init --pack 
Please input the name of the initialized pack: MyNewPack
Successfully created the pack test in: MyIntegration

Do you want to fill pack's metadata file? Y/N y

Display name of the pack: MyNewPack

Description of the pack: A description for my newly created pack.

Support type of the pack: 
[1] demisto
[2] partner
[3] developer
[4] community

Enter option: 2

Server min version: 5.0.0

Author of the pack: Partner name 

The url of support, should represent your GitHub account (optional): https://github.com/<PartnerGitAccount>

The email in which you can be contacted in: partner@partner.com

Pack category options: 
[1] Analytics & SIEM
[2] Utilities
[3] Messaging
[4] Endpoint
[5] Network Security
[6] Vulnerability Management
[7] Case Management
[8] Forensics & Malware Analysis
[9] IT Services
[10] Data Enrichment & Threat Intelligence
[11] Authentication
[12] Database
[13] Deception
[14] Email Gateway

Enter option: 1

Tags of the pack, comma separated values: 
Created pack metadata at path : MyNewPack/metadata.json

Do you want to create an integration in the pack? Y/N y
Please input the name of the initialized integration: test
Do you want to use the directory name as an ID for the integration? Y/N y
Finished creating integration: MyNewPack/Integrations/test.

Commit and push

The last step is to commit your changes and push them to the origin to verify that the pre-commit checks pass.

You can also run the hooks locally using Demisto SDK by running the following commands. See this README for more information about Demisto SDK commands and formats.

  • demisto-sdk format - Auto-corrects formatting for the validation to pass.
  • demisto-sdk validate -g - Validates the integrity of the YAML files, and verifies they follow the pre-set roles.
  • demisto-sdk pre-commit -i <The path to your changed/newly added content entity> - Runs lint and pytest on the changed Python files.

  1. Run git commit -m '[some commit message]', which automatically runs the pre-validation checks.

    (venv) sb@dddd:~/demisto/content$ git commit -m 'Initial commit of MyIntegration'
Validating files...
Starting validating files structure
Using git
Running validation on branch my_integration_name
Validates only committed files
Starting validation against origin/master
The files are valid
Starting secrets detection
Finished validating secrets, no secrets were found.

Skipping running dev tasks (flake8, mypy, pylint, pytest). If you want to run this as part of the precommit hook
set CONTENT_PRECOMMIT_RUN_DEV_TASKS=1. You can add the following line to ~/.zshrc:
echo "export CONTENT_PRECOMMIT_RUN_DEV_TASKS=1" >> ~/.zshrc

Or if you want to manually run dev tasks: ./Tests/scripts/pkg_dev_test_tasks.py -d <integration/script dir>
Example: ./Tests/scripts/pkg_dev_test_tasks.py -d Scripts/ParseEmailFiles

On branch my_integration_name
Untracked files:
        .python-version

nothing added to commit but untracked files present

    Note

    • Ignore the .python-version file warning, that is generated by pyenv and is not added to the repository.
    • Since there are no files yet in the directory you created (for example, Integrations/MyIntegration), it will not show up in your branch after the commit. However, you can verify that all the components are in place.

  2. Push to your branch with the command git push origin [branch_name]. You will be prompted for your GitHub credentials.

    (venv) sb@dddd:~/demisto/content$ git push origin my_integration_name
Username for 'https://github.com': [omitted]
Password for 'https://[omitted]@github.com':
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'my_integration_name' on GitHub by visiting:
remote:      https://github.com/[omitted]/content/pull/new/my_integration_name
remote:
To https://github.com/[omitted]/content
 * [new branch]          my_integration_name -> my_integration_name
(venv) sb@dddd:~/demisto/content$

  3. Go back to GitHub and under your fork you should be able to see that there is a new branch with the name you provided (for example, my_integration_name).

    xsiam-github-branch.png

    The development environment for Cortex XSIAM is complete, with a fully configured virtual environment where you can run different validation and utility scripts. You can now start writing your code.