How to build XQL queries ↗

The Cortex Query Language (XQL) enables you to query data ingested into Cortex XSIAM for rigorous endpoint and network event analysis. To help you create an effective XQL query with the proper syntax, the query field in the user interface provides suggestions and definitions as you type.

XQL forms queries in stages. Each stage performs a specific query operation and is separated by a pipe character ( ). Queries require a dataset, or data source, to run against. You can either query the Cortex Data Model (XDM) or you can query specific datasets. In a dataset query, unless otherwise specified, the query runs against the xdr_data dataset, which contains all log information that Cortex XSIAM collects from all Cortex product agents, including EDR data, and PAN NGFW data. In XDM queries, you must specify the dataset mapped to the XDM that you want to run your query against.

Forensic datasets are not included by default in XQL query results, unless the dataset query is explicitly defined to use a forensic dataset.

Which datasets are mapped to XDM?

The Cortex Query Language (XQL) supports a single Cortex Data Model (XDM), which is a normalized data structure. Datasets are mapped to the XDM in 3 different ways:

  1. Automatic default mappings, including the following:
    • The xdr_data dataset is automatically mapped to the XDM with some data mapping exceptions.
    • Next-Generation Firewall (NGFW) network log data are mapped to the XDM from the following datasets:
      • panw_ngfw_traffic_raw
      • panw_ngfw_threat_raw
      • panw_ngfw_url_raw
      • panw_ngfw_filedata_raw
      • panw_ngfw_globalprotect_raw
      • panw_ngfw_hipmatch_raw
  2. Out-of-the-box mappings of the datasets as part of the Data Model Rules via the Marketplace. For more information, see Cortex Marketplace.
  3. You can create your own mappings by creating your own Data Model Rules. For more information, see Create Data Model Rules.

For more information on the XDM Schema, specifically the fields, fieldsets, fields designated as ENUMS (CONST), and aliases, see the Cortex XSIAM Data Model Schema.

XDM query syntax

The basic syntax structure for querying the Cortex Data Model (XDM) is either:

datamodel dataset in (<dataset_name>,...) …
    | <STAGE> ...
    | <STAGE> ...
    | <STAGE> ...

or

datamodel dataset = <dataset_name> …
    | <STAGE> ...
    | <STAGE> ...
    | <STAGE> ...

In a query using the datamodel command, a query runs against the specified datasets, which contain log information ingested by Cortex XSIAM. You can also install Marketplace Content Packs, or map an ingested dataset into the XDM, to query additional datasets.

Adding a wildcard suffix (*) is supported in the <dataset_name>, which matches all datasets that are mapped to the data model and begin with the specified text. For example, datamodel dataset = xdr* or datamodel dataset in (xdr*).

When querying the XDM, fields that are not mapped to the XDM are accessible by <dataset>.<field>. They can be used at any stage of a datamodel query.

When creating XDM queries, auto-suggestions are available, according to the existing XDM fields.

Dataset query syntax

In a dataset query, unless otherwise specified, the query runs against the xdr_data dataset, which contains all log information that Cortex XSIAM collects from all Cortex product agents, including EDR data, and PAN NGFW data. In a dataset query, if you are running your query against a dataset that has been set as default, there is no need to specify a dataset. Otherwise, specify a dataset in your query. The Dataset Queries lists the available datasets, depending on system configuration.

  • Users with different dataset permissions can receive different results for the same XQL query.
  • An administrator or a user with a predefined user role can create and view queries built with an unknown dataset that currently does not exist in Cortex XSIAM. All other users can only create and view queries built with an existing dataset.
  • When you have more than one dataset or lookup, you can change your default dataset by navigating to Settings → Configurations → Data Management → Dataset Management, right-click on the appropriate dataset, and select Set as default. For more information about setting default datasets, see Dataset management.

The basic syntax structure for querying datasets that are not mapped to the XDM is:

dataset = <dataset name> 
    | <stage1> ...
    | <stage2> ... 
    | <stage3> ...

or

dataset in (<dataset name>)
    | <stage1> ...
    | <stage2> ...
    | <stage3> ...

You can specify a dataset using one of the following formats, which is based on the data retention offerings available in Cortex XSIAM.

  • Hot Storage queries use the format dataset = <dataset name>. This is the default option.

    Example 88.

    dataset = xdr_data

  • Cold Storage queries use the format cold_dataset = <dataset name>.

    cold_dataset = xdr_data

    You can build a query that investigates data in both a cold dataset and a hot dataset in the same query. In addition, as the hot storage dataset format is the default option and represents the fully searchable storage, this format is used throughout this guide for investigation and threat hunting. For more information on hot and cold storage, see Dataset management.

When using the hot storage default format, this returns every xdr_data record contained in your Cortex XSIAM instance over the time range that you provide to the Query Builder user interface. This can be a large amount of data, which may take a long time to retrieve. You can use a limit stage to specify how many records you want to retrieve.

There is no practical limit to the number of stages that you can specify. See Stages for information on all the supported stages.

In the xdr_data dataset, every user field included in the raw data for network, authentication, and login events has an equivalent normalized user field associated with it that displays the user information in the following standardized format:

<company domain>\<username>

For example, the login_data field has the login_data_dst_normalized_user field to display the content in the standardized format. To ensure the most accurate results, we recommend that you use these normalized_user fields when building your queries.

Additional components

XQL queries can contain different components, such as functions and stages, depending on the type of query you want to build. For a complete list of the syntax options available with example queries, see Stages and Functions.