Datasets and presets

Every Cortex Query Language (XQL) dataset query begins by identifying a data source that the query will run against. Each data source has a unique name, and a series of fields. Your query specifies the data source, and then provides stages that identify fields of interest and perform operations against those fields.

You can query against either datasets or [Presets](#presets) in a dataset query. XQL supports using different languages for dataset and field names. In addition, the dataset formats supported are dependent on the data retention offerings available in Cortex XSIAM according to whether you want to query hot storage (default) or cold storage. For more information, see [XQL Language Structure](xql-language-structure).

### Datasets

The standard, built-in data source that is available in every Cortex XSIAM instance is the `xdr_data` dataset. This is a very large dataset with many available fields. For more information about this dataset, see [Cortex XQL Schema Reference Guide](https://app.gitbook.com/s/6UN9P7f8B5L9QmLYI9Te/). Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, the dataset formats supported are dependent on the data retention offerings available in Cortex XSIAM according to whether you want to query hot storage (default) or cold storage. For more information, see [XQL Language Structure](xql-language-structure).

This dataset is comprised of both raw Endpoint Detection and Response (EDR) events reported by the Cortex XSIAM agent, and of logs from different sources such as third-party logs. To help you investigate events more efficiently, Cortex XSIAM also stitches these logs and events together into common schemas called stories. These stories are available using the Cortex XSIAM [Presets](#presets).

#### Building queries in XQL

When building queries in XQL, keep the following in mind about datasets:

* Use the `dataset` keyword to specify a dataset on your query.
* Create custom datasets using the [target](https://app.gitbook.com/s/fUtoMSNyY2P8jbK3cQsM/readme/stages/target) stage.
* Dataset names can use uppercase characters, but in queries dataset names are always treated as if they are lowercase. In addition, dataset names are supported using different languages, numbers (**`0-9`**), and underscores (**`_`**). Yet, underscores cannot be the first character of the name.
* Upon ingestion, all fields are retained even fields with a null value. You can also use XQL to query parsing rules for null values.
* Schema changes to datasets may not be reflected in the autocomplete suggestions and definitions as you type in real time the XQL query and can appear with a slight delay.

#### Available datasets

Depending on your integrations, you can have the following datasets available for queries:

| Data | Dataset |
| --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Active Directory via Cloud Identity Engine | <p><code>pan_dss_raw</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>Note</strong></p><p>To set up this Cloud Identity Engine (previously called Directory Sync Service (DSS)) dataset, you need to set up a Cloud Identity Engine. Otherwise, you will not have a pan_dss_raw dataset. For more information, see <a href="../../../onboard-cortex-xsiam/deployment-steps/set-up-cloud-identity-engine">Set up Cloud Identity Engine</a>.</p></div> |
| Asset groups | <p><code>asset_groups</code><br>Provides metadata for asset groups. Use this dataset to retrieve the human-readable group name for use in queries, reports, and dashboards.</p> |
| Issues table in Cortex XSIAM | <p>issues</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><ul><li><code>INFO</code> issues are not included in this dataset.</li><li>This dataset includes issues from the Security and Health domains. For more information, see <a href="../../../detect-investigate-and-respond-to-threats/investigation-and-response/investigate-issues/overview-of-the-issues-page">Overview of the Issues page</a>.</li></ul></div> |
| Authentication logs (subset of xdr\_data) | <p>Authentication logs, such as Okta: <code>auth_logs</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>The fields contained in this dataset are a subset of the fields in the <code>xdr_data</code> dataset.</p></div> |
| AWS CloudTrail and Amazon CloudWatch | `<Vendor>_<Product>_raw` |
| Azure Event Hub | <ul><li>All logs: <code>MSFT_Azure_raw</code></li><li>Normalize and enrich audit logs: <code>cloud_audit_logs</code></li></ul> |
| Azure Network Watcher | <ul><li>All logs: <code>MSFT_Azure_raw</code></li><li>Normalize and enrich flow logs: <strong><code>xdr_dataset</code></strong> dataset with a preset called <strong><code>network_story</code></strong></li></ul> |
| BeyondTrust Privilege Management Cloud | `beyondtrust_privilege_management_raw` |
| Box | <p>Events (admin_logs)</p><ul><li><code>box_admin_logs_raw</code></li></ul><p>Box Shield Alerts</p><ul><li><code>box_shield_alerts_raw</code></li></ul><p>Users</p><ul><li><code>box_users_raw</code></li></ul><p>Groups</p><ul><li><code>box_groups_raw</code></li></ul> |
| Checkpoint FW1/VPN1 | `<Vendor>_<Product>_raw` |
| Cisco ASA | <p>Cisco ASA firewalls or Cisco AnyConnect VPN</p><ul><li><code>cisco_asa_raw</code></li></ul> |
| Collector status change audit for collection integrations, custom collectors, and marketplace collectors. | `collection_auditing` |
| Corelight Zeek | `corelight_zeek_raw` |
| Correlation rule executions | `correlations_auditing` |
| Cortex Data Lakes | `xdr_data` |
| Cortex XDR Collectors | `panw_xdrc_raw` |
| Cortex XSIAM Host Firewall enforcement events | `host_firewall_events` |
| CrowdStrike FDR | <ul><li><code>crowdstrike_falcon_incident_raw</code></li><li><code>crowdstrike_fdr_raw</code></li></ul> |
| CSV files in shared Windows directory | Custom datasets: Select from pre-existing user-created datasets or add a new dataset. |
| Database data (MySQL, PostgreSQL, MSSQL, and Oracle) | `<Vendor>_<Product>_raw` |
| Data ingestion health metrics | <p>Datasets:</p><ul><li><code>data_ingestion_health</code></li></ul><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>IMPORTANT: This dataset will not be updated after June 2024. Use the <code>health_alerts</code> dataset instead.</p></div><ul><li><code>metrics_source</code></li></ul><p>Presets:</p><ul><li><code>data_ingestion_metrics</code> (this preset will be deprecated in the next release and replaced by metrics_view).</li><li><code>metrics_view</code></li></ul> |
| Dropbox | <p>Events</p><ul><li><code>dropbox_events_raw</code></li></ul><p>Member Devices</p><ul><li><code>dropbox_members_devices_raw</code></li></ul><p>Users</p><ul><li><code>dropbox_users_raw</code></li></ul><p>Groups</p><ul><li><code>dropbox_groups_raw</code></li></ul> |
| Elasticsearch Filebeat | `<Vendor>_<Product>_raw` |
| Elasticsearch Winlogbeat | <p><code>&#x3C;Vendor>_&#x3C;Product>_raw</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>If the vendor and product are not specified in the Winlogbeat profile’s configuration file, Cortex XSIAM creates a default dataset called <code>microsoft_windows_raw</code>.</p></div> |
| Errors related to Parsing Rules and Data Model Rules | `parsing_rules_errors` |
| Errors related to event forwarding | `event_forwarding_errors` |
| Forcepoint DLP | `forcepoint_dlp_endpoint_raw` |
| Fortinet Fortigate | `<Vendor>_<Product>_raw` |
| GlobalProtect access authentication logs | <p><code>xdr_data</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>To ensure GlobalProtect access authentication logs are sent to Cortex AgentiX, verify that your PANW firewall’s Log Settings for GlobalProtect has the Cortex Data Lake checkbox selected.</p></div> |
| Google Cloud Platform (GCP) logs | <ul><li>All log types: <code>google_cloud_logging_raw</code></li><li><p>Normalize and enrich audit and flow logs: <code>cloud_audit_logs</code></p><ul><li>Audit logs: <code>cloud_audit_logs</code></li><li>Network flow logs: <code>xdr_dataset</code> dataset with a preset called <code>network_story</code></li></ul></li></ul> |
| Google Kubernetes Engine (GKE) | `<Vendor>_<Product>_raw` |
| Google Workspace | <ul><li>Google Chrome: <code>google_workspace_chrome_raw</code></li><li>Admin Console: <code>google_workspace_admin_console_raw</code></li><li>Google Chat: <code>google_workspace_chat_raw</code></li><li>Enterprise Groups: <code>google_workspace_enterprise_groups_raw</code></li><li>Login: <code>google_workspace_login_raw</code></li><li>Rules: <code>google_workspace_rules_raw</code></li><li>Google drive: <code>google_workspace_drive_raw</code></li><li>Token: <code>google_workspace_token_raw</code></li><li>User Accounts: <code>google_workspace_user_accounts_raw</code></li><li>SAML: <code>google_workspace_saml_raw</code></li><li>Alerts: <code>google_workspace_alerts_raw</code></li><li>Emails: <code>google_gmail_raw</code></li></ul> |
| Host Inventory and Vulnerability Assessment | <ul><li><p>Datasets</p><ul><li><code>host_inventory</code></li><li><code>va_cves</code></li><li><code>va_endpoints</code></li></ul></li><li><p>Presets</p><ul><li><code>host_inventory</code></li><li><code>host_inventory_accessibility</code></li><li><code>host_inventory_applications</code></li><li><code>host_inventory_auto_runs</code></li><li><code>host_inventory_cpus</code></li><li><code>host_inventory_daemons</code></li><li><code>host_inventory_disks</code></li><li><code>host_inventory_drivers</code></li><li><code>host_inventory_endpoints</code></li><li><code>host_inventory_extensions</code></li><li><code>host_inventory_groups</code></li><li><code>host_inventory_kbs</code></li><li><code>host_inventory_mounts</code></li><li><code>host_inventory_services</code></li><li><code>host_inventory_shares</code></li><li><code>host_inventory_users</code></li><li><code>host_inventory_volumes</code></li><li><code>host_inventory_vss</code></li></ul></li></ul> |
| Cases table in Cortex XSIAM | `cases` |
| Indicators | `indicators` |
| IT performance metrics | `it_metrics` |
| JSON or text logs from third-party source over HTTP | `<Vendor>_<Product>_raw` |
| Login logs (subset of xdr\_data) | <p>Login logs, such as WEC: <code>login_logs</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>The fields contained in this dataset are a subset of the fields in the <code>xdr_data</code> dataset.</p></div> |
| Logs from third party source over FTP, FTPS, or SFTP | `<Vendor>_<Product>_raw` |
| Microsoft Defender for Endpoint | `msft_defender_raw` |
| Microsoft 365 (email) | <ul><li><code>msft_o365_emails_raw</code></li><li><code>msft_o365_users_raw</code></li><li><code>msft_o365_groups_raw</code></li><li><code>msft_o365_devices_raw</code></li><li><code>msft_o365_mailboxes_raw</code></li><li><code>msft_o365_rules_raw</code></li><li><code>msft_o365_contacts_raw</code></li></ul> |
| Microsoft Office 365 | <ul><li><p>Microsoft Office 365 audit events from Management Activity API:</p><ul><li>Azure AD Activity Logs: <code>msft_o365_azure_ad_raw</code></li><li>Exchange Online: <code>msft_o365_exchange_online_raw</code></li><li>Sharepoint Online: <code>msft_o365_sharepoint_online_raw</code></li><li>DLP: <code>msft_o365_dlp_raw</code></li><li>General: <code>msft_o365_general_raw</code></li></ul></li><li>Microsoft Office 365 emails via Microsoft’s Graph API: <code>msft_o365_emails_raw</code></li><li>Azure AD authentication events from Microsoft Graph API: <code>msft_azure_ad_raw</code></li><li>Azure AD audit events from Microsoft Graph API: <code>msft_azure_ad_audit_raw</code></li><li>Alerts from Microsoft Graph Security API: <code>msft_graph_security_alerts_raw</code></li></ul> |
| NetFlow | <ul><li><code>ip_flow_ip_flow_raw</code> (default)</li><li>When configured, uses the format <code>&#x3C;Vendor>_&#x3C;Product>_raw</code></li></ul> |
| Network Share logs | `<Vendor>_<Product>_raw` |
| Okta | `okta_sso_raw` |
| OneLogin | <p>Log collection</p><ul><li><code>onelogin_events_raw</code></li></ul><p>Directory</p><ul><li><code>onelogin_users_raw</code></li><li><code>onelogin_groups_raw</code></li><li><code>onelogin_apps_raw</code></li></ul> |
| PANW EDR | `xdr_data` |
| PANW IOT Security | <p>Alerts</p><ul><li><code>panw_iot_security_alerts_raw</code></li></ul><p>Devices</p><ul><li><code>panw_iot_security_devices_raw</code></li></ul> |
| PANW NGFW | <p><code>panw_ngfw__raw</code></p><p>Supports the following logs.</p><ul><li>Authentication Logs: <code>panw_ngfw_auth_raw</code></li><li><p>Configuration Logs: <code>panw_ngfw_config_raw</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>Prisma Access firewalls do not send configuration logs to the Structured Log Storage (SLS).</p></div></li><li>File Data Logs: <code>panw_ngfw_filedata_raw</code></li><li>Global Protect Logs: <code>panw_ngfw_globalprotect_raw</code></li><li>Hipmatch Logs: <code>panw_ngfw_hipmatch_raw</code></li><li>System Logs: <code>panw_ngfw_system_raw</code></li><li>Threat Logs: <code>panw_ngfw_threat_raw</code></li><li>Traffic Logs: <code>panw_ngfw_traffic_raw</code></li><li>*URL Logs: <code>panw_ngfw_url_raw</code></li><li>User ID Logs: <code>panw_ngfw_userid_raw</code></li><li>Tunnel Logs: <code>panw_ngfw_tunnel_raw</code></li><li>Configuration Logs: <code>panw_ngfw_config_raw</code></li></ul><p>*These datasets use the query field names as described in the <a href="https://docs.paloaltonetworks.com/cortex/cortex-data-lake/log-forwarding-schema-reference.html">Cortex schema</a> documentation.</p> |
| PingFederate | `ping_identity_pingfederate_raw` |
| PingOne for Enterprise | `pingone_sso_raw` |
| Playbook runs | `playbook_runs` |
| Playbook tasks | `playbook_tasks` |
| Prisma Browser | `panw_prisma_access_browser_raw` |
| Prisma Cloud | `prisma_cloud_raw` |
| Prisma Cloud Compute | `prisma_cloud_compute_raw` |
| Proofpoint Targeted Attack Protection | `proofpoint_tap_raw` |
| Scripts and commands metrics | `scripts_and_commands_metrics` |
| SentinelOne DeepVisibility | `sentinelone_deep_visibility_raw` |
| ServiceNow CMDB | A ServiceNow CMDB dataset is created for each table configured for data collection using the format `servicenow_cmdb_<table name>_raw`. |
| Salesforce.com | <ul><li><code>salesforce_connectedapplication_raw</code></li><li><code>salesforce_permissionset_raw</code></li><li><code>salesforce_profile_raw</code></li><li><code>salesforce_groupmember_raw</code></li><li><code>salesforce_group_raw</code></li><li><code>salesforce_user_raw</code></li><li><code>salesforce_userrole_raw</code></li><li><code>salesforce_document_raw</code></li><li><code>salesforce_contentfolder_raw</code></li><li><code>salesforce_attachment_raw</code></li><li><code>salesforce_contentdistribution_raw</code></li><li><code>salesforce_tenantsecuritylogin_raw</code></li><li><code>salesforce_useraccountteammember_raw</code></li><li><code>salesforce_tenantsecurityuserperm_raw</code></li><li><code>salesforce_account_raw</code></li><li><code>salesforce_audit_raw</code></li><li><code>salesforce_login_raw</code></li><li><code>salesforce_eventlogfile_raw</code></li></ul> |
| Syslog/CEF | `<CEFVendor>_<CEFProduct>_raw` |
| USB devices connect and disconnect events reported by the agent | <p><code>xdr_data</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><ul><li>You can query in XQL for this data and build widgets based on the <code>xdr_data</code> dataset or using the preset <code>device_control</code>.</li><li>To view in an XQL query these events, the Device Configuration of the endpoint profile must be set to Block. Otherwise, the USB events are not captured. The events are also captured when a group of device types are blocked on the endpoints with a permanent or temporary exception in place. For more information, see <a href="../../../../protect-your-endpoints/endpoint-security/install-and-manage-endpoints/harden-endpoint-security/device-control#ingest-connect-and-disconnect-events-of-usb-devices">Ingest Connect and Disconnect Events of USB Devices</a>.</li></ul></div> |
| VPN logs (subset of `xdr_data`) | <p>VPN logs, such as GlobalProtect: <code>vpn_logs</code></p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>The fields contained in this dataset are a subset of the fields in the <code>xdr_data</code> dataset.</p></div> |
| Windows Endpoints using Cortex XDR Forensics Add-on | <ul><li><code>forensics_amcache</code></li><li><code>forensics_application_resource_usage</code></li><li><code>forensics_arp_cache</code></li><li><code>forensics_background_activity_monitor</code></li><li><code>forensics_chrome_history</code></li><li><code>forensics_cid_size_mru</code></li><li><code>forensics_command_history</code></li><li><code>forensics_dns_cache</code></li><li><code>forensics_edge_anaheim_history</code></li><li><code>forensics_edge_spartan_history</code></li><li><code>forensics_event_log</code></li><li><code>forensics_file_access</code></li><li><code>forensics_file_listing</code></li><li><code>forensics_firefox_history</code></li><li><code>forensics_handles</code></li><li><code>forensics_hosts_file</code></li><li><code>forensics_internet_explorer_history</code></li><li><code>forensics_jumplist</code></li><li><code>forensics_last_visited_pidl_mru</code></li><li><code>forensics_log_me_in</code></li><li><code>forensics_net_sessions</code></li><li><code>forensics_network</code></li><li><code>forensics_network_connectivity_usage</code></li><li><code>forensics_network_data_usage</code></li><li><code>forensics_open_save_pidl_mru</code></li><li><code>forensics_port_listing</code></li><li><code>forensics_prefetch</code></li><li><code>forensics_process_execution</code></li><li><code>forensics_process_listing</code></li><li><code>forensics_psreadline</code></li><li><code>forensics_recent_files</code></li><li><code>forensics_recentfilecache</code></li><li><code>forensics_recycle_bin</code></li><li><code>forensics_registry</code></li><li><code>forensics_remote_access</code></li><li><code>forensics_seven_zip_folder_history</code></li><li><code>forensics_shellbags</code></li><li><code>forensics_shimcache</code></li><li><code>forensics_team_viewer</code></li><li><code>forensics_typed_paths</code></li><li><code>forensics_typed_urls</code></li><li><code>forensics_user_access_logging</code></li><li><code>forensics_user_assist</code></li><li><code>forensics_windows_activities</code></li><li><code>forensics_winrar_arc_history</code></li><li><code>forensics_word_wheel_query</code></li></ul> |
| Windows event logs via Cortex XDR Windows agents | `microsoft_windows_raw` |
| Windows Event Collector (WEC) | <ul><li><code>xdr_data</code></li><li><code>microsoft_windows_raw</code></li></ul> |
| Windows DHCP using Elasticsearch Filebeat | `microsoft_dhcp_raw` |
| Windows DNS Debug using Elasticsearch Filebeat | <p>Raw Data</p><ul><li><code>microsoft_dns_raw</code></li></ul><p>Normalized Stories</p><ul><li><code>xdr_data</code> with the preset called <code>network_story</code>.</li></ul> |
| Workday | `workday_workday_raw` |
| Zscaler Cloud Firewall | <p>ZIA</p><ul><li>Firewall logs: <code>zscaler_nssfwlog_raw</code></li><li>Web logs: <code>zscalar_nssweblog_raw</code></li></ul><p>ZPA</p><ul><li><code>zscaler_zpa_raw</code></li></ul> |

### Presets

Presets offer groupings of `xdr_data` fields that are useful for analyzing specific areas of network and endpoint activity. All of the fields available for a preset are also available on the larger `xdr_data` dataset, but by using the preset your query can run more efficiently. Presets are sorted at random by the first one million results found.

Two of the available presets are stories. These contain information stitched together from Cortex XSIAM agent events and log files to form a common schema. They are `authentication_story` and `network_story`.

You use the `preset` keyword to specify a dataset in your query.