AzureDataExplorer
Use the Azure Data Explorer integration to collect and analyze data inside Azure Data Explorer clusters, and to manage search queries.
Analytics & SIEM · Azure Data Explorer
Details
| ID | AzureDataExplorer |
|---|---|
| Provider | Microsoft |
| Category | Analytics & SIEM |
| From Version | 6.0.0 |
| Docker Image | demisto/auth-utils:1.0.0.10133006 |
| Supported Modules | Agentix XSIAM |
README
Use the Azure Data Explorer integration to collect and analyze data inside Azure Data Explorer clusters, and to manage search queries.
This integration was integrated and tested with version V1 of AzureDataExplorer.
Authorization
In order to connect to the Azure Data Explorer using either Cortex XSOAR Azure App or the Self-Deployed Azure App, use one of the following methods:
- Authorization Code Flow (Recommended).
- Device Code Flow.
- Client Credentials Flow.
Self-Deployed Azure App
To use a self-configured Azure application, you need to add a new Azure App Registration in the Azure Portal.
To add the registration, refer to the following Microsoft article steps 1-8.
Required permissions
- Azure Data Explorer - permission
user_impersonationof type Delegated. - Microsoft Graph - permission
offline_accessof type Delegated.
To add a permission:
- Navigate to Azure Poral > Home > App registrations.
- Search for your app under ‘all applications’.
- Click API permissions > Add permission.
- Search for the specific Microsoft API and select the specific permission of type Delegated.
Authentication Using the Authorization Code Flow (recommended)
- In the Authentication Type field, select the Authorization Code option.
- In the Application ID field, enter your Client/Application ID.
- In the Client Secret field, enter your Client Secret.
- In the Tenant ID field, enter your Tenant ID .
- In the Application redirect URI field, enter your Application redirect URI.
- Save the instance.
- Run the
!azure-data-explorer-generate-login-urlcommand in the War Room and follow the instruction. - Save the instance.
Authentication Using the Device Code Flow
Use the device code flow
to link Azure Data Explorer with Cortex XSOAR.
- Fill in the required parameters.
- In the Authentication Type field, select the Device Code option.
- Run the !azure-data-explorer-auth-start command.
- Follow the instructions that appear.
- Run the !azure-data-explorer-auth-complete command.
At end of the process you’ll see a message that you’ve logged in successfully.
Cortex XSOAR Azure App
In order to use the Cortex XSOAR Azure application, use the default application ID (a9ce8db2-847a-46af-9bfb-725d8a8d3c53).
Authentication Using the Client Credentials Flow
-
Assign Azure roles using the Azure portal Microsoft article
Note: In the Select members section, assign the application you created earlier.
-
To configure a Microsoft integration that uses this authorization flow with a self-deployed Azure application:
a. In the Authentication Type field, select the Client Credentials option.
b. In the Application ID field, enter your Client/Application ID.
e. In the Tenant ID field, enter your Tenant ID .
f. In the Client Secret field, enter your Client Secret.
g. Click Test to validate the URLs, token, and connection
h. Save the instance.
Configure Azure Data Explorer in Cortex
| Parameter | Description | Required |
|---|---|---|
| Cluster URL (e.g. https://help.kusto.windows.net) | True | |
| Application ID | True | |
| Client Activity Prefix | A customized prefix of the client activity identifier for the query execution. For example, for a prefix value of ‘XSOAR-DataExplorer’, the client activity ID will be in the format of: ‘XSOAR-DataExplorer;<UUID>’. | True |
| Trust any certificate (not secure) | False | |
| Use system proxy settings | False | |
| Authentication Type | Type of authentication - could be Authorization Code Flow (recommended), Device Code Flow or Client Credentials Flow. | False |
| Tenant ID | For Authorization Code or Client Credentials Flows. | False |
| Client Secret | For Authorization Code or Client Credentials Flows. | False |
| Application redirect URI (for Authorization Code mode) | False | |
| Authorization code | for Authorization Code mode - received from the authorization step. see Detailed Instructions (?) section | False |
Commands
You can execute these commands from the CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.
azure-data-explorer-search-query-execute
Execute a Kusto Query Language (KQL) query against the given database inside a cluster. The Kusto query is a read-only request to process data and return results. To learn more about KQL go to https://docs.microsoft.com/en-us/azure/kusto/query/.
Base Command
azure-data-explorer-search-query-execute
Input
| Argument Name | Description | Required |
|---|---|---|
| query | Kusto Query Language (KQL) search query to execute on given database. | Required |
| database_name | The name of the database to execute the query on. | Required |
| timeout | The timeout for the execution of the search query on the server side. The timeout is a float number in minutes that ranges from 0 to 60. Default is 5. | Optional |
Context Output
| Path | Type | Description |
|---|---|---|
| AzureDataExplorer.SearchQueryResults.Query | String | The executed query on the given database. |
| AzureDataExplorer.SearchQueryResults.ClientActivityID | String | The Client Activity ID. A unique identifier of the executed query. |
| AzureDataExplorer.SearchQueryResults.PrimaryResults | Unknown | The results of the query execution. |
| AzureDataExplorer.SearchQueryResults.Database | String | The database against which the query will be executed. |
Command Example
!azure-data-explorer-search-query-execute database_name=Samples query="StormEvents| limit 1"
Context Example
{
"AzureDataExplorer": {
"SearchQueryResults": {
"ClientActivityID": "XSOAR-DataExplorer;759d43a9-cdc1-4882-8f8b-3e8d8a703f1e",
"Database": "Samples",
"PrimaryResults": [
{
"BeginLat": 28.0393,
"BeginLocation": "MELBOURNE BEACH",
"BeginLon": -80.6048,
"DamageCrops": 0,
"DamageProperty": 0,
"DeathsDirect": 0,
"DeathsIndirect": 0,
"EndLat": 28.0393,
"EndLocation": "MELBOURNE BEACH",
"EndLon": -80.6048,
"EndTime": "2007-09-29T08:11:00",
"EpisodeId": 11091,
"EpisodeNarrative": "Showers and thunderstorms lingering along the coast produced waterspouts in Brevard County.",
"EventId": 61032,
"EventNarrative": "A waterspout formed in the Atlantic southeast of Melbourne Beach and briefly moved toward shore.",
"EventType": "Waterspout",
"InjuriesDirect": 0,
"InjuriesIndirect": 0,
"Source": "Trained Spotter",
"StartTime": "2007-09-29T08:11:00",
"State": "ATLANTIC SOUTH",
"StormSummary": {
"Details": {
"Description": "A waterspout formed in the Atlantic southeast of Melbourne Beach and briefly moved toward shore.",
"Location": "ATLANTIC SOUTH"
},
"EndTime": "2007-09-29T08:11:00.0000000Z",
"StartTime": "2007-09-29T08:11:00.0000000Z",
"TotalDamages": 0
}
}
],
"Query": "StormEvents| limit 1"
}
}
}
Human Readable Output
Results of executing search query with client activity ID: XSOAR-DataExplorer;759d43a9-cdc1-4882-8f8b-3e8d8a703f1e
Begin Lat Begin Location Begin Lon Damage Crops Damage Property Deaths Direct Deaths Indirect End Lat End Location End Lon End Time Episode Id Episode Narrative Event Id Event Narrative Event Type Injuries Direct Injuries Indirect Source Start Time State Storm Summary 28.0393 MELBOURNE BEACH -80.6048 0 0 0 0 28.0393 MELBOURNE BEACH -80.6048 2007-09-29T08:11:00 11091 Showers and thunderstorms lingering along the coast produced waterspouts in Brevard County. 61032 A waterspout formed in the Atlantic southeast of Melbourne Beach and briefly moved toward shore. Waterspout 0 0 Trained Spotter 2007-09-29T08:11:00 ATLANTIC SOUTH TotalDamages: 0
StartTime: 2007-09-29T08:11:00.0000000Z
EndTime: 2007-09-29T08:11:00.0000000Z
Details: {“Description”: “A waterspout formed in the Atlantic southeast of Melbourne Beach and briefly moved toward shore.”, “Location”: “ATLANTIC SOUTH”}
azure-data-explorer-search-query-list
List search queries that have reached a final state in the given database. A database admin or database monitor can see any command that was invoked on their database. Other users can only see queries that they themselves invoked.
Base Command
azure-data-explorer-search-query-list
Input
| Argument Name | Description | Required |
|---|---|---|
| database_name | The name of the database from which to list the completed search queries. . | Required |
| client_activity_id | The client activity ID property of the search query. Use this value to get a specific search query. | Optional |
| limit | The maximum number of completed queries to return. Default is 50. | Optional |
| page | The page number from which to start a search. Default is 1. | Optional |
| page_size | The maximum number of completed queries to return per page. If this argument is not provided, an automatic pagination will be made according to the limit argument. | Optional |
Context Output
| Path | Type | Description |
|---|---|---|
| AzureDataExplorer.SearchQuery.ClientActivityId | String | The client activity ID. A unique identifier of the query execution. |
| AzureDataExplorer.SearchQuery.Text | String | The search query text. |
| AzureDataExplorer.SearchQuery.Database | String | The name of the database that the search query is run on. |
| AzureDataExplorer.SearchQuery.StartedOn | Date | The query execution start time in UTC. |
| AzureDataExplorer.SearchQuery.LastUpdatedOn | Date | The last update time of the query. |
| AzureDataExplorer.SearchQuery.Duration | Date | The search query runtime. |
| AzureDataExplorer.SearchQuery.State | String | The search query state. |
| AzureDataExplorer.SearchQuery.RootActivityId | String | The root activity ID. |
| AzureDataExplorer.SearchQuery.User | String | The user who performed the query. |
| AzureDataExplorer.SearchQuery.FailureReason | String | The reason for query failure. |
| AzureDataExplorer.SearchQuery.TotalCpu | String | The total CPU clock time (User mode + Kernel mode) consumed by this query. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Memory.Hits | Number | The number of cache hits. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Memory.Misses | Number | The number of cache misses. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Disk.Hits | Number | The number of disk hits. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Disk.Misses | Number | The number of disk misses. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.Hot.HitBytes | Number | The amount of data (in bytes) which was found in the hot data cache of the table’s extents, during the search query execution. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.Hot.MissBytes | Number | The amount of data (in bytes) which was not found in the hot data cache of the table’s extents, during the search query execution. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.Hot.RetrieveBytes | Number | The amount of data (in bytes) that was retrieved from hot data cache of the table’s extents, during the search query execution. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.Cold.HitBytes | Number | The amount of data (in bytes) which was found in the cold data cache of the table’s extents, during the search query execution. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.Cold.MissBytes | Number | The amount of data (in bytes) which was not found in the cold data cache of the table’s extents, during the search query execution. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.Cold.RetrieveBytes | Number | The amount of data (in bytes) that was retrieved from cold data cache during the search query execution. |
| AzureDataExplorer.SearchQuery.CacheStatistics.Shards.BypassBytes | Number | The amount of data (in bytes) that was bypassed (reloaded) in the cache of the table’s extents during the search query execution. |
| AzureDataExplorer.SearchQuery.Application | String | The application name that invoked the command. |
| AzureDataExplorer.SearchQuery.MemoryPeak | Number | The peak memory usage of the query execution. |
| AzureDataExplorer.SearchQuery.ScannedExtentsStatistics.MinDataScannedTime | Date | The minimum data scan time. |
| AzureDataExplorer.SearchQuery.ScannedExtentsStatistics.MaxDataScannedTime | Date | The maximum data scan time. |
| AzureDataExplorer.SearchQuery.ScannedExtentsStatistics.TotalExtentsCount | Number | The total number of extents which were used during the query execution. |
| AzureDataExplorer.SearchQuery.ScannedExtentsStatistics.ScannedExtentsCount | Number | The number of extents which were scanned during the query execution. |
| AzureDataExplorer.SearchQuery.ScannedExtentsStatistics.TotalRowsCount | Number | The total row count of extents which were used during the query execution. |
| AzureDataExplorer.SearchQuery.ScannedExtentsStatistics.ScannedRowsCount | Number | The number of scanned rows of an extent during query execution. |
| AzureDataExplorer.SearchQuery.Principal | String | The principal that invoked the query. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.SecurityTokenPresent | Boolean | Whether the security token is present in the request or not. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.AuthorizationScheme | String | The authorization scheme. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.RequestHostName | String | The hostname of the request. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.LocalClusterName | String | The cluster name. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.OriginClusterName | String | The origin cluster name. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.api_version | String | The API version. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.request_readonly | Boolean | Whether the request is read-only or not. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.servertimeout | Number | The server timeout value. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.servertimeoutorigin | String | The server timeout origin. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.query_datascope | Number | The query datascope. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.query_fanout_nodes_percent | Number | The percentage of the query nodes in the cluster to use per subquery distribution operation. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.query_fanout_threads_percent | Number | The percentage of CPUs the cluster will assign on each node. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.maxmemoryconsumptionperiterator | Number | The maximum amount of memory that a single query plan result set iterator can hold. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.max_memory_consumption_per_query_per_node | Number | The maximum amount of memory that can be used on a single node for a specific query. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.truncationmaxsize | Number | The maximum overall data size returned by the query, in bytes. |
| AzureDataExplorer.SearchQuery.ClientRequestProperties.Options.truncationmaxrecords | Number | The maximum number of records returned by the query. |
| AzureDataExplorer.SearchQuery.ResultSetStatistics.TableCount | Number | The number of tables that were retrieved following search query execution. |
| AzureDataExplorer.SearchQuery.ResultSetStatistics.TablesStatistics.RowCount | Number | The row count of the table retrieved following search query execution. |
| AzureDataExplorer.SearchQuery.ResultSetStatistics.TablesStatistics.TableSize | Number | The total size in bytes of the table retrieved following search query execution. |
| AzureDataExplorer.SearchQuery.WorkloadGroup | String | The workload group which the query was assigned to. The query is executed using the policies assigned to the workload group. There are two pre-defined workload groups (internal and default) and up to 10 custom workload groups which may be defined at the cluster level. |
Command Example
!azure-data-explorer-search-query-list database_name=Samples limit=1
Context Example
{
"AzureDataExplorer": {
"SearchQuery": {
"Application": "KusWeb",
"CacheStatistics": {
"Disk": {
"Hits": 0,
"Misses": 0
},
"Memory": {
"Hits": 0,
"Misses": 0
},
"Shards": {
"BypassBytes": 0,
"Cold": {
"HitBytes": 0,
"MissBytes": 0,
"RetrieveBytes": 0
},
"Hot": {
"HitBytes": 0,
"MissBytes": 0,
"RetrieveBytes": 0
}
}
},
"ClientActivityId": "KustoWebV2;f1be2c7e-f810-437b-a1f8-f8bbbedf238d",
"ClientRequestProperties": {
"AuthorizationScheme": "Bearer",
"LocalClusterName": "https://help.kusto.windows.net/",
"Options": {
"api_version": "v2",
"max_memory_consumption_per_query_per_node": 2000000000,
"maxmemoryconsumptionperiterator": 5368709120,
"query_datascope": 1,
"query_fanout_nodes_percent": 100,
"query_fanout_threads_percent": 100,
"query_language": "csl",
"queryconsistency": "strongconsistency",
"request_app_name": "KusWeb",
"request_readonly": true,
"request_readonly_hardline": false,
"servertimeout": 600000000,
"truncationmaxrecords": 500000,
"truncationmaxsize": 67108864
},
"OriginClusterName": "https://help.kusto.windows.net/",
"RequestHostName": "https://help.kusto.windows.net:443/",
"SecurityTokenPresent": true
},
"Database": "Samples",
"Duration": "0:00:00",
"FailureReason": "[none]",
"LastUpdatedOn": "2021-11-24T15:15:27",
"MemoryPeak": 0,
"Principal": "aaduser=xxx-xxxx-xxxx",
"ResultSetStatistics": {
"TableCount": 2,
"TablesStatistics": [
{
"RowCount": 0,
"TableSize": 0
},
{
"RowCount": 2,
"TableSize": 1244
}
]
},
"RootActivityId": "2b9e0ec8-f6b0-407e-90b6-68eba3777564",
"ScannedExtentsStatistics": {
"MaxDataScannedTime": null,
"MinDataScannedTime": null,
"ScannedExtentsCount": 0,
"ScannedRowsCount": 0,
"TotalExtentsCount": 0,
"TotalRowsCount": 0
},
"StartedOn": "2021-11-24T15:15:27",
"State": "Completed",
"Text": "set notruncation;\nCovid19",
"TotalCpu": "0:00:00",
"User": "dataExplorer@qmasterslabgmail.onmicrosoft.com",
"WorkloadGroup": "default"
}
}
}
Human Readable Output
List of Completed Search Queries
Showing 0 to 1 records out of 134.
Client Activity Id User Text Database Started On Last Updated On State KustoWebV2;f1be2c7e-f810-437b-a1f8-f8bbbedf238d dataExplorer@qmasterslabgmail.onmicrosoft.com set notruncation;
Covid19Samples 2021-11-24T15:15:27 2021-11-24T15:15:27 Completed
azure-data-explorer-running-search-query-list
List currently executing search queries in the given database. A database admin or database monitor can see any search query that was invoked on their database.
Other users can only see search queries that they themselves invoked.
Base Command
azure-data-explorer-running-search-query-list
Input
| Argument Name | Description | Required |
|---|---|---|
| database_name | The database name. | Required |
| client_activity_id | The client activity ID property of the search query. Use this to get a specific running search query. | Optional |
| limit | The maximum number of running queries to return. Default is 50. | Optional |
| page | The page number from which to start a search. Default is 1. | Optional |
| page_size | The maximum number of running queries to return per page. If this argument is not provided, an automatic pagination will be made according to the limit argument. | Optional |
Context Output
| Path | Type | Description |
|---|---|---|
| AzureDataExplorer.RunningSearchQuery.ClientActivityId | String | The client activity ID. A unique identifier of the query execution. |
| AzureDataExplorer.RunningSearchQuery.Text | String | The search query text. |
| AzureDataExplorer.RunningSearchQuery.Database | String | The name of the database that the search query is run on. |
| AzureDataExplorer.RunningSearchQuery.StartedOn | Date | The query execution start time in UTC. |
| AzureDataExplorer.RunningSearchQuery.LastUpdatedOn | String | The last update time of the query. |
| AzureDataExplorer.RunningSearchQuery.Duration | Date | The search query runtime duration. |
| AzureDataExplorer.RunningSearchQuery.State | String | The search query state. |
| AzureDataExplorer.RunningSearchQuery.RootActivityId | String | The root activity ID. |
| AzureDataExplorer.RunningSearchQuery.User | String | The user who performed the query. |
| AzureDataExplorer.RunningSearchQuery.FailureReason | String | The reason for query failure. |
| AzureDataExplorer.RunningSearchQuery.TotalCpu | String | The total CPU clock time (User mode + Kernel mode) consumed by this query. |
| AzureDataExplorer.RunningSearchQuery.CacheStatistics | Unknown | The cache statistics. |
| AzureDataExplorer.RunningSearchQuery.Application | String | The application name that invoked the command. |
| AzureDataExplorer.RunningSearchQuery.MemoryPeak | Number | The peak memory usage of the running query execution. |
| AzureDataExplorer.RunningSearchQuery.ScannedExtentsStatistics | Unknown | The scanned extent count. |
| AzureDataExplorer.RunningSearchQuery.Principal | String | The principal that invoked the query. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.SecurityTokenPresent | Boolean | Whether the security token is present in the request or not. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.AuthorizationScheme | String | The authorization scheme. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.RequestHostName | String | The hostname of the request. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.LocalClusterName | String | The cluster name. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.OriginClusterName | String | The origin cluster name. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.api_version | String | The API version. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.request_readonly | Boolean | Whether the request is read-only or not. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.servertimeout | Number | The server timeout value. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.servertimeoutorigin | String | The server timeout origin. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.query_datascope | Number | The query datascope. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.query_fanout_nodes_percent | Number | The percentage of the query nodes in the cluster to use per subquery distribution operation. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.query_fanout_threads_percent | Number | The percentage of CPUs the cluster will assign on each node. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.maxmemoryconsumptionperiterator | Number | The maximum amount of memory that a single query plan result set iterator can hold. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.max_memory_consumption_per_query_per_node | Number | The maximum amount of memory that can be used on a single node for a specific query. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.truncationmaxsize | Number | The maximum overall data size returned by the query, in bytes. |
| AzureDataExplorer.RunningSearchQuery.ClientRequestProperties.Options.truncationmaxrecords | Number | The maximum number of records returned by the query. |
| AzureDataExplorer.RunningSearchQuery.ResultSetStatistics | Unknown | The result set statistics. |
| AzureDataExplorer.RunningSearchQuery.WorkloadGroup | String | The workload group. |
Command Example
!azure-data-explorer-running-search-query-list database_name=Samples limit=1
Context Example
{
"AzureDataExplorer": {
"RunningSearchQuery": {
"Application": "KusWeb",
"CacheStatistics": "null",
"ClientActivityId": "KustoWebV2;c6ff3e99-d2cb-4a3e-ab05-955ae383a7c6",
"ClientRequestProperties": "{\"SecurityTokenPresent\":true,\"AuthorizationScheme\":\"Bearer\",\"RequestHostName\":\"https://help.kusto.windows.net:443/\",\"LocalClusterName\":\"https://help.kusto.windows.net/\",\"OriginClusterName\":\"https://help.kusto.windows.net/\",\"Options\":{\"servertimeout\":600000000,\"queryconsistency\":\"strongconsistency\",\"query_language\":\"csl\",\"request_readonly\":true,\"request_readonly_hardline\":false,\"api_version\":\"v2\",\"request_app_name\":\"KusWeb\",\"query_datascope\":1,\"query_fanout_nodes_percent\":100,\"query_fanout_threads_percent\":100,\"maxmemoryconsumptionperiterator\":5368709120,\"max_memory_consumption_per_query_per_node\":2000000000,\"truncationmaxsize\":67108864,\"truncationmaxrecords\":500000}}",
"Database": "Samples",
"Duration": "0:00:00",
"FailureReason": "",
"LastUpdatedOn": "2021-11-24T15:16:34",
"MemoryPeak": 0,
"Principal": "aaduser=xxx-xxxx-xxxx",
"ResultSetStatistics": "null",
"RootActivityId": "c8233607-30a9-4cc0-9c54-ec716e5fc246",
"ScannedExtentsStatistics": "null",
"StartedOn": "2021-11-24T15:16:34",
"State": "InProgress",
"Text": "set notruncation;\nCovid19_Bing",
"TotalCpu": "0:00:00",
"User": "dataExplorer@qmasterslabgmail.onmicrosoft.com",
"WorkloadGroup": "default"
}
}
}
Human Readable Output
List of Currently running Search Queries
Showing 0 to 1 records out of 2.
Client Activity Id User Text Database Started On Last Updated On State KustoWebV2;c6ff3e99-d2cb-4a3e-ab05-955ae383a7c6 dataExplorer@qmasterslabgmail.onmicrosoft.com set notruncation;
Covid19_BingSamples 2021-11-24T15:16:34 2021-11-24T15:16:34 InProgress
azure-data-explorer-running-search-query-cancel
Starts a best-effort attempt to cancel a specific running search query in the specified database.
Base Command
azure-data-explorer-running-search-query-cancel
Input
| Argument Name | Description | Required |
|---|---|---|
| client_activity_id | The client activity ID of the query to delete. | Required |
| database_name | The database name. | Required |
| reason | The reason for canceling the running query. . | Optional |
Context Output
| Path | Type | Description |
|---|---|---|
| AzureDataExplorer.CanceledSearchQuery.RunningQueryCanceled | Boolean | Whether the query was successfully canceled or not. |
| AzureDataExplorer.CanceledSearchQuery.ClientRequestId | String | The client activity ID of the cancelled query. |
| AzureDataExplorer.CanceledSearchQuery.ReasonPhrase | String | The reason for canceling the running query. |
Command Example
!azure-data-explorer-running-search-query-cancel database_name=Samples client_activity_id=xxxx-xxxxx-xxxxx
Context Example
{
"AzureDataExplorer": {
"CanceledSearchQuery": {
"ClientRequestId": "xxxx-xxxxx-xxxxx",
"ReasonPhrase": "None",
"RunningQueryCanceled": false
}
}
}
Human Readable Output
Canceled Search Query xxxx-xxxxx-xxxxx
Client Request Id Reason Phrase Running Query Canceled xxxx-xxxxx-xxxxx None false
azure-data-explorer-auth-start
Run this command to start the authorization process and follow the instructions in the command results.
Base Command
azure-data-explorer-auth-start
Input
There are no input arguments for this command.
Context Output
There is no context output for this command.
azure-data-explorer-auth-complete
Run this command to complete the authorization process. This should be used after running the azure-data-explorer-auth-start command.
Base Command
azure-data-explorer-auth-complete
Input
There are no input arguments for this command.
Context Output
There is no context output for this command.
azure-data-explorer-auth-reset
Run this command if for some reason you need to rerun the authentication process.
Base Command
azure-data-explorer-auth-reset
Input
There are no input arguments for this command.
Context Output
There is no context output for this command.
azure-data-explorer-auth-test
Run this command to test the connectivity to Azure Data Explorer.
Base Command
azure-data-explorer-auth-test
Input
There are no input arguments for this command.
Context Output
There is no context output for this command.
azure-data-explorer-generate-login-url
Generate the login url used for Authorization code flow.
Base Command
azure-data-explorer-generate-login-url
Input
There are no input arguments for this command.
Context Output
There is no context output for this command.
Command Example
azure-data-explorer-generate-login-url
Human Readable Output
Authorization instructions
- Click on the login URL to sign in and grant Cortex XSOAR permissions for your Azure Service Management.
You will be automatically redirected to a link with the following structure:
REDIRECT_URI?code=AUTH_CODE&session_state=SESSION_STATE- Copy the
AUTH_CODE(without thecode=prefix, and thesession_stateparameter)
and paste it in your instance configuration under the Authorization code parameter.
Configuration parameters
cluster_url— Cluster URL (e.g. https://help.kusto.windows.net) (required)client_id— Application ID (required)client_activity_prefix— Client Activity Prefix (required)insecure— Trust any certificate (not secure)proxy— Use system proxy settingsauthentication_type— Authentication Type (required)tenant_id— Tenant IDcredentials— Client Secretredirect_uri— Application redirect URIauth_code— Authorization code
Commands (9)
-
azure-data-explorer-auth-completeRun this command to complete the authorization process. This should be used after running the azure-data-explorer-auth-start command.
-
azure-data-explorer-auth-resetRun this command if for some reason you need to rerun the authentication process.
-
azure-data-explorer-auth-startRun this command to start the authorization process and follow the instructions in the command results.
-
azure-data-explorer-auth-testRun this command to test the connectivity to Azure Data Explorer.
-
azure-data-explorer-generate-login-urlGenerate the login url used for Authorization code flow.
-
azure-data-explorer-running-search-query-cancelStarts a best-effort attempt to cancel a specific running search query in the specified database.
-
azure-data-explorer-running-search-query-listList currently executing search queries in the given database. A database admin or database monitor can see any search query that was invoked on their database. Other users can only see search queries that they themselves invoked.
-
azure-data-explorer-search-query-executeExecute a Kusto Query Language (KQL) query against the given database inside a cluster. The Kusto query is a read-only request to process data and return results. To learn more about KQL go to https://docs.microsoft.com/en-us/azure/kusto/query/.
-
azure-data-explorer-search-query-listList search queries that have reached a final state in the given database. A database admin or database monitor can see any command that was invoked on their database. Other users can only see queries that they themselves invoked.
import json import pytest """MOCK PARAMETERS """ CLUSTER_URL = "https://help.kusto.windows.net" APPLICATION_ID = "xxx-xxx-xxx" QUERY_URL_SUFFIX = "/v1/rest/query" MANAGEMENT_URL_SUFFIX = "/v1/rest/mgmt" CLIENT_ACTIVITY_PREFIX = "XSOAR-DataExplorerIntegation" CLIENT_ACTIVITY_ID = "XSOAR-DataExplorer1;xxxx-xxxxx-xxxxx-xxxx" DATABASE_NAME = "Samples" QUERY = "test_query" def load_mock_response(file_name: str) -> str: """ Load mock file that simulates an API response. Args: file_name (str): Name of the mock response JSON file to return. Returns: str: Mock file content. """ with open(f"test_data/{file_name}", encoding="utf-8") as mock_file: return mock_file.read() def mock_client(auth_type: str = "Device Code"): from AzureDataExplorer import DataExplorerClient return DataExplorerClient(CLUSTER_URL, APPLICATION_ID, CLIENT_ACTIVITY_PREFIX, False, False, auth_type) def test_execute_search_query_command(requests_mock): """ Scenario: execute search query against given database. Given: - User has provided valid credentials. - Database name provided. - KQL query provided. When: - azure-data-explorer-execute-search-query command called. Then: - Ensure number of items is correct. - Ensure outputs prefix is correct. - Validate outputs' fields. """ from AzureDataExplorer import search_query_execute_command mock_response = json.loads(load_mock_response("execute_query.json")) url = f"{CLUSTER_URL}{QUERY_URL_SUFFIX}" requests_mock.post("https://login.microsoftonline.com/organizations/oauth2/v2.0/token", json={}) requests_mock.post(url, json=mock_response) result = search_query_execute_command(mock_client(), {"database_name": DATABASE_NAME, "query": "StormEvents | take 3"}) outputs = result.outputs assert len(outputs) == 4 assert outputs["Database"] == DATABASE_NAME assert outputs["Query"] == "StormEvents | take 3" assert outputs["PrimaryResults"][0]["StartTime"] == "2007-09-29T08:11:00" assert outputs["PrimaryResults"][0]["EndLocation"] == "MELBOURNE BEACH" assert outputs["PrimaryResults"][0]["EndLat"] == 28.0393 assert result.outputs_prefix == "AzureDataExplorer.SearchQueryResults" @pytest.mark.parametrize( "test_input,outputs_size,expected_activity,query", [ ({"database_name": DATABASE_NAME}, 2, "KPC.execute;43b2bbf0-1d81-4c6d-9312-3fb93ae84d50", "StormEvents | take 10"), ( {"database_name": DATABASE_NAME, "page": 2, "page_size": 1}, 1, "KPC.execute;463b6d97-09ef-4d06-99f8-b4ee21fc6620", "StormEvents | take 20", ), ], ) def test_list_search_queries_command(test_input, outputs_size, expected_activity, query, requests_mock): """ Scenario: execute search query against given database. Given: - User has provided valid credentials. - Database name provided. When: - azure-data-explorer-list-search-queries command called. Then: - Ensure number of items is correct. - Ensure outputs prefix is correct. - Validate outputs' fields. """ from AzureDataExplorer import search_queries_list_command mock_response = json.loads(load_mock_response("list_completed_queries.json")) url = f"{CLUSTER_URL}{MANAGEMENT_URL_SUFFIX}" requests_mock.post(url, json=mock_response) requests_mock.post("https://login.microsoftonline.com/organizations/oauth2/v2.0/token", json={}) result = search_queries_list_command(mock_client(), test_input) outputs = result.outputs assert len(outputs) == outputs_size assert outputs[0]["ClientActivityId"] == expected_activity assert outputs[0]["Text"] == query assert result.outputs_prefix == "AzureDataExplorer.SearchQuery" def test_list_search_running_queries_command(requests_mock): """ Scenario: execute search query against given database. Given: - User has provided valid credentials. - Database name provided. When: - azure-data-explorer-list-running-search-query command called. Then: - Ensure number of items is correct. - Ensure outputs prefix is correct. - Validate outputs' fields. """ from AzureDataExplorer import running_search_queries_list_command mock_response = json.loads(load_mock_response("list_running_queries.json")) url = f"{CLUSTER_URL}{MANAGEMENT_URL_SUFFIX}" requests_mock.post(url, json=mock_response) requests_mock.post("https://login.microsoftonline.com/organizations/oauth2/v2.0/token", json={}) result = running_search_queries_list_command(mock_client(), {"database_name": DATABASE_NAME, "page": 1, "limit": 1}) outputs = result.outputs assert len(outputs) == 1 assert outputs[0]["Text"] == "set notruncation;\r\nCovid19_Bing |limit 1000000000000" assert outputs[0]["ClientActivityId"] == "KustoWebV2;a9f21b87-bfab-4cec-953f-3f3ba9a5dded" assert result.outputs_prefix == "AzureDataExplorer.RunningSearchQuery" @pytest.mark.parametrize( "response_mock_file,test_input,expected_reason", [ ("cancel_query.json", {"database_name": DATABASE_NAME, "client_activity_id": CLIENT_ACTIVITY_ID}, "None"), ( "cancel_query_reason.json", { "database_name": DATABASE_NAME, "client_activity_id": CLIENT_ACTIVITY_ID, "reason": "Query cancelled by the user's request", }, "Query cancelled by the user's request", ), ], ) def test_cancel_running_search_query_command(response_mock_file, test_input, expected_reason, requests_mock): """ Scenario: execute search query against given database. Given: - User has provided valid credentials. - Database name provided. When: - azure-data-explorer-cancel-running-search-query command called. Then: - Ensure number of items is correct. - Ensure outputs prefix is correct. """ from AzureDataExplorer import running_search_query_cancel_command mock_response = json.loads(load_mock_response(response_mock_file)) url = f"{CLUSTER_URL}{MANAGEMENT_URL_SUFFIX}" requests_mock.post(url, json=mock_response) requests_mock.post("https://login.microsoftonline.com/organizations/oauth2/v2.0/token", json={}) cancel_query_result = running_search_query_cancel_command(mock_client(), test_input) outputs = cancel_query_result.outputs assert len(outputs) == 1 assert outputs[0]["ReasonPhrase"] == expected_reason assert cancel_query_result.outputs_prefix == "AzureDataExplorer.CanceledSearchQuery" def test_retrieve_common_request_body(): """ Scenario: Retrieve the body argument for requests call. Given: - Database name provided. - A response returned from the API. When: - Before every request method in AzureDataExplorer client. Then: - Ensure that the retrieved body is as expected. """ from AzureDataExplorer import retrieve_common_request_body body = retrieve_common_request_body(DATABASE_NAME, QUERY) assert body["db"] == DATABASE_NAME assert body["csl"] == QUERY def test_convert_kusto_response_to_dict(): """ Scenario: Convert kusto Response Dataset to dict object. Given: - User has provided valid credentials. - Database name provided. - A response returned from the API. When: - Following a successful API call to Azure Data Explorer API. Then: - Ensure number of items is correct. - Ensure that time attributes in str format """ from azure.kusto.data.response import KustoResponseDataSetV1 from AzureDataExplorer import convert_kusto_response_to_dict mock_response = json.loads(load_mock_response("execute_query.json")) kusto_format_response = KustoResponseDataSetV1(mock_response) dict_kusto = convert_kusto_response_to_dict(kusto_format_response, page=1, limit=1) assert len(dict_kusto) == 1 assert type(dict_kusto[0]["StartTime"]) is str assert type(dict_kusto[0]["EndTime"]) is str def test_format_header_for_list_commands(): """ Scenario: Format the header of readable output in list commands. Given: - Base command header. - Number retrieved results. - Number of total pages. - The client entered page number. - The client entered limit number. When: - azure-data-explorer-running-search-query-list command called. - azure-data-explorer-search-query-list command called. Then: - Ensure the header is in the right format. """ from AzureDataExplorer import format_header_for_list_commands readable_output_header = format_header_for_list_commands("List of Completed Search Queries", 1, 1, 1, 1) assert ( readable_output_header == "List of Completed Search Queries \nShowing" " page 1 out of 1 total pages. Current page size: 1." ) def test_calculate_total_request_timeout(): """ Scenario: Calculates total request timeout in search query execution API call. Given: - Client's server timeout argument. When: - azure-data-explorer-search-query-execution command called. Then: - Ensure total request timeout calculated correctly. """ from AzureDataExplorer import calculate_total_request_timeout total_request_timeout = calculate_total_request_timeout(5) assert total_request_timeout == 320 def test_validate_list_command_arguments(): """ Scenario: Validation list commands optional arguments. Given: - Number retrieved results. - Limit number. When: - azure-data-explorer-running-search-query-list command called. - azure-data-explorer-search-query-list command called. Then: - Ensure that exception is raised when page number and limit number are invalid. """ from AzureDataExplorer import validate_list_command_arguments try: validate_list_command_arguments(1, 0, 1) except ValueError as v_error: assert str(v_error) == "Page and limit arguments must be integers greater than 0." @pytest.mark.parametrize( "auth_type, expected_results", [ ("Device Code", "Please enable the integration and run `!azure-data-explorer-auth-start`"), ("Authorization Code", "When using user auth flow configuration, "), ], ) def test_test_module_command(mocker, auth_type, expected_results): """ Given: - Case 1: Integration params with 'Device' as auth_type. - Case 2: Integration params with 'User Auth' as auth_type. When: - Calling test-module command. Then - Assert the right exception was thrown. - Case 1: Should throw an exception related to Device-code-flow config and return True. - Case 2: Should throw an exception related to User-Auth-flow config and return True. """ from AzureDataExplorer import test_module client = mock_client(auth_type) with pytest.raises(Exception) as e: test_module(client) assert expected_results in str(e.value) def test_generate_login_url(mocker): """ Given: - Self-deployed are true and auth code are the auth flow When: - Calling function azure-data-explorer-generate-login-url Then: - Ensure the generated url are as expected. """ # prepare import AzureDataExplorer import demistomock as demisto from AzureDataExplorer import main redirect_uri = "redirect_uri" tenant_id = "tenant_id" client_id = "client_id" cluster_url = "https://help.kusto.windows.net" mocked_params = { "redirect_uri": redirect_uri, "cluster_url": cluster_url, "self_deployed": "True", "tenant_id": tenant_id, "client_id": client_id, "authentication_type": "Authorization Code", "credentials": {"identifier": client_id, "password": "client_secret"}, } mocker.patch.object(demisto, "params", return_value=mocked_params) mocker.patch.object(demisto, "command", return_value="azure-data-explorer-generate-login-url") mocker.patch.object(AzureDataExplorer, "return_results") # call main() # assert expected_url = ( f"[login URL](https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize?" "response_type=code" f"&scope=offline_access%20{cluster_url}/.default" f"&client_id={client_id}&redirect_uri={redirect_uri})" ) res = AzureDataExplorer.return_results.call_args[0][0].readable_output assert expected_url in res