APIMetricsValidation
This integration runs through all scenarios as defined in the API Metrics expected results guide.
Utilities · Developer Tools
Details
| ID | APIMetricsValidation |
|---|---|
| Provider | APImetrics |
| Category | Utilities |
| From Version | 6.8.0 |
| Docker Image | demisto/python3:3.12.13.10116658 |
| Supported Modules | Agentix XSIAM |
README
test-scenario-one
Test Scenario One - All 10 Pass on first try
Base Command
test-scenario-one
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-two
Test Scenario Two - 5 Pass on first try, 5 error on quota and are scheduled
Base Command
test-scenario-two
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-three
Test Scenario Three - 5 Fail on second try
Base Command
test-scenario-three
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-four
Test Scenario Four - 2 Succeed, 3 Fail on retry
Base Command
test-scenario-four
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-five
Test Scenario Five - 10 IPs as a string, 1 success
Base Command
test-scenario-five
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-six
Test Scenario Six - 10 items are in 1 call. 1 fail.
Base Command
test-scenario-six
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-seven
Test Scenario Seven - 10 items are in 1 call. 1 fail. No metrics are returned.
Base Command
test-scenario-seven
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-eight
Test Scenario Eight - 10 items are in 1 call. 1 fail. No metrics returned. Timeout.
Base Command
test-scenario-eight
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-nine
This is the test PB scenario
Base Command
test-scenario-nine
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
test-scenario-ten
This command is executed five times in the PB
Base Command
test-scenario-ten
Input
| Argument Name | Description | Required |
| — | — | — |
Context Output
There is no context output for this command.
Configuration parameters
testing— This is not needed (required)
Commands (10)
-
test-scenario-eightTest Scenario Eight - 10 items are in 1 call. 1 fail. No metrics returned. Timeout.
-
test-scenario-fiveTest Scenario Five - 10 IPs as a string, 1 success
-
test-scenario-fourTest Scenario Four - 2 Succeed, 3 Fail on retry
-
test-scenario-nineThis is the test PB scenario
-
test-scenario-oneTest Scenario One - All 10 Pass on first try
-
test-scenario-sevenTest Scenario Seven - 10 items are in 1 call. 1 fail. No metrics are returned.
-
test-scenario-sixTest Scenario Six - 10 items are in 1 call. 1 fail.
-
test-scenario-tenThis command is executed five times in the PB
-
test-scenario-threeTest Scenario Three - 5 Fail on second try
-
test-scenario-twoTest Scenario Two - 5 Pass on first try, 5 error on quota and are scheduled
import demistomock as demisto # noqa: F401 from CommonServerPython import * # noqa: F401 TEST_BANK_SCENARIO_TEN_ITEMS = """oneone,twotwo,threethree, fourfour, fivefive, sixsix, sevenseven, eighteight, ninenine, tenten""" TEST_BANK_SCENARIO_FIVE_ITEMS = """sixsix, sevenseven, eighteight, ninenine, tenten""" def scenario_one(): """ This is a test scenario where all 10 entries succeed Assumptions: Command has not been run before. Returns: 10 successful entries 1 ExecutionMetrics object containing 10 success """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_TEN_ITEMS) for url in urls: # Every url will pass here execution_metrics.success += 1 command_results.append(CommandResults(readable_output=f"Item - {url} has been processed")) command_results.append(execution_metrics.metrics) return command_results def scenario_two(): """ This is a test scenario where 5 entries are processed and 5 are failed with QuotaError Assumptions: Command has not been run before. Returns: 5 successful entries 5 Scheduled entries 1 ExecutionMetrics object containing 5 success and 5 quota errors """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_TEN_ITEMS) items_to_schedule = [] for idx, url in enumerate(urls): # First 5 succeed if idx < 5: execution_metrics.success += 1 command_results.append(CommandResults(readable_output=f"Item - {url} has been processed")) # Next 5 fail with QuotaError else: execution_metrics.quota_error += 1 items_to_schedule.append(url) scheduled_command = ScheduledCommand( command="test-scenario-two", next_run_in_seconds=20, args={"polling": True, "items_to_schedule": items_to_schedule}, timeout_in_seconds=40, items_remaining=len(items_to_schedule), ) command_results.append(CommandResults(scheduled_command=scheduled_command)) if execution_metrics.metrics is not None: command_results.append(execution_metrics.metrics) return command_results def scenario_three(): """ This is a test scenario where 5 entries failed with QuotaError Assumptions: Command has been run once before. Returns: 5 Scheduled entries No ExecutionMetrics object """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_FIVE_ITEMS) items_to_schedule = [] for _idx, url in enumerate(urls): items_to_schedule.append(url) # All 5 items fail on retry here, and it's expected that all 5 iterations are previously scheduled continue scheduled_command = ScheduledCommand( command="test-scenario-three", next_run_in_seconds=20, args={"polling": True, "items_to_schedule": items_to_schedule}, timeout_in_seconds=40, items_remaining=len(items_to_schedule), ) command_results.append(CommandResults(scheduled_command=scheduled_command)) if execution_metrics.metrics is not None: command_results.append(execution_metrics.metrics) return command_results def scenario_four(): """ This is a test scenario where 5 entries failed with QuotaError Assumptions: Command has been run once before. Returns: 5 Scheduled entries 1 ExecutionMetrics object containing 2 successful """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_FIVE_ITEMS) items_to_schedule = [] if not demisto.args().get("polling"): for idx, url in enumerate(urls): if idx < 2: execution_metrics.success += 1 command_results.append(CommandResults(readable_output=f"Item - {url} has been processed")) else: items_to_schedule.append(url) continue scheduled_command = ScheduledCommand( command="test-scenario-four", next_run_in_seconds=20, args={"polling": True, "items_to_schedule": items_to_schedule}, timeout_in_seconds=40, items_remaining=len(items_to_schedule), ) command_results.append(CommandResults(scheduled_command=scheduled_command)) if execution_metrics.metrics is not None: command_results.append(execution_metrics.metrics) return command_results def scenario_five(): """ This is a test scenario where all 10 entries succeed Assumptions: Command has not been run before. Returns: 1 successful entry 1 ExecutionMetrics object containing 10 success """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_TEN_ITEMS) execution_metrics.success += 1 command_results.append(CommandResults(readable_output=f"Item - {urls} have been processed")) command_results.append(execution_metrics.metrics) return command_results def scenario_six(): """ This is a test scenario where 10 items are in a single api call. That one api call fails and is scheduled. Assumptions: Command has not been run before. Returns: 1 Scheduled entry """ execution_metrics = ExecutionMetrics() command_results: list = [] execution_metrics.quota_error += 1 scheduled_command = ScheduledCommand( command="test-scenario-six", next_run_in_seconds=20, args={"polling": True, "items_to_schedule": TEST_BANK_SCENARIO_TEN_ITEMS}, timeout_in_seconds=40, items_remaining=len(TEST_BANK_SCENARIO_TEN_ITEMS), ) command_results.append(CommandResults(scheduled_command=scheduled_command)) if execution_metrics.metrics is not None: command_results.append(execution_metrics.metrics) return command_results def scenario_seven(): """ This is a test scenario where 10 items are in a single api call. That one api call fails and is scheduled. Assumptions: Command has been run once before. Returns: 1 Scheduled entries No ExecutionMetrics object """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_TEN_ITEMS) items_to_schedule = [urls] scheduled_command = ScheduledCommand( command="test-scenario-seven", next_run_in_seconds=20, args={"polling": True, "items_to_schedule": items_to_schedule}, timeout_in_seconds=40, items_remaining=len(items_to_schedule), ) command_results.append(CommandResults(scheduled_command=scheduled_command)) if execution_metrics.metrics is not None: command_results.append(execution_metrics.metrics) return command_results def scenario_eight(): """ This is a test scenario where 10 items are in a single api call. That one api call fails and is scheduled. Assumptions: Command has been run once before. Metrics were reported the first run. Returns: 1 Scheduled entries No ExecutionMetrics object Notes: To simulate that this command has been re-ran before, we are not returning _any_ metrics. """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_TEN_ITEMS) items_to_schedule = [urls] scheduled_command = ScheduledCommand( command="test-scenario-seven", next_run_in_seconds=20, args={"polling": True, "items_to_schedule": items_to_schedule}, timeout_in_seconds=40, items_remaining=1, ) command_results.append(CommandResults(scheduled_command=scheduled_command)) if execution_metrics.metrics is not None: command_results.append(execution_metrics.metrics) return command_results def scenario_nine(): """ This is a test scenario which is ran in a playbook Assumptions: Command has not been run before. Returns: 10 successful entries 1 ExecutionMetrics object containing 10 success """ execution_metrics = ExecutionMetrics() command_results: list = [] urls = argToList(TEST_BANK_SCENARIO_FIVE_ITEMS) for url in urls: # Every url will pass here execution_metrics.quota_error += 1 command_results.append(CommandResults(readable_output=f"Item - {url} has been processed")) command_results.append(execution_metrics.metrics) return command_results def scenario_ten(): """ This is a test scenario which is returns only one successful entry. Assumptions: Ran only in a playbook Returns: 1 successful entry 1 ExecutionMetrics object containing 1 success """ execution_metrics = ExecutionMetrics() command_results: list = [] execution_metrics.success += 1 command_results.append(CommandResults(readable_output="Item has been processed")) command_results.append(execution_metrics.metrics) return command_results def main(): command = demisto.command() demisto.debug(f"Command being called is {command}") if demisto.args().get("polling") and command != "test-scenario-four": # This is to isolate a scenario where the command is run in polling mode sys.exit(0) try: if command == "test-scenario-one": results = scenario_one() return_results(results) elif command == "test-scenario-two": results = scenario_two() return_results(results) elif command == "test-scenario-three": results = scenario_three() return_results(results) elif command == "test-scenario-four": results = scenario_four() return_results(results) elif command == "test-scenario-five": results = scenario_five() return_results(results) elif command == "test-scenario-six": results = scenario_six() return_results(results) elif command == "test-scenario-seven": results = scenario_seven() return_results(results) elif command == "test-scenario-eight": results = scenario_eight() return_results(results) elif command == "test-scenario-nine": results = scenario_nine() return_results(results) return_error("This is an error") elif command == "test-scenario-ten": results = scenario_ten() return_results(results) elif command == "test-module": demisto.results("ok") else: raise NotImplementedError(f"{command} command is not implemented.") # Log exceptions and return errors except Exception as e: return_error(f"Failed to execute {command} command.\nError:\n{e!s}") """ ENTRY POINT """ if __name__ in ("__main__", "__builtin__", "builtins"): main()