Content Testing (v2.1.12)

Overview

This content pack provides tools for creating automated tests and test cases for XSOAR content and testing within XSOAR. Assessment tools identify potential impacts of an upgrade to a Marketplace content pack, display an XSOAR object’s version history when using Save Version, and list automations used in playbooks. Playbook analysis provides average and maximum task duration across a range of incidents as well as task runnging status. Test data is created by pulling incident fields and context from existing incidents, manual updates of individual fields and context keys, and saving data to XSOAR lists for use in testing. Test cases are defined that load appropriate test data and execute automation, playbook, and sub-playbook tests. The UnitTest incident type, UnitTestingLayout layout, and UnitTestingTopLevel playbook provide an example of a content testing environment and use of the testing tools. These objects can be modified as needed for specific testing requirements or guide implementation of a custom content testing environment.

• Provide guidance for what tests are required for a change
  • Marketplace content pack updates
  • Custom content change and release to production
• Automated unit and regression testing of XSOAR content
  • automations
  • commands
  • playbooks
  • sub-playbooks
• Configure incident fields and context for testing
• Save incident fields and context for regression testing
• Define and save test cases
• Adhoc testing of content
• Tests are dymanically added to an empty testing playbook
  • Testing workplan and war room document testing performed
  • Testing results displayed in the incident layout

Getting Started

The quickest way to get started is to create a new incident of the type UnitTesting. From there, execute the assessment tools, review the results in each tab of the incident layout, and from the Content Testing tab, exercise the content testing tools with the mock playbooks and lists.

XSOAR prerequesits are:

• XSOAR API Key
• Enabled REST API integration instance
• Enabled Demisto Lock integration instance

Prior to content testing using the supplied examples, five XSOAR lists must be created:

• UnitTestingAutomations
• UnitTestingSubplaybooks
• UnitTestCase
• TestFields
• TestContext

Below is example content for each of the three lists.

UnitTestingAutomations:

ip,{"ip":"8.8.8.8", "using":"VirusTotal (API v3)"}
http,{"method":"GET", "url":"https://www.google.com"}

UnitTestingSubplaybooks:

MockSubplaybook,{"input1":"value1", "input2":"value2"}

For the example test case, use the content testing tools to create and save XSOAR lists with incident fields and context. Two XSOAR lists to create are:

• TestFields
• TestContext

The test case XSOAR list to create for the example:

UnitTestCase:

LoadFields|TestFields
LoadContext|TestContext
Automation|UnitTestingAutomations
Subplaybook|UnitTestingSubplaybooks
Playbook|MockPlaybook

Test Environment Creation

• Determine test cases required based on upgraded content packs or updated custom content
• Create a new incident of the UnitTesting incident type
• From the Content Assessment tab, use the Assessment Actions
  • Review the Content Pack Details tab
  • Review the Commit History tab for history of XSOAR content local changes (when using dev to prod)
  • Review the Playbook Automations tab
• Update UnitTestTopLevel playbook or create a testing playbook with section headers where test tasks are assigned
• In the UnitTesting incident layout and the Content Testing tab
  • In the Execute Tests section add buttons or multi-select fields and configure the script arguments
    • For buttons, use the UnitTest automation
    • For multi-select fields with an array of playbooks, use the UnitTestMultiSelect automation
    • For button automation, the arguments are (see the automation sections for details on each automation):
      • addAfter - task ID in the playbook to add the test task after, usually a section header
      • playbook - specific playbook to launch or allow user to input a comma separated list. Blank if testType is Subplaybook
      • testType - type of test being executed
        • Playbook
        • Subplaybook
        • Multiselect
        • Automation
      • listName - if testType is Subplaybook or Automation, the name of the list to read input from
    • For multi-select of playbooks to test, use the UnitTestMultiSelect script
    • For a multi-select field parameters
      • Edit the unittestplaybooks field and set the values to playbook names to test
      • See caveat below
• Load the incident fields from an existing incident
• If needed, load the context from an existing incident
• Update individual fields and context keys as needed for specific testing
• For regression testing and test cases, save incident fields and context to XSOAR lists

Test Execution

• Create a new incident of the UnitTesting incident type (or a custom type)
• On the Content Testing tab in the Execute Test section
  • For each test, load incident fields and context and as needed
  • Execute the content tests using buttons and fields
  • Review the Test Results section, Work Plan tab, and War Room tab to verify test results

Test Case Execution

• Create and save XSOAR lists with incident fields and context needed for the test case
  • Example uses TestFields
  • Example uses TestContext
• Define the test case in an XSOAR list (example uses UnitTestCase)
• The use the two action on the Content Testing tab in the Execute Test Cases section to load the incident fields and context from the test case list and then execute the testing commands
  • Prepare Data for Test Case
  • Execute Test Case

Testing List Formats

Automation Unit Test List Format

• <automation name/command name>,<JSON command arguments>

Subplaybook Unit Test List Format

• <subplaybook name>,<JSON subplaybook inputs>

Test Case List Format

Test cases are defined by content testing commands in an XSOAR list. The following test commands are supported:

• LoadFields

  <XSOAR list name where test incident fields are saved>

• LoadContext

  <XSOAR list name where test context are saved>

• Automation

  <XSOAR list name with automation commands and their arguments>

• Subplaybook

  <XSOAR list name with subplaybooks and their inputs>

• Playbook

  <comma separated list of playbook names>

Caveats

• The multi-select option has a hard coded task number of where to add test tasks in the top level testing playbook. If creating a custom testing playbook, modify the MULTISELECT constant in the UnitTestMultiSelect automation to match your playbook
• If custom fields exist required for testing but not added to all incident types, add them to the UnitTesting incident type, or any custom testing incident type created
• Playbooks do not return errors: the “true/false” test results displayed indicate the playbook was was successfully added and executed to the top level testing playbook. Review the testing incident’s war room and work plan to assess actual results of the playbook and playbook tasks

Content Testing Objects Included in the Content Pack

Playbook

• UnitTestTopLevel
• MockPlaybook
• MockSubplaybook

Incident Type

• UnitTesting

Incident Layout

• UnitTestLayout

Incident Fields

• contenttestingcontentautomations
• contenttestingcontentdetails
• contenttestingcontentimpacts
• contenttestingcommithistory
• contenttestingcoverage
• contenttestingpbainfo
• contenttestingdependencies
• contenttestingunittestresults
• contenttestingunittestplaybooks

Assessment Automations

• ChangeHistory
• ListPlaybookAutomationsCommands
• UpgradeCheck

Content Testing Automations

• UnitTest
• UnitTestCase
• UnitTestCasePrep
• UnitTestCoverage
• UnitTestLoadContext
• UnitTestLoadContextList
• UnitTestLoadFields
• UnitTestLoadFieldsList
• UnitTestMultiSelect
• UnitTestResults
• UnitTestSaveContextList
• UnitTestSaveFieldsList
• UnitTestSetField
• UnitTestSubplaybookPrep

Playbook Analysis Automations

• UnitTestPBAStats
• UnitTestPBATaskAvg
• UnitTestPBATaskMax
• UnitTestPlaybookAnalyzer

Assessment Automations

UpgradeCheck

This automation looks at Marketplace content packs requiring updates and assesses impacts on existing custom content. The assessment results are stored in a field (contenttestingcontentimpacts) and displayed in a tab of in the testing incident’s layout. Upgraded content pack details are summarized in another field (contenttestingcontentdetails) and displayed in a tab for copying to test documentation.

Inputs

None

Outputs

None

ChangeHistory

This automation searchs for the version history of all current local changes in XSOAR when using dev to prod and a remote repository. The version history is stored in a field (contenttestingcommithistory) and displayed in a tab of in the testing incident’s layout. Version history is created when an XSOAR object such as a playbook is saved using the Save Version button.

Inputs

None

Outputs

None

ListPlaybookAutomationsCommand

This automation searches playbooks for automation or integration commands used by a playbook and stores them in a field (contenttestingcontentautomations) for display in the testing incident layout. This assists identifying playbooks and automations requiring testing as part of a change to XSOAR content.

Inputs

None

Outputs

None

Content Testing Automations

UnitTest

This automation runs each type of test, dynamically adds tasks to the testing playbook, and displays the results in the testing incident’s layout.

Inputs

Argument Name	Description
playbook	Playbook to execute, blank if testType is Subplaybook or Automation
addAfter	Playbook task (typically a playbook section header) numeric ID to add this testing task after
testType	Type of test to run: Playbook, Subplaybook, Automation, Multiselect
listName	XSOAR list name to read sub-playbooks or automations and their inputs or arguments from. Blank if testType is Playbook or Multiselect

Outputs

None

UnitTestCoverage

Looks at all the content tests that have been executed using the specified playbook and whether each executable (regular, conditional, sub-playbook, and data collection) task was executed. Places Markdown formatted results in the contenttestingcoverage field for display in the incident layout.

Inputs

Argument Name	Description
playbook	Name of the playbook to assess test coverage

Outputs

None

UnitTestResults

This automation displays the results in the testing incident layout and must be executed within a Demisto lock to avoid concurrent updates the the test results grid field. It is invoked by UnitTest.

Inputs

Argument Name	Description
cmds	Array of automation or command names or playbook names executed
tasks	Array of tasks names, created for each testing task
gridfield	The grid incident field to store test results in (contenttestingunittestresults)
status	Array of result of the command (true, false)

Outputs

UnitTestLoadContext

This automation loads context from an existing incident into the current testing incident.

Inputs

Argument Name	Description
id	Incident numeric ID to load context from

Outputs

None

UnitTestSaveContextList

This automation saves context from the current testing incident to an XSOAR list.

Inputs

Argument Name	Description
list	XSOAR list name to save context to

Outputs

None

UnitTestLoadContextList

This automation loads context from an XSOAR list into the current testing incident.

Inputs

Argument Name	Description
list	XSOAR list name to load context from

Outputs

None

UnitTestLoadFields

This automation loads incident fields from an existing incident into the current testing incident.

Inputs

Argument Name	Description
id	Incident numeric ID to load incident fields from

Outputs

None

UnitTestSaveFieldsList

This automation saves incident fields from the current testing incident to an XSOAR list.

Inputs

Argument Name	Description
list	XSOAR list name to save incident fields to

Outputs

None

UnitTestLoadFieldsList

This automation loads incident fields from an XSOAR list into the current testing incident.

Inputs

Argument Name	Description
list	XSOAR list name to load incident fields from

Outputs

None

UnitTestSetField

This automation sets an incident field value in the current testing incident.

Inputs

Argument Name	Description
field	Incident field name to set
value	Value to set incident field to

Outputs

None

UnitTestMultiSelect

This automation is used if a set of playbooks are provided in a multi-select incident field. The selected playbooks are passed to the UnitTest automation. UnitTestMultiSelect hard codes the testing playbook task numeric ID to add these tasks after and assumes the TopLevelUnitTesting playbook is used. If using another testing playbook, edit the MULTISELECT constant in this automation to specify the task location to add the playbook testing task.

Inputs

Argument Name	Description
new	Playbook names selected from the array in the field

Outputs

None

UnitTestSubplaybookPrep

This automation loads context providing sub-playbook inputs. The sub-playbook testing list used is the same as used by UnitTesting. Context must be preloaded since caching of context updates does not occur until an automation exits. This automation must execute prior to UnitTesting executing against the “Subplaybook” type of test that requires playbook inputs.

Inputs

Argument Name	Description
listName	XSOAR list name to read subplaybook context inputs from

Outputs

None

UnitTestCase

This automation is used to execute a test case defined in an XSOAR list.

Inputs

Argument Name	Description
testName	Test case name (not used)
listName	XSOAR list name to read test case from
addAfter	Playbook task numeric ID to add this testing task after

Outputs

None

UnitTestCasePrep

These automation loads context and incident fields defined in a test case XSOAR list to configure it for the test case. Context must be preloaded since caching of context updates does not occur until an automation exits - this automation must execute prior to executing UnitTestCase.

Inputs

Argument Name	Description
testName	Test case name (not used)
listName	XSOAR list name to read test case from

Outputs

None

UnitTestLoadContext

This automation loads context from an existing incident into the current testing incident.

Inputs

Argument Name	Description
id	Incident numeric ID to load context from

Outputs

None

Playbook Analyzer Automations

UnitTestPlaybook Analyzer

Analyzes playbook dependencies on automations and sub-playbooks as well as tasks execution metrics over a set of selected incidents. Generates minumum, average, and maximum execution times for each task in the specified playbook. Execution statistics are stored in the PlaybookStatistics context key for display in the layout. Content dependencies are stored in the contenttestingdependencies field and task execution information is stored contenttestingpbainfo field for display in the layout.

Inputs

Argument Name	Description
playbook	Name of the playbook to analyze
occurred	Incident occurrance time and include in the analysis

Outputs

None

UnitTestPBAStats

General dynamic section script that creates and displays a bar widget that shows the task status (running, executed, not started, waiting, and error) counts for tasks from the data in the PlaybookStatistics context key.

Inputs

None

Outputs

None

UnitTestPBATaskAvg

General dynamic section script that creates and displays a pie widget of the average task durations for each task in the playbook based on the data in the PlaybookStatistics context key.

Inputs

None

Outputs

None

UnitTestPBATaskMax

General dynamic section script that creates and displays a pie widget of the maximum task durations for each task in the playbook based on the data in the PlaybookStatistics context key.

Inputs

None

Outputs

None