FindSimilarIncidents Deprecated

Deprecated. Use DBotFindSimilarIncidents instead. Finds similar incidents by common incident keys, labels, custom fields or context keys. It's highly recommended to use incident keys if possible (e.g., "type" for the same incident type). For best performance, it's recommended to avoid using context keys if possible (for example, if the value also appears in a label key, use label). This automation runs using the default Limited User role, unless you explicitly change the permissions. For more information, see the section about permissions here: - For Cortex XSOAR 6 see https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.x/Cortex-XSOAR-Playbook-Design-Guide/Automations - For Cortex XSOAR 8 Cloud see https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8/Cortex-XSOAR-Cloud-Documentation/Create-a-script - For Cortex XSOAR 8.7 On-prem see https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8.7/Cortex-XSOAR-On-prem-Documentation/Create-a-script

python · Common Scripts

Details

ID	`FindSimilarIncidents`
Language	python
From Version	5.0.0
Docker Image	`demisto/python3:3.10.13.83255`
Tags	dedup duplicate incidents

README

Deprecated. Use DBotFindSimilarIncidents instead.

Finds similar incidents by common incident keys, labels, custom fields or context keys.
It’s highly recommended to use incident keys if possible (e.g., “type” for the same incident type).
For best performance, it’s recommended to avoid using context keys if possible (for example, if the value also appears in a label key, use label).

This automation runs using the default Limited User role, unless you explicitly change the permissions.
For more information, see the section about permissions here: For Cortex XSOAR 6, see the https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.x/Cortex-XSOAR-Playbook-Design-Guide/Automations for Cortex XSOAR 8 Cloud, see the https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8/Cortex-XSOAR-Cloud-Documentation/Create-a-script for Cortex XSOAR 8 On-prem, see the https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8.7/Cortex-XSOAR-On-prem-Documentation/Create-a-script.

Script Data

Name	Description
Script Type	python2
Tags	dedup, duplicate, incidents
Cortex XSOAR Version	5.0.0

Used In

This script is used in the following playbooks and scripts.

Cortex XDR incident handling v2
DeDup incidents
Dedup - Generic
Dedup - Generic v2
Dedup - Generic v3
Handle Darktrace Model Breach
JOB - Integrations and Incidents Health Check
Palo Alto Networks - Endpoint Malware Investigation v2
Palo Alto Networks - Endpoint Malware Investigation v3
Shift handover

Inputs

Argument Name	Description
similarIncidentKeys	A comma-separated list of identical incident keys.
similarLabelsKeys	A comma-separated list of similar label keys. Comma separated value. Also supports allowing X different words between labels, within the following way: label_name:X, where X is the number of words. X can also be ‘’ for contains. For example: the value “Email/subject:” will consider email subject similar, if one is substring of the other.
similarContextKeys	A comma-separated list of similar context keys. Also supports allowing X different words between values (see the labels description).
similarCustomFields	A comma-separated list of Similar custom fields keys. Also supports allowing X different words between values (see the labels description).
ignoreClosedIncidents	Whether to ignore closed incidents as duplicate candidates. Can be “yes” (ignore) or “no” (don’t ignore). The default value is “yes”.
maxNumberOfIncidents	Maximum number of incidents to query.
hoursBack	Query incidents in the last X hours. Supports float value.
timeField	Filter incidents by this time field.
maxResults	Maximum number of results to display.
similarIncidentFields	A comma-separated list of similar incident fields keys. Also supports allowing X different words between values (see the labels description).
filterQuery	Use this query condition when fetching duplicate incidents.
incidentFieldsAppliedCondition	The condition to apply between incident fields. Can be “OR” or “AND”. This will apply only for fields with “exact match”.
skipMissingValues	Whether to skip the incident if it does not have specific key. Can be “yes” (skip) or “no” (don’t skip). The default value is “yes”. WARNING: if no fields exist in the incident, random incidents might be returned as results due to the empty condition.

Outputs

Path	Description	Type
similarIncident.rawId	Similar incident ID.	string
isSimilarIncidentFound	Whether a similar incident was found (“true” or “false”).	boolean
similarIncident	Similar incident.	unknown
similarIncident.name	Similar incident name.	string

Source

FindSimilarIncidentsV2.py FindSimilarIncidentsV2_test.py FindSimilarIncidentsV2.yml

Download History Raw

import pytest

from CommonServerPython import *
import demistomock as demisto

default_args = {
    'hoursBack': 5,
    'timeField': 'created',
    'ignoreClosedIncidents': 'yes',
    'maxNumberOfIncidents': 3000,
    'maxResults': 10,
    'skipMissingValues': 'yes',
    'incidentFieldsAppliedCondition': 'AND'
}

incident1 = {
    'id': 1,
    'name': 'This is incident1',
    'type': 'Phishing',
    'severity': 0,
    'status': 1,
    'created': '2019-01-02',
    'closed': '0001-01-01T00:00:00Z',
    'labels': [{'type': 'subject', 'value': 'This subject1'}, {'type': 'unique', 'value': 'This subject1'}],
    'attachment': [{'name': 'Test word1 word2'}]
}

context1 = {
    'simpleValue': 'simple',
    'listValue': [{'name': 'test1'}, {'name': 'test2'}],
    'dictListValue': {'test': ['1', '2']},
    'simpleListValue': ['1', '2', '3'],
    'dictListValue2': {'first': ['1', '2', '3'], 'second': ['1']},
    'multipleDictsValue': {
        'test':
            {
                'first':
                    {
                        'word1': '1', 'word2': '2'
                    },
                'second': ['word3', 'word4']
            }
    }
}
context2 = context1

context3 = {
    'listValue': [{'name': 'test1'}, {'name': 'test3'}],
    'dictListValue': {'test': ['2', '1']},
    'simpleListValue': ['2', '1', '3'],
    'dictListValue2': {'first': ['1', '2', '3']},
    'multipleDictsValue': {
        'test':
            {
                'first':
                    {
                        'word1': '4', 'word2': '2'
                    },
                'second': ['word3', 'word4']
            }
    }
}

incident2 = {
    'id': 2,
    'name': 'This is incident2',
    'type': 'Phishing',
    'severity': 0,
    'status': 1,
    'created': '2019-01-02',
    'closed': '0001-01-01T00:00:00Z',
    'labels': [{'type': 'subject', 'value': 'This subject2'}],
    'attachment': [{'name': 'Test word1'}]
}

incident1_dup = {
    'id': 3,
    'name': 'This is incident1',
    'type': 'Phishing',
    'severity': 0,
    'status': 1,
    'created': '2019-01-01',
    'closed': '0001-01-01T00:00:00Z',
    'labels': [{'type': 'subject', 'value': 'This subject1'}],
    'attachment': [{'name': 'Test word1 word2'}]
}

incident1_dup2 = {
    'id': 4,
    'name': 'This is incident1',
    'type': 'Phishing',
    'severity': 0,
    'status': 1,
    'created': '2019-01-01',
    'closed': '0001-01-01T00:00:00Z',
    'labels': [{'type': 'subject', 'value': 'This subject1'}],
    'attachment': [{'name': 'Test word1 word2'}]
}

incident_by_keys = [
    {
        'CustomFields': {},
        'account': '',
        'activated': '0001-01-01T00:00:00Z',
        'attachment': None,
        'autime': 1550670443962164000,
        'canvases': None,
        'category': '',
        'closeNotes': '',
        'closeReason': '',
        'closed': '0001-01-01T00:00:00Z',
        'closingUserId': '',
        'created': '2019-02-20T15:47:23.962164+02:00',
        'details': '',
        'droppedCount': 0,
        'dueDate': '2019-03-02T15:47:23.962164+02:00',
        'hasRole': False,
        'id': '1',
        'investigationId': '1',
        'isPlayground': False,
        'labels': [{'type': 'Instance', 'value': 'test'},
                   {'type': 'Brand', 'value': 'Manual'}],
        'lastOpen': '0001-01-01T00:00:00Z',
        'linkedCount': 0,
        'linkedIncidents': None,
        'modified': '2019-02-20T15:47:27.158969+02:00',
        'name': '1',
        'notifyTime': '2019-02-20T15:47:27.156966+02:00',
        'occurred': '2019-02-20T15:47:23.962163+02:00',
        'openDuration': 0,
        'owner': 'analyst',
        'parent': '',
        'phase': '',
        'playbookId': 'playbook0',
        'previousRoles': None,
        'rawCategory': '',
        'rawCloseReason': '',
        'rawJSON': '',
        'rawName': '1',
        'rawPhase': '',
        'rawType': 'Unclassified',
        'reason': '',
        'reminder': '0001-01-01T00:00:00Z',
        'roles': None,
        'runStatus': 'waiting',
        'severity': 0,
        'sla': 0,
        'sourceBrand': 'Manual',
        'sourceInstance': 'amichay',
        'status': 1,
        'type': 'Unclassified',
        'version': 6
    }
]


@pytest.fixture(autouse=True)
def mock_demistoVersion(mocker):
    mocker.patch.object(demisto, 'demistoVersion', return_value={'platform': 'xsoar'})


def execute_command(command, args=None):
    if command == 'getIncidents':
        entry = {}
        entry['Type'] = entryTypes['note']
        entry['Contents'] = {}
        entry['Contents']['data'] = [incident1_dup, incident2, incident1_dup2]
        return [entry]
    elif command == 'getContext':
        if args['id'] == 1:
            return [{'Contents': {'context': context1}}]
        elif args['id'] == 2:
            return [{'Contents': {'context': context2}}]
        elif args['id'] == 3:
            return [{'Contents': {'context': context3}}]
        else:
            return []
    else:
        return []


def test_get_incidents_by_keys():
    from FindSimilarIncidentsV2 import get_incidents_by_keys
    res = get_incidents_by_keys({u'name': u'\U0001f489'}, 'created', '2020-10-07T19:49:37.392378+03:00', '7137', 72,
                                False, '1000', 'status:Closed', 'AND')
    assert res == incident_by_keys


def test_build_similar_keys_list():
    """
    Given
        - key and value of the incident fields.
    When
        - build_incident_fields_query is called
    Then
        - Ensure that the query matches the type of the incident field: int -> ":=", str -> "=".
    """
    from FindSimilarIncidentsV2 import build_incident_fields_query

    int_res = build_incident_fields_query({u'sla': 0})
    assert int_res == ['sla:=0']

    str_res = build_incident_fields_query({u'employeeid': u'1111'})
    assert str_res == [u'employeeid="1111"']

    list_res = build_incident_fields_query({u'test': [u'name1', 0]})
    assert list_res == [u'test="name1"', 'test:=0']

    list_res = build_incident_fields_query({u'test': []})
    assert list_res == ['test=[]']

    escape_res = build_incident_fields_query({u'test': u'"C:\\test\\escape\\sequence" test'})
    assert escape_res == [u'test="\\"C:\\test\\escape\\sequence\\" test"']


def test_similar_incidents_fields(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarLabelsKeys': 'subject'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])

    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 2
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3


def test_similar_incidents_fields_with_diff(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarLabelsKeys': 'subject:1'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])

    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 3
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 4
    assert result['EntryContext']['similarIncidentList'][2]['rawId'] == 2


def test_similar_incidents_missing_fields(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name,emailbody', 'similarLabelsKeys': 'emailbody'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)

    with pytest.raises(SystemExit) as err:
        args['skipMissingValues'] = 'no'
        main()
    assert err.type == SystemExit


def test_similar_incidents_list_field(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'attachment.name:1'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 3
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 4
    assert result['EntryContext']['similarIncidentList'][2]['rawId'] == 2


def test_similar_incidents_no_results(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarLabelsKeys': 'unique'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)

    with pytest.raises(SystemExit) as err:
        main()
    assert err.type == SystemExit


def test_similar_incidents_order(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarLabelsKeys': 'subject'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])

    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 2
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 4


def test_similar_context_simple_value(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'simpleValue'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2


def test_similar_context_list_value(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'listValue.name'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'listValue'})

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2


def test_similar_context_dict_list_value(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'dictListValue'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2

    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'dictListValue.test'})
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 2
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 2


def test_similar_context_dict_list_value2(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'dictListValue2'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2

    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'dictListValue2.first'})
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 2
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 2

    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'dictListValue2.second'})
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2


def test_similar_context_multiple_dicts_value(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'multipleDictsValue'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2

    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'multipleDictsValue.test.second'})
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 2
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 2

    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'multipleDictsValue.test'})
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2

    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'multipleDictsValue.test.first'})
    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 1
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 2


def test_similar_context_simple_list_value(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'similarIncidentFields': 'name', 'similarContextKeys': 'simpleListValue'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)

    result = main()
    assert len(result['EntryContext']['similarIncidentList']) == 2
    assert result['EntryContext']['similarIncidentList'][0]['rawId'] == 3
    assert result['EntryContext']['similarIncidentList'][1]['rawId'] == 2


def test_similar_context_missing_key(mocker):
    from FindSimilarIncidentsV2 import main
    args = dict(default_args)
    args.update({'skipMissingValues': 'no', 'similarContextKeys': 'missingKey'})

    mocker.patch.object(demisto, 'args', return_value=args)
    mocker.patch.object(demisto, 'incidents', return_value=[incident1])
    mocker.patch.object(demisto, 'executeCommand', side_effect=execute_command)
    mocker.patch.object(demisto, 'context', return_value=context1)
    mocker.patch.object(demisto, 'dt', side_effect=dt_res)

    with pytest.raises(ValueError, match="Error: Missing context key for incident: missingKey"):
        main()


def test_build_incident_query():
    from FindSimilarIncidentsV2 import build_incident_query
    res = build_incident_query('type="test"', True, '826', 'name:incident1')

    assert res == '(-id:826) and (type="test" and -status:Closed) and (name:incident1)'


def test_build_incident_query_without_similar_keys_query():
    from FindSimilarIncidentsV2 import build_incident_query
    res = build_incident_query('', True, '826', 'name:incident1')

    assert res == '(-id:826) and (-status:Closed) and (name:incident1)'


def test_build_incident_query_without_extra_query():
    from FindSimilarIncidentsV2 import build_incident_query
    res = build_incident_query('', True, '826', '')

    assert res == '(-id:826) and (-status:Closed)'


def test_build_incident_query_without_similar_keys_query_and_ignore_closed():
    """
    Given:
        - Fields for creating query without similar_keys_query and ignore_closed flag.

    When:
        - Building incident query using 'build_incident_query' function.

    Then:
        - Ensure the query was created correctly and that the query string got the values needed
          from the formating.
    """
    from FindSimilarIncidentsV2 import build_incident_query
    res = build_incident_query('', False, '826', '')

    assert res == '(-id:826)'


def dt_res(context, keys_to_search):

    keys_list = keys_to_search.split('.')
    context_key_value = None
    for key in keys_list:
        if context_key_value:
            if isinstance(context_key_value, list):
                list_value = []
                for value in context_key_value:
                    if isinstance(value, dict):
                        if value.get(key):
                            list_value.append(value[key])
                    else:
                        list_value.append(value)
                context_key_value = list_value

            elif isinstance(context_key_value, dict):
                dict_list_value = []
                for dict_key in context_key_value.keys():
                    if dict_key == key:
                        if isinstance(context_key_value[dict_key], list):
                            dict_list_value.append(sorted(context_key_value[dict_key]))
                        else:
                            dict_list_value.append(context_key_value[dict_key])

                context_key_value = dict_list_value

            else:
                context = context_key_value
        else:
            context_key_value = context.get(key)
    return context_key_value