DedupBy

This transformer will remove elements of the array that contain an identical combination of values for the keys given.

python · Filters And Transformers

Details

IDDedupBy
Languagepython
From Version6.8.0
Docker Imagedemisto/python3:3.12.13.10404775
Tagstransformer general

README

This transformer will remove elements of the array that contain an identical combination of values for the keys given.

Script Data

Name Description
Script Type python3
Tags transformer, general
Cortex XSOAR Version 6.8.0

Inputs

Argument Name Description
value The array to deduplicate
keys Comma-separated list of keys to identify a value

Outputs

There are no outputs for this script.

Examples-1

Here is a table to be used as samples in the Examples-1.

DestinationIP SourceIP
1.1.1.1 192.168.1.1
1.1.1.1 192.168.1.1
1.1.1.1 192.168.1.2
1.1.1.1 192.168.1.2
1.1.1.1 192.168.1.3
1.1.1.1 192.168.1.3
2.2.2.2 192.168.1.1
2.2.2.2 192.168.1.1
2.2.2.2 192.168.1.2
2.2.2.2 192.168.1.2
2.2.2.2 192.168.1.3
2.2.2.2 192.168.1.3

The JSON data is below to be given to the value argument parameter of the transformer for the samples.

[
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.3"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.3"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.3"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.3"
  }
]

Deduplicate by SourceIP.

keys: SourceIP

Output

It will give you the result below.
It’s guaranteed to keep the original order, and gives you the first record when multiple records are found by collecting keys given.

DestinationIP SourceIP
1.1.1.1 192.168.1.1
1.1.1.1 192.168.1.2
1.1.1.1 192.168.1.3 
[
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.3"
  }
]

Deduplicate by SourceIP and DestinationIP.

keys: SourceIP, DestinationIP

Output

It will give you the result below.
It’s guaranteed to keep the original order, and gives you the first record when multiple records are found by collecting keys given.

DestinationIP SourceIP
1.1.1.1 192.168.1.1
1.1.1.1 192.168.1.2
1.1.1.1 192.168.1.3
2.2.2.2 192.168.1.1
2.2.2.2 192.168.1.2
2.2.2.2 192.168.1.3 
[
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "1.1.1.1",
    "SourceIP": "192.168.1.3"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.1"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.2"
  },
  {
    "DestinationIP": "2.2.2.2",
    "SourceIP": "192.168.1.3"
  }
]

Examples-2

Here is an array to be used as samples in the Examples-2.
It will be given to the value argument parameter of the transformer for the samples.

[
  null,
  1,
  {
    "key": "value1"
  },
  2,
  0.5,
  0,
  0.5,
  "aaa",
  1,
  {
    "key": "value1"
  },
  null,
  "aaa",
  "ZZZ"
]

Deduplicate an array without keys.

keys:

Output

It will give you the result below.
It’s guaranteed to keep the original order, and gives you the first record when multiple records are found by collecting keys given.

[
  null,
  1,
  {
    "key": "value1"
  },
  2,
  0.5,
  0,
  "aaa",
  "ZZZ"
]

Source

DedupBy.py DedupBy_test.py DedupBy.yml
Download History Raw 
import json

import DedupBy
import demistomock as demisto


def test_1(mocker):
    with open("./test_data/test-1.json") as f:
        test_list = json.load(f)

    for case in test_list:
        value = case["value"]
        expected = case["result"]
        for args in case.get("args") or [{}]:
            keys = args.get("keys")

            mocker.patch.object(demisto, "args", return_value={"value": value, "keys": keys})
            mocker.patch.object(DedupBy, "return_results")
            DedupBy.main()
            assert DedupBy.return_results.call_count == 1
            ret = DedupBy.return_results.call_args[0][0]
            assert ret == expected


def test_performance_large_input(mocker):
    """
    Given: A large dataset with 10,000 items (100 unique items with 100 duplicates each)
    When: Deduplicating by SourceIP and DestinationIP keys
    Then: Should return exactly 100 unique items
    """

    # Create a large dataset with duplicates
    # Each item is a dict with nested structures
    large_value = []
    num_unique = 100
    duplicates_per_item = 100

    for i in range(num_unique):
        # Create multiple duplicates of each unique item
        for _ in range(duplicates_per_item):
            large_value.append(
                {
                    "SourceIP": f"192.168.1.{i}",
                    "DestinationIP": f"10.0.0.{i}",
                    "Port": 443,
                    "Protocol": "HTTPS",
                    "Metadata": {
                        "timestamp": "2026-02-18T00:00:00Z",
                        "severity": "high",
                        "tags": ["network", "security", "monitoring"],
                    },
                }
            )

    mocker.patch.object(demisto, "args", return_value={"value": large_value, "keys": "SourceIP,DestinationIP"})
    mocker.patch.object(DedupBy, "return_results")

    DedupBy.main()
    result = DedupBy.return_results.call_args[0][0]

    assert len(result) == num_unique, f"Expected {num_unique} unique items, got {len(result)}"


def test_performance_comparison_no_keys(mocker):
    """
    Given: A dataset with 2,500 items (500 unique items with 5 duplicates each) containing complex nested objects
    When: Deduplicating entire objects without specifying keys
    Then: Should return exactly 500 unique items
    """

    # Create dataset with complex nested objects
    large_value = []
    for i in range(500):
        for _ in range(5):  # 5 duplicates each
            large_value.append({"id": i, "data": {"nested": {"deep": {"value": f"item_{i}", "list": [1, 2, 3, 4, 5]}}}})

    mocker.patch.object(demisto, "args", return_value={"value": large_value})
    mocker.patch.object(DedupBy, "return_results")

    DedupBy.main()
    result = DedupBy.return_results.call_args[0][0]

    assert len(result) == 500, f"Expected 500 unique items, got {len(result)}"