SetGridField

Creates a Grid table from items or key-value pairs.

Details

ID	`SetGridField`
Language	python
From Version	5.0.0
Docker Image	`demisto/pandas:1.0.0.10120494`

README

Update Grid Table from items or key value pairs.

Script Data

Name	Description
Script Type	python3
Tags
XSOAR Version	5.0.0

Inputs

Argument Name	Description
context_path	Context path to list of items with similar properties or key value pairs.
grid_id	Grid ID to modify. This argument can be either: 1) Grid name as it appears in the layout. 2) Grid “Machine name”, as can be found in the grid incident field editor under Settings->Advanced->Fields (Incidents).
overwrite	True if to overwrite Grid Data, False otherwise.
columns	Comma-separated list of column header names, for example: columns=”columnheader1,columnheader2,..”
keys	Keys to retrieve from items or "*" for max keys (limited when item list to columns amount) - Key will not be columns correlated. If you want to leave an empty column, provide a place holder name that should not be in the context data such as “PLACE_HOLDER” Make sure the key is lower case and does not contain spaces. For example, for a column header named USER ID, key=”userid”.
sort_by	Columns names by which to sort the rows.
unpack_nested_elements	Set to ‘true’ to unpack nested elements.
keys_from_nested	Keys to retrieve from nested dictionaries. Can be used only when the unpack_nested_elements argument is set to false. Keys will not be columns correlated. Default is all keys. Note: when the number of values exceeds the number of columns, it truncates the last values that are outside the range for table.

Command Example

Assume the following:

Entry Context:

{
  "EWS": {
    "Items": {
      "HeadersMap": {
        "X-MS-Exchange-Organization-AuthSource": "Value1",
        "Received": "Value2",
        "Thread-Index": "Value3",
        "Accept-Language": "Value4"
      },
      "headers": [
        {
          "name": "name1",
          "value": "value1"
        },
        {
          "name": "name2",
          "value": "value2"
        },
        {
          "name": "name3",
          "value": "value3"
        },
        {
          "name": "name4",
          "value": "value4"
        }
      ]
    }
  }
}

Grid: \

Considering the following cases:

Key value to Grid:

```shell script
!SetGridField columns=”columnheader1,columnheader2” context_path=EWS.Items.HeadersMap grid_id=mygrid
keys=”Received,Thread-Index,X-MS-Exchange-Organization-AuthSource,Accept-Language”


Grid after update: \
![Grid](../../doc_files/grid_key_value_update.png)

2. List of item properties to Grid:

```shell script
!SetGridField columns="columnheader1,columnheader2" context_path=EWS.Items.headers grid_id=mygrid 
keys="name, value"

Grid after update: \
Grid

Entry Context:

{
    "PaloAltoNetworksXDR": {
        "RiskyUser": [
            {
                "email": null,
                "id": "1",
                "norm_risk_score": 1000,
                "reasons": [
                    {
                        "date created": "2023-08-20",
                        "description": "test",
                        "points": 90,
                        "severity": "test",
                        "status": "test"
                    },
                    {
                        "date created": "2023-08-20",
                        "description": "test",
                        "points": 90,
                        "severity": "test",
                        "status": "test"
                    }
                ],
                "risk_level": "HIGH",
                "score": 244,
                "type": "user"
            }
        ]
    }
}

!SetGridField_CopyForInvestigation columns=`User id,Risk level,Score,Reasons` grid_id=xdrriskyusers context_path=`PaloAltoNetworksXDR.RiskyUser` keys=`id,risk_level,score,reasons` keys_from_nested=description,points,severity

nested_dict_grid

Troubleshooting

The first time you run SetGridField on a newly created grid field, you may see an error similar to the following:

Screen Shot 2021-12-21 at 10 36 03 PM

To resolve the error:

Make sure the grid field is associated with the incident type the field is being used in.
Run the following command to initialize the grid field: !setIncident <GRID_FIELD_NAME>=[]

Source

SetGridField.py SetGridField_test.py SetGridField.yml

Download History Raw

import json

import demistomock as demisto
import pytest


def util_load_json(path):
    path = f"./test_data/{path}"
    with open(path, encoding="utf-8") as f:
        return json.loads(f.read())


@pytest.mark.parametrize(
    argnames="phrase, norm_phrase",
    argvalues=[("TestPhrase", "testphrase"), ("Test_phrase", "testphrase"), ("test_phrase", "testphrase")],
)
def test_normalized_string(phrase: str, norm_phrase: str):
    from SetGridField import normalized_string

    assert norm_phrase == normalized_string(phrase)


@pytest.mark.parametrize(
    argnames="before_dict, keys, max_keys, after_dict",
    argvalues=[
        ({"a": 1, "b": 2}, ["a"], None, {"a": 1}),
        ({"a": 1, "b": 2}, ["*"], 1, {"a": 1}),
        ({"a": 1, "b": 2}, ["*"], 2, {"a": 1, "b": 2}),
        ({"a": 1, "b": [1, 2, 3]}, ["a"], None, {"a": 1}),
    ],
)
def test_filter_the_dict(before_dict: dict, keys: dict, max_keys: int, after_dict: dict):
    from SetGridField import filter_dict

    assert after_dict == filter_dict(dict_obj=before_dict, keys=keys, max_keys=max_keys)


@pytest.mark.parametrize(
    argnames="entry_context, raise_exception, unpack_nested",
    argvalues=[
        ([{"a": "val", "b": "val"}], False, False),
        ([{"a": [], "b": "val"}], False, False),
        ([{"a": [], "b": "val"}], False, False),
        (["a", "b", 1, False], False, False),
        (["a", "b", 1, False, [{}, "a"]], True, False),
    ],
)
def test_validate_entry_context(capfd, entry_context: dict, raise_exception: bool, unpack_nested: bool):
    from SetGridField import validate_entry_context

    if raise_exception:
        # disabling the stdout check cause along with the exception, we write additional data to the log.
        with pytest.raises(ValueError), capfd.disabled():
            validate_entry_context(context_path="Path", entry_context=entry_context, unpack_nested_elements=unpack_nested)
    else:
        validate_entry_context(context_path="Path", entry_context=entry_context, unpack_nested_elements=unpack_nested)


@pytest.mark.parametrize(
    argnames="keys, columns, dt_response_json, expected_json, unpack_nested",
    argvalues=[
        (["name", "value"], ["col1", "col2"], "context_entry_list.json", "expected_list_grid.json", False),
        (["*"], ["col1", "col2"], "context_entry_dict.json", "expected_dict_grid.json", False),
        (["*"], ["col1"], "context_entry_list_of_values.json", "expected_list_of_values_grid.json", False),
        (["*"], ["col1", "col2"], "context_entry_dict_with_elements.json", "expected_dict_with_elements_grid.json", True),
        (
            ["firstname", "lastname", "email"],
            ["Fname", "Lname", "Email"],
            "context_single_dict_with_keys.json",
            "expected_single_dict_with_keys_grid.json",
            False,
        ),
        (
            ["firstname", "lastname", "email"],
            ["Fname", "Lname", "Email"],
            "context_entry_list_of_dicts.json",
            "expected_list_of_dicts_grid.json",
            False,
        ),
        (
            ["firstname", "lastname", "email", "phones"],
            ["Fname", "Lname", "Email", "Phones"],
            "context_entry_list_of_dicts_complex.json",
            "expected_list_of_dicts_complex.json",
            False,
        ),
    ],
)
def test_build_grid(mocker, keys: list, columns: list, dt_response_json: str, expected_json: str, unpack_nested: bool):
    """
    Given
    - script args
    - a file
    When
    - build_grid command
    Then
    - Validate that the grid was created with the correct column names
    """
    import pandas as pd
    import SetGridField

    mocker.patch.object(SetGridField, "demisto")
    SetGridField.demisto.dt.return_value = util_load_json(dt_response_json)
    expected_grid = util_load_json(expected_json)

    assert (
        pd.DataFrame(expected_grid).to_dict()
        == SetGridField.build_grid(
            context_path=mocker.MagicMock(),
            keys=keys,
            columns=columns,
            unpack_nested_elements=unpack_nested,
            keys_from_nested=["*"],
        ).to_dict()
    )


very_long_column_name = 11 * "column_name_OF_LEN_264__"


@pytest.mark.parametrize(
    argnames="keys, columns, unpack_nested_elements, dt_response_path, expected_results_path",
    argvalues=[
        (
            ["name", "value"],
            ["col!@#$%^&*()ע_1", very_long_column_name],
            False,
            "context_entry_list_missing_key.json",
            "expected_list_grid_none_value.json",
        )
    ],
)
def test_build_grid_command(
    mocker, keys: list[str], columns: list[str], unpack_nested_elements: bool, dt_response_path: str, expected_results_path: str
):
    """
    Given
    - script args
    - a file
    When
    - build_grid_command command
    Then
    - Validate that the grid was created with the correct column names
    """
    import json

    import SetGridField

    mocker.patch.object(SetGridField, "get_current_table", return_value=[])
    mocker.patch.object(SetGridField, "demisto")
    SetGridField.demisto.dt.return_value = util_load_json(dt_response_path)

    results = SetGridField.build_grid_command(
        grid_id="test",
        context_path=mocker.MagicMock(),
        keys=keys,
        columns=columns,
        overwrite=True,
        sort_by=None,
        unpack_nested_elements=unpack_nested_elements,
        keys_from_nested=["*"],
    )

    expected_results = util_load_json(expected_results_path)
    assert json.dumps(results) == json.dumps(expected_results)


@pytest.mark.parametrize(
    argnames="keys, columns, unpack_nested_elements, dt_response_path, expected_results_path",
    argvalues=[
        (
            ["firstname", "lastname", "email"],
            ["fname", "lname", "email"],
            False,
            "context_entry_list_of_dicts_non_sorted.json",
            "expected_entry_list_of_dicts_sorted.json",
        )
    ],
)
def test_build_grid_command_with_sort_by(
    mocker, keys: list[str], columns: list[str], unpack_nested_elements: bool, dt_response_path: str, expected_results_path: str
):
    """
    Given
    - script args, including sort_by
    - a file
    When
    - build_grid_command command
    Then
    - Validate that the grid was created with the correct column names and sorted correctly
    """
    import json

    import SetGridField

    mocker.patch.object(SetGridField, "get_current_table", return_value=[])
    mocker.patch.object(SetGridField, "demisto")

    SetGridField.demisto.dt.return_value = util_load_json(dt_response_path)

    results = SetGridField.build_grid_command(
        grid_id="test",
        context_path=mocker.MagicMock(),
        keys=keys,
        columns=columns,
        overwrite=True,
        sort_by=["fname"],
        unpack_nested_elements=unpack_nested_elements,
        keys_from_nested=["*"],
    )

    expected_results = util_load_json(expected_results_path)
    assert json.dumps(results) == json.dumps(expected_results)


@pytest.mark.parametrize(
    argnames="keys, columns, unpack_nested_elements, dt_response_path, expected_results_path",
    argvalues=[
        (
            ["col1", "col2"],
            ["col1", "col2"],
            False,
            "context_entry_list_of_dicts_non_sorted_multi.json",
            "expected_entry_list_of_dicts_sorted_multi.json",
        )
    ],
)
def test_build_grid_command_with_multi_sort_by(
    mocker, keys: list[str], columns: list[str], unpack_nested_elements: bool, dt_response_path: str, expected_results_path: str
):
    """
    Given
    - script args, including multi sort_by cols
    - a file
    When
    - build_grid_command command
    Then
    - Validate that the grid was created with the correct column names and sorted correctly
    """
    import json

    import SetGridField

    mocker.patch.object(SetGridField, "get_current_table", return_value=[])
    mocker.patch.object(SetGridField, "demisto")

    SetGridField.demisto.dt.return_value = util_load_json(dt_response_path)
    results = SetGridField.build_grid_command(
        grid_id="test",
        context_path=mocker.MagicMock(),
        keys=keys,
        columns=columns,
        overwrite=True,
        sort_by=["col1", "col2"],
        unpack_nested_elements=unpack_nested_elements,
        keys_from_nested=["*"],
    )

    expected_results = util_load_json(expected_results_path)
    assert json.dumps(results) == json.dumps(expected_results)


def side_effect_for_execute_command(command: str, arguments: dict):
    if command == "getIncidents":
        return [{"Type": 1, "Contents": {"ErrorsPrivateDoNotUse": None, "data": [], "total": 0}}]
    if command == "setIncident":
        return None
    return {}


def test_main_does_not_raises_error_in_xsoar(mocker):
    """
     Given
    - An output from executeCommand in XSOAR.
     When
    - Execute SetGridField.
    Then
    - Verify that no error message was raised.
    """
    import SetGridField

    mocker.patch.object(
        demisto,
        "args",
        return_value={
            "grid_id": "grid_id",
            "keys": "key1,key2",
            "columns": "col1,col2",
            "sort_by": "col1",
            "overwrite": "True",
            "unpack_nested_elements": "False",
        },
    )
    mocker.patch.object(demisto, "incident", return_value={"CustomFields": None})
    mocker.patch.object(SetGridField, "is_xsiam_or_xsoar_saas", return_value=False)
    mocker.patch.object(SetGridField, "get_current_table", return_value=[])
    mocker.patch.object(SetGridField, "build_grid_command", return_value=[{"name": "name", "readable_name": "readable_name"}])
    mocker.patch.object(demisto, "executeCommand", side_effect=side_effect_for_execute_command)
    mocked_return_err = mocker.patch.object(SetGridField, "return_error")
    SetGridField.main()
    assert not mocked_return_err.called


def test_main_raises_error_in_xsiam(mocker):
    """
     Given
    - An output from executeCommand in XSIAM.
     When
    - Execute SetGridField.
    Then
    - Verify that an error message was raised.
    """
    import SetGridField

    mocker.patch.object(
        demisto,
        "args",
        return_value={
            "grid_id": "grid_id",
            "keys": "key1,key2",
            "columns": "col1,col2",
            "sort_by": "col1",
            "overwrite": "True",
            "unpack_nested_elements": "False",
        },
    )
    mocker.patch.object(SetGridField, "is_xsiam_or_xsoar_saas", return_value=True)
    mocker.patch.object(demisto, "incident", return_value={"CustomFields": None})
    mocker.patch.object(SetGridField, "get_current_table", return_value=[])
    mocker.patch.object(SetGridField, "build_grid_command", return_value=[{"name": "name", "readable_name": "readable_name"}])
    mocker.patch.object(demisto, "executeCommand", side_effect=side_effect_for_execute_command)
    mocker_return_error = mocker.patch("SetGridField.return_error")
    SetGridField.main()
    assert mocker_return_error.called


def test_get_current_table_exception(mocker):
    """
     Given
    - An output from demisto.incident in XSOAR.
     When
    - Execute get_current_table.
    Then
    - Verify that an error message was raised.
    """

    import SetGridField

    mocker.patch.object(SetGridField, "is_xsiam_or_xsoar_saas", return_value=False)
    mocker.patch.object(demisto, "incident", return_value={"CustomFields": None, "isPlayground": True})
    with pytest.raises(Exception):
        SetGridField.get_current_table("grid_id")