ExtractIndicatorsFromTextFile

Extract indicators from a text-based file. Indicators that can be extracted: * IP * Domain * URL * File Hash * Email Address This automation runs using the default Limited User role, unless you explicitly change the permissions. For more information, see the section about permissions here: - For Cortex XSOAR 6 see https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.x/Cortex-XSOAR-Playbook-Design-Guide/Automations - For Cortex XSOAR 8 Cloud see https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8/Cortex-XSOAR-Cloud-Documentation/Create-a-script - For Cortex XSOAR 8.7 On-prem see https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8.7/Cortex-XSOAR-On-prem-Documentation/Create-a-script

python · Common Scripts

Details

ID	`ExtractIndicatorsFromTextFile`
Language	python
From Version	5.0.0
Docker Image	`demisto/python3:3.12.13.10404775`

README

Extract indicators from a text-based file.
Indicators that can be extracted:

IP
Domain
URL
File Hash
Email Address

This automation runs using the default Limited User role, unless you explicitly change the permissions.
For more information, see the section about permissions here: For Cortex XSOAR 6, see the https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.x/Cortex-XSOAR-Playbook-Design-Guide/Automations for Cortex XSOAR 8 Cloud, see the https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8/Cortex-XSOAR-Cloud-Documentation/Create-a-script for Cortex XSOAR 8 On-prem, see the https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8.7/Cortex-XSOAR-On-prem-Documentation/Create-a-script.

Script Data

Name	Description
Script Type	python2
Cortex XSOAR Version	5.0.0

Used In

This script is used in the following playbooks and scripts.

Extract Indicators From File - Generic
Extract Indicators From File - Generic v2

Inputs

Argument Name	Description
entryID	The War-Room entryID of the file to read.
maxFileSize	Maximal file size to load, in bytes. Default is 1000000 (1MB).

Outputs

Path	Description	Type
Domain.Name	Extracted domains	string
Account.Email.Address	Extracted emails	string
File.MD5	Extracted MD5	string
File.SHA1	Extracted SHA1	string
File.SHA256	Extracted SHA256	string
IP.Address	Extracted IPs	string
URL.Data	Extracted URLs	string

Source

ExtractIndicatorsFromTextFile.py ExtractIndicatorsFromTextFile_test.py ExtractIndicatorsFromTextFile.yml

Download History Raw

import pytest
from ExtractIndicatorsFromTextFile import *


def execute_command(command, args):
    if command == "getFilePath":
        return [{"Contents": {"path": "./test_data/test_file.txt"}}]
    if command == "extractIndicators":  # noqa: RET503
        return [{"Contents": '{"IP": ["1.1.1.1"]}'}]


def test_extract_indicators(mocker):
    """
    Given:
        A file containing an indicator.

    When:
        Running script on file

    Then:
        Validate the right output returns.
    """
    mocker.patch.object(demisto, "executeCommand", side_effect=execute_command)
    args: Dict[str, str] = {}
    results = extract_indicators_from_file(args)
    assert results == {
        "Contents": '{"IP": ["1.1.1.1"]}',
        "ContentsFormat": "text",
        "EntryContext": {"IP": ["1.1.1.1"]},
        "HumanReadable": "### IP\n- 1.1.1.1\n",
        "Type": 1,
    }


def test_extract_indicators_no_file():
    """
    Given:
        Name of file that does not exist.

    When:
        Running script on file

    Then:
        Validate the right output returns.
    """
    args = {"maxFileSize": 1024**2}
    with pytest.raises(FileNotFoundError) as e:
        extract_indicators_from_file(args)
        if not e:
            pytest.fail()


@pytest.mark.parametrize("params", [('{"IP": ["1.1.1.1"]}', "### IP\n- 1.1.1.1\n"), ("a", 'JSON Decode failed on "a"')])
def test_string_to_markdown(capfd, params):
    """
    Given:
        JSON of an indicator with type as a key

    When:
        Running script on file

    Then:
        Validate the right output returns.
    """
    input, expected_output = params
    output = string_to_markdown(input)
    out, err = capfd.readouterr()
    assert output == expected_output


@pytest.mark.parametrize("filePath, res", [("./test_data/test_file.txt", "1.1.1.1"), ("./test_data/latin-file.txt", "áÈË")])
def test_read_encoded_file(filePath, res):
    """
    # DON'T EDIT THE TEST FILES
    # this breaks encoding.
    # instead use encoding script like this -
        import binascii
        open('./test_data/latin-file.txt', mode='w', encoding='latin-1').write('áÈË')

        Given:
            file path to an encoded file.

        When:
            Running read_file_with_encoding_detection function

        Then:
            Validate the right data is returned.
    """
    assert read_file_with_encoding_detection(filePath, 1024**2) == res