RecordedFutureLists

Search and manage watchlists in Recorded Future.

Data Enrichment & Threat Intelligence · Recorded Future Intelligence

Details

IDRecordedFutureLists
ProviderMastercard
CategoryData Enrichment & Threat Intelligence
From Version6.5.0
Docker Imagedemisto/python3:3.12.13.10116658
Supported ModulesAgentix XSIAM

README

Search and manage watchlists in Recorded Future
This integration was integrated and tested with version 1.1.1 of RecordedFutureLists

Configure Recorded Future - Lists in Cortex

Parameter Description Required
API URL Default URL: https://api.recordedfuture.com/gw/xsoar/ True
API Token   True
Trust any certificate (not secure)   False
Use system proxy settings   False

Commands

You can execute these commands from the CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

recordedfuture-lists-search


Search for lists in Recorded Future

Base Command

recordedfuture-lists-search

Input

Argument Name Description Required
list_names Freetext name to search for. Optional
contains Filter lists based on entity types, will only include lists with the entity types specified. Default value “” includes all types. Possible values are: entity, source, text, custom, ip, domain, tech_stack, industry, brand, partner, industry_peer, location, supplier, vulnerability, company, hash, operation, attacker, target, method. Optional
limit Limits the amount of returned results. Optional
include Include all search results. Default is to exclude all lists owned by the system user. Possible values are: all. Optional

Context Output

Path Type Description
RecordedFuture.List.id String Unique id of the list in Recorded Future
RecordedFuture.List.name String Name of the list in Recorded Future
RecordedFuture.List.type String Recorded future entity type
RecordedFuture.List.created String Timestamp of creation
RecordedFuture.List.updated String Timestamp of last update to the list
RecordedFuture.List.owner_id String Unique id of the owner in Recorded Future
RecordedFuture.List.owner_name String Readable name of list in Recorded Future

recordedfuture-lists-add-entities


Add entities to a list, separate entities by commas. “NOTE:” if entity type is specified, only one entity type can be added with each action.
When adding IDs use the following for Recorded Future light entities:

  • IPaddress: “ip:x.x.x.x”
  • Domain: “idn:example.xyz”
  • Hash: “hash:examplehashvalue”
  • Email: “email:example@example.xyz”
  • Url: “url:https://example.xyz”

Base Command

recordedfuture-lists-add-entities

Input

Argument Name Description Required
list_id Id of the list that should be added, can be found by running !recordedfuture-lists-search with the corresponding filters or in the Recorded Future portal. Required
entity_ids Specific ids from Recorded Future separated by comma, For urls containing commas: replace comma with %2C. Optional
freetext_names Freetext names will be matched to Recorded Future ids separated by comma, this alernative will add the best match in the Recorded Future data. For urls containing commas: escape with %2C. Optional
entity_type Type of the entities that should be added. Use together with freetext_names to improve entity resolution. Possible values are: ip, domain, malware, url, hash, cve, company, person, product, industry, country, attack-vector, operation, mitre-identifier, malware-category. Optional

Context Output

Path Type Description
RecordedFuture.List.Entities.name String Name of the entity in the list
RecordedFuture.List.Entities.type String The Recorded Future entity type resolved during the action
RecordedFuture.List.Entities.id String Unique id of the entity in Recorded Future
RecordedFuture.List.Entities.input_value String The value inputted to the command
RecordedFuture.List.Entities.action_result String Entity specific result for the action

recordedfuture-lists-remove-entities


Remove entities from a list. Separate entities with commas. “NOTE:” If entity type is specified, only one entity type can be added with each action.
When adding IDs use the following for Recorded Future light entities:

  • IPaddress: “ip:x.x.x.x”
  • Domain: “idn:example.xyz”
  • Hash: “hash:examplehashvalue”
  • Email: “email:example@example.xyz”
  • Url: “url:https://example.xyz”

Base Command

recordedfuture-lists-remove-entities

Input

Argument Name Description Required
list_id ID of the list that should be removed. Can be found by running !recordedfuture-lists-search with the corresponding filters or in the Recorded Future portal. Required
entity_ids A comma-separated list of specific IDs from Recorded Future. For URLs containing commas, replace comma with %2C. Optional
freetext_names A comma-separated list of freetext names to be matched to Recorded Future IDs. This will remove the best match in the Recorded Future data. For URLs containing commas, escape with %2C. Optional
entity_type Type of the entities that should be removed. Use together with freetext_names to improve entity resolution. Possible values are: ip, domain, malware, url, hash, cve, company, person, product, industry, country, attack-vector, operation, mitre-identifier, malware-category. Optional

Context Output

Path Type Description
RecordedFuture.List.Entities.name String Name of the entity in the list
RecordedFuture.List.Entities.type String The Recorded Future entity type resolved during the action.
RecordedFuture.List.Entities.id String Unique ID of the entity in Recorded Future.
RecordedFuture.List.Entities.input_value String The value inputted to the command.
RecordedFuture.List.Entities.action_result String Entity specific result for the action.

recordedfuture-lists-entities


Get the entities that are currently in the given lists.

Base Command

recordedfuture-lists-entities

Input

Argument Name Description Required
list_ids A comma-separated list of Recorded Future list IDs. Required

Context Output

Path Type Description
RecordedFuture.List.id String Unique ID of the list in Recorded Future.
RecordedFuture.List.name String Name of the list in Recorded Future.
RecordedFuture.List.type String Recorded Future entity type.
RecordedFuture.List.Entities.name String Name of the entity in the list.
RecordedFuture.List.Entities.type String The Recorded Future entity type resolved during the action.
RecordedFuture.List.Entities.id String Unique ID of the entity in Recorded Future.

Configuration parameters

  • server_url — API URL (required)
  • token — (required)
  • insecure — Trust any certificate (not secure)
  • proxy — Use system proxy settings

Commands (4)

  • recordedfuture-lists-add-entities

    Add entities to a list, separate entities by commas. "NOTE:" if entity type is specified, only one entity type can be added with each action.

  • recordedfuture-lists-entities

    Get the entities that are currently in the given lists.

  • recordedfuture-lists-remove-entities

    Remove entities from a list. Separate entities with a comma. "NOTE:" If entity type is specified, only one entity type can be added with each action.

  • recordedfuture-lists-search

    Search for lists in Recorded Future.

import demistomock as demisto  # noqa: F401
from CommonServerPython import *  # noqa: F401

"""Recorded Future Lists Integration for Demisto."""

import platform
import json

# flake8: noqa: F402,F405 lgtm

STATUS_TO_RETRY = [500, 501, 502, 503, 504]

# disable insecure warnings
requests.packages.urllib3.disable_warnings()  # type: ignore

__version__ = "1.1.1"


# === === === === === === === === === === === === === === ===
# === === === === Recorded Future API Client === === === ====
# === === === === === === === === === === === === === === ===


class Client(BaseClient):
    def whoami(self) -> Dict[str, Any]:
        return self._http_request(
            method="get",
            url_suffix="info/whoami",
            timeout=60,
        )

    def _call(self, url_suffix, **kwargs):
        json_data = {
            "demisto_command": demisto.command(),
            "demisto_args": demisto.args(),
        }
        if "demisto_args" in kwargs:
            if args := kwargs.get("demisto_args"):
                json_data.update({"demisto_args": args})
            kwargs.pop("demisto_args")

        method = kwargs.get("method", "post")

        request_kwargs = {
            "method": method,
            "url_suffix": url_suffix,
            "json_data": json_data,
            "timeout": 90,
            "retries": 3,
            "status_list_to_retry": STATUS_TO_RETRY,
        }

        request_kwargs.update(kwargs)

        try:
            response = self._http_request(**request_kwargs)

            if isinstance(response, dict) and response.get("return_error"):
                # This will raise the Exception or call "demisto.results()" for the error and sys.exit(0).
                return_error(**response["return_error"])

        except DemistoException as err:
            if "404" in str(err):
                return CommandResults(
                    outputs_prefix="",
                    outputs={},
                    raw_response={},
                    readable_output="No results found.",
                    outputs_key_field="",
                )
            else:
                raise err

        return response

    ####################################################
    ################## List operations #################
    ####################################################

    def list_search(self) -> Dict[str, Any]:
        parsed_args = demisto.args()
        if list_names := parsed_args.get("list_names"):
            parsed_args["list_names"] = list_names.split(",")
        if types := parsed_args.get("contains"):
            parsed_args["contains"] = types.split(",")
        """Search for lists in Recorded Future"""
        return self._call(url_suffix="/v2/lists/search", demisto_args=parsed_args)

    ####################################################
    ################ Entity operations #################
    ####################################################

    def entity_operation(self, operation) -> Dict[str, Any]:
        parsed_args = demisto.args()

        list_id = parsed_args.pop("list_id")

        if entity_ids := parsed_args.get("entity_ids"):
            parsed_args["entity_ids"] = entity_ids.split(",")

        if freetext_names := parsed_args.get("freetext_names"):
            parsed_args["freetext_names"] = freetext_names.split(",")

        if not ((entity_ids and not freetext_names) or (not entity_ids and freetext_names)):  # XOR entity_ids and freetext_names
            raise ValueError(
                "Command expected 1 of parmeters: entity_ids or freetext_names, to be specified."
                f" Got {len([x for x in [entity_ids, freetext_names] if x])} specified."
            )

        return self._call(
            url_suffix=f"/v2/lists/{list_id}/entities/{operation}",
            demisto_args=parsed_args,
        )

    def entity_fetch(self) -> Dict[str, Any]:
        parsed_args = demisto.args()
        if list_ids := parsed_args.get("list_ids"):
            parsed_args["list_ids"] = list_ids.split(",")
        return self._call(url_suffix="/v2/lists/entities/lookup", demisto_args=parsed_args)


# === === === === === === === === === === === === === === ===
# === === === === === === ACTIONS === === === === === === ===
# === === === === === === === === === === === === === === ===


class Actions:
    def __init__(self, rf_client: Client):
        self.client = rf_client

    def _process_result_actions(self, response: Union[dict, CommandResults]) -> List[CommandResults]:
        if isinstance(response, CommandResults):
            # Case when we got 404 on response, and it was processed in self.client._call() method.
            return [response]
        elif not isinstance(response, dict):
            # In case API returned a str - we don't want to call "response.get()" on a str object.
            return None  # type: ignore

        result_actions: Union[List[dict], None] = response.get("result_actions")

        if not result_actions:
            return None  # type: ignore

        command_results: List[CommandResults] = []
        for action in result_actions:
            if "CommandResults" in action:
                command_results.append(CommandResults(**action["CommandResults"]))

        return command_results

        #######################################################
        #################### List actions #####################
        #######################################################

    def list_search_command(self) -> List[CommandResults]:
        response = self.client.list_search()
        return self._process_result_actions(response=response)

        #######################################################
        ################### Entity actions ####################
        #######################################################

    def entity_add_command(self) -> List[CommandResults]:
        response = self.client.entity_operation("add")
        return self._process_result_actions(response=response)

    def entity_remove_command(self) -> List[CommandResults]:
        response = self.client.entity_operation("remove")
        return self._process_result_actions(response=response)

    def entities_get_command(self) -> List[CommandResults]:
        response = self.client.entity_fetch()
        return self._process_result_actions(response=response)


# === === === === === === === === === === === === === === ===
# === === === === === === === MAIN === === === === === === ==
# === === === === === === === === === === === === === === ===


def main() -> None:
    """Main method used to run actions."""
    try:
        demisto_params = demisto.params()
        base_url = demisto_params.get("server_url", "").rstrip("/")
        verify_ssl = not demisto_params.get("insecure", False)
        proxy = demisto_params.get("proxy", False)

        headers = {
            "X-RFToken": demisto_params["token"].get("password"),
            "X-RF-User-Agent": (
                f'RecordedFutureLists.py/{__version__} ({platform.platform()}) '
                f'XSOAR/{__version__} '
                f'RFClient/{__version__} (Cortex_XSOAR_{demisto.demistoVersion()["version"]})'
            ),
        }
        client = Client(base_url=base_url, verify=verify_ssl, headers=headers, proxy=proxy)
        command = demisto.command()
        actions = Actions(client)

        if command == "test-module":
            # This is the call made when pressing the integration Test button.
            # Returning 'ok' indicates that the integration works like it suppose to and
            # connection to the service is successful.
            # Returning 'ok' will make the test result be green.
            # Any other response will make the test result be red.

            try:
                client.whoami()
                return_results("ok")
            except Exception as err:
                message = str(err)
                try:
                    error = json.loads(str(err).split("\n")[1])
                    if "fail" in error.get("result", {}).get("status", ""):
                        message = error.get("result", {})["message"]
                except Exception:
                    message = (
                        f"Unknown error. Please verify that the API URL and Token are correctly configured. RAW Error: {err}"
                    )
                raise DemistoException(f"Failed due to - {message}")

        #######################################################
        ################### List commands #####################
        #######################################################

        elif command == "recordedfuture-lists-search":
            return_results(actions.list_search_command())
        #######################################################
        ################## Entity commands ####################
        #######################################################

        elif command == "recordedfuture-lists-add-entities":
            return_results(actions.entity_add_command())

        elif command == "recordedfuture-lists-remove-entities":
            return_results(actions.entity_remove_command())

        elif command == "recordedfuture-lists-entities":
            return_results(actions.entities_get_command())

    except Exception as e:
        return_error(message=f"Failed to execute {demisto.command()} command: {str(e)}")


if __name__ in ("__main__", "__builtin__", "builtins"):
    main()