FeedUstaThreatStream

Fetches indicators from the USTAv4 Threat Stream feed. The indicators can be of type malicious URLs or malware hashes.

Data Enrichment & Threat Intelligence · USTAv4 Cyber Threat Intelligence Platform · Feed

Details

IDFeedUstaThreatStream
ProviderAnomali
CategoryData Enrichment & Threat Intelligence
From Version6.10.0
Docker Imagedemisto/python3:3.12.13.10116658
Supported ModulesAgentix XSIAM

README

This integration fetches indicators from the USTAv4 Threat Stream feed. The indicators can be of type malicious URLs or malware hashes.
This integration was integrated and tested with version 4.1.0 of FeedUstaThreatStream.

Configure USTAv4 Threat Stream IOC Feed in Cortex

  1. Navigate to Settings > Integrations > Servers & Services.
  2. Search for USTAv4 Threat Stream IOC Feed.
  3. Click Add instance to create and configure a new integration instance.

    Parameter Description Required
    Fetch indicators   False
    Server’s URL   True
    API Key The API Key to use for connection True
    IOC Feed Type   True
    Indicator Reputation Indicators from this integration instance will be marked with this reputation False
    Source Reliability Reliability of the source providing the intelligence data True
    Traffic Light Protocol Color The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed False
    Feed Fetch Interval   False
    Bypass exclusion list When selected, the exclusion list is ignored for indicators from this feed. This means that if an indicator from this feed is on the exclusion list, the indicator might still be added to the system. False
    Trust any certificate (not secure)   False
    Use system proxy settings   False
        False
        False
    Tags Supports CSV values. False
  4. Click Test to validate the URLs, token, and connection.

Commands

You can execute these commands from the Cortex XSOAR CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

usta-tsa-search-malware-hash


Search malware hash indicators from the feed.

Base Command

usta-tsa-search-malware-hash

Input

Argument Name Description Required
limit The maximum number of results to return. Default is 10. Optional
hash The hash to search for. It can be a SHA-1, SHA-256, or MD5 hash. Required

Context Output

Path Type Description
USTA.ThreatStreamMalwareHashes.id String The ID of the alert
USTA.ThreatStreamMalwareHashes.hashes[0].sha256 String The SHA-256 hash of the malware
USTA.ThreatStreamMalwareHashes.hashes[0].md5 String The MD5 hash of the malware
USTA.ThreatStreamMalwareHashes.hashes[0].sha1 String The SHA-1 hash of the malware
USTA.ThreatStreamMalwareHashes.tags Array The tags of the malware
USTA.ThreatStreamMalwareHashes.created Date The creation date of the malware
USTA.ThreatStreamMalwareHashes.valid_from Date The valid from date of the malware
USTA.ThreatStreamMalwareHashes.valid_until Date The valid until date of the malware

Command Example

!usta-tsa-search-malware-hash hash=d5d8c33957e90d1caca4b5207d8da5ab1bc4caa9f702abc0ec006d0518ea9aec

Context Example

{
    "USTA" :{
        "ThreatStreamMalwareHashes":[
             {
                "id": "bf89614f-0ec8-4a88-a4e7-085b113a871b",
                "hashes": {
                    "sha256": "d5d8c33957e90d1caca4b5207d8da5ab1bc4caa9f702abc0ec006d0518ea9aec",
                    "sha1": "659661291eb5fd6452d6cabdc24cd9fbc1fb17f7",
                    "md5": "4a15ed0feb9e90b56e82c2e45a3b3f5e"
                },
                "tags": [
                    "SnakeKeylogger"
                ],
                "valid_from": "2024-11-22T07:30:07.000Z",
                "valid_until": "2025-11-22T07:30:07.000Z",
                "created": "2024-11-22T07:37:40.729Z"
            }
        ]
    }
}

usta-tsa-search-malicious-url


Search malicious URL indicators from the feed.

Base Command

usta-tsa-search-malicious-url

Input

Argument Name Description Required
limit The maximum number of results to return. Default is 10. Optional
url The URL to search for. Required

Context Output

Path Type Description
USTA.ThreatStreamMaliciousUrls.id String The ID of the alert
USTA.ThreatStreamMaliciousUrls.url String The URL of the malicious site
USTA.ThreatStreamMaliciousUrls.is_domain Boolean Whether the malicious site is a domain
USTA.ThreatStreamMaliciousUrls.ip_addresses Array The IP addresses of the malicious site
USTA.ThreatStreamMaliciousUrls.tags Array The tags of the malicious site
USTA.ThreatStreamMaliciousUrls.created Date The creation date of the malicious site
USTA.ThreatStreamMaliciousUrls.valid_from Date The valid from date of the malicious site
USTA.ThreatStreamMaliciousUrls.valid_until Date The valid until date of the malicious site

Command Example

!usta-tsa-search-malicious-url url=http://192.168.100.1:38082/i

Context Example


{
    "USTA" :{
        "ThreatStreamMaliciousUrls":[
             {
                "id": "28cffb9a-add5-480c-8968-539863695770",
                "url": "http://192.168.100.1:38082/i",
                "host": "192.168.100.1",
                "is_domain": false,
                "ip_addresses": [
                    "192.168.100.1"
                ],
                "tags": [
                    "elf.mozi"
                ],
                "valid_from": "2024-11-22T07:24:06.000Z",
                "valid_until": "2025-11-22T07:24:06.000Z",
                "created": "2024-11-22T08:30:03.055Z"
            }
        ]
    }
}

usta-tsa-search-phishing-site


Search malicious URL indicators from the feed.

Base Command

usta-tsa-search-phishing-site

Input

Argument Name Description Required
limit The maximum number of results to return. Default is 10. Optional
url The URL to search for. Required

Context Output

Path Type Description
USTA.ThreatStreamPhishingSites.id String The ID of the alert
USTA.ThreatStreamPhishingSites.url String The URL of the phishing site
USTA.ThreatStreamPhishingSites.is_domain Boolean Whether the phishing site is a domain
USTA.ThreatStreamPhishingSites.ip_addresses Array The IP addresses of the phishing site
USTA.ThreatStreamPhishingSites.created Date The creation date of the phishing site

Command Example

!usta-tsa-search-phishing-site url=example.com

Context Example

{
    "USTA" :{
        "ThreatStreamMaliciousUrls":[
            {
                "id": 219286,
                "url": "https://example.com",
                "host": "example.com",
                "is_domain": true,
                "ip_addresses": [
                    "192.168.100.1"
                ],
                "country": null,
                "created": "2024-02-05T15:23:11.646011Z"
            }
        ]
    }
}

Configuration parameters

  • feed — Fetch indicators
  • url — Server's URL (required)
  • api_key — API Key (required)
  • ioc_feed_type — IOC Feed Type (required)
  • feedReputation — Indicator Reputation
  • feedReliability — Source Reliability (required)
  • tlp_color — Traffic Light Protocol Color
  • feedFetchInterval — Feed Fetch Interval
  • feedBypassExclusionList — Bypass exclusion list
  • insecure — Trust any certificate (not secure)
  • proxy — Use system proxy settings
  • feedExpirationPolicy
  • feedExpirationInterval
  • feedTags — Tags

Commands (3)

  • usta-tsa-search-malicious-url

    Search malicious URL indicators from the feed.

  • usta-tsa-search-malware-hash

    Search malware hash indicators from the feed.

  • usta-tsa-search-phishing-site

    Search malicious URL indicators from the feed.

import json

import demistomock as demisto  # noqa: F401
import pytest
from CommonServerPython import tableToMarkdown
from FeedUstaThreatStream import (
    Client,
    check_module,
    fetch_indicators_command,
    main,
    parse_malicious_urls,
    parse_malware_hashes,
    parse_phishing_sites,
    search_malicious_urls_command,
    search_malware_hashes_command,
    search_phishing_site_command,
)

client = Client(base_url="", verify=False, headers={}, proxy=False)


def util_load_json(path):
    with open(path, encoding="utf-8") as f:
        return json.loads(f.read())


def test_check_module(mocker):
    mock_response = util_load_json("test_data/auth_success_response.json")
    mocker.patch.object(client, "check_auth", return_value=mock_response)
    response = check_module(client)
    assert response == "ok"


def test_search_phishing_site_command(mocker):
    mock_response = json.loads(
        """
    {
        "count": 1,
        "next": null,
        "previous": null,
        "results": [
            {
                "id": 219286,
                "url": "https://phishingwebsite.org",
                "host": "phishingwebsite.org",
                "is_domain": true,
                "ip_addresses": [
                    "127.0.0.1"
                ],
                "country": null,
                "created": "2024-02-05T15:23:11.646011Z"
            }
        ]
    }"""
    )

    mocker.patch.object(client, "search_iterator_without_pagination", return_value=mock_response)
    args = {"limit": "10", "offset": "0", "indicator": "example.com"}
    response = search_phishing_site_command(client, args)
    human_readable = tableToMarkdown("Indicators from USTA Feed (phishing-sites):", mock_response["results"])
    assert response.raw_response == mock_response["results"]
    assert response.readable_output == human_readable
    assert len(response.raw_response) == 1


@pytest.mark.parametrize(
    "command, expected_output",
    [
        (search_phishing_site_command, "No entries."),
        (search_malware_hashes_command, "No entries."),
        (search_malicious_urls_command, "No entries."),
    ],
)
def test_search_commands_no_results_found(mocker, command, expected_output):
    mock_response = json.loads(
        """
        {
            "count": 0,
            "next": null,
            "previous": null,
            "results": []
        }"""
    )
    mocker.patch.object(client, "search_iterator_without_pagination", return_value=mock_response)
    args = {"limit": "10", "offset": "0", "indicator": "exampleurl.com"}
    response = command(client, args)
    assert response.raw_response == mock_response["results"]
    assert len(response.raw_response) == 0
    assert expected_output in response.readable_output


@pytest.mark.parametrize(
    "command, expected_output",
    [
        (search_phishing_site_command, "Indicators from USTA Feed (phishing-sites):"),
        (search_malware_hashes_command, "Indicators from USTA Feed (malware-hashes):"),
        (search_malicious_urls_command, "Indicators from USTA Feed (malicious-urls):"),
    ],
)
def test_search_commands_with_results(mocker, command, expected_output):
    mock_response = json.loads(
        """
        {
            "count": 1,
            "next": null,
            "previous": null,
            "results": [
                {
                    "id": 219286,
                    "url": "https://example.com",
                    "host": "example.com",
                    "is_domain": true,
                    "ip_addresses": [
                        "127.0.0.1"
                    ],
                    "country": null,
                    "created": "2024-02-05T15:23:11.646011Z"
                }
            ]
        }"""
    )
    mocker.patch.object(client, "search_iterator_without_pagination", return_value=mock_response)
    args = {"limit": "10", "offset": "0", "indicator": "example.com"}
    response = command(client, args)
    assert response.raw_response == mock_response["results"]
    assert len(response.raw_response) == 1
    assert expected_output in response.readable_output


@pytest.mark.parametrize(
    "parser_function, indicator_json",
    [
        (parse_phishing_sites, "test_data/fetch_indicators_phishing_sites.json"),
        (parse_malicious_urls, "test_data/fetch_indicators_malicious_urls.json"),
        (parse_malware_hashes, "test_data/fetch_indicators_malware_hashes.json"),
    ],
)
def test_parse_functions(mocker, parser_function, indicator_json):
    mock_response = util_load_json(indicator_json)
    for each in mock_response:
        expected_result = parser_function(each["rawJSON"])
        assert expected_result["rawJSON"] == each["rawJSON"]
        assert expected_result["value"] == each["value"]


@pytest.mark.parametrize(
    "ioc_type, indicator_json",
    [
        ("phishing-sites", "test_data/fetch_indicators_phishing_sites.json"),
        ("malicious-urls", "test_data/fetch_indicators_malicious_urls.json"),
        ("malware-hashes", "test_data/fetch_indicators_malware_hashes.json"),
    ],
)
def test_fetch_indicators_command(mocker, ioc_type, indicator_json):
    all_indicators = []
    for each in util_load_json(indicator_json):
        all_indicators.append(each["rawJSON"])

    mocker.patch.object(Client, "build_iterator", return_value=all_indicators)
    params = {
        "ioc_feed_type": ioc_type,
    }
    next_run, indicators = fetch_indicators_command(client=Client, last_run={}, params=params)

    assert len(indicators) == len(all_indicators)
    assert next_run[ioc_type]["created"] == all_indicators[0]["created"]


def test_fetch_indicators_command_no_results(mocker):
    mocker.patch.object(Client, "build_iterator", return_value=[])
    params = {
        "ioc_feed_type": "phishing-sites",
    }
    next_run, indicators = fetch_indicators_command(client=Client, last_run={}, params=params)

    assert len(indicators) == 0
    assert next_run == {}


def test_main_cmd_fetch_indicators(mocker):
    mocker.patch.object(
        demisto,
        "params",
        return_value={
            "url": "https://example.com",
            "api_key": "API_KEY",
            "insecure": True,
            "proxy": False,
            "first_fetch": "3 days",
            "status": "open",
            "max_fetch": 50,
        },
    )

    Client(base_url="", verify=False, headers={}, proxy=False)
    mocker.patch.object(client, "build_iterator", return_value=[{}])
    mocker.patch.object(demisto, "command", return_value="fetch-indicators")
    mocker.patch.object(demisto, "createIndicators")
    mocker.patch.object(demisto, "setIntegrationContext")
    mocker.patch.object(demisto, "setLastRun")
    main()
    demisto.setLastRun.assert_called_once()