CyrenThreatInDepth

Threat InDepth's actionable and contextualized intelligence helps enterprises improve their threat detection and response by providing unprecedented visibility into new email-borne security threats faster than other security vendors.

Data Enrichment & Threat Intelligence · Cyren Threat InDepth Threat Intelligence · Feed

Details

ID	`CyrenThreatInDepth`
Provider	Communitake
Category	Data Enrichment & Threat Intelligence
From Version	6.0.0
Docker Image	`demisto/python3:3.12.13.10116658`
Supported Modules	Agentix XSIAM

README

Threat InDepth’s correlated and contextualized intelligence helps enterprises improve their threat detection and response by providing unprecedented visibility into new email-borne security threats and actionable insights to make meaningful response decisions. By correlating insights gathered across email content, web traffic, and suspicious files; Cyren provides security teams with a multi-dimensional presentation of critical threat characteristics.

Benefits include

Access to Cyren’s GlobalView™ Threat Intelligence Cloud that provides the earliest visibility into new and evolving attacks on a global basis
Comprehensive, multi-dimensional presentation of critical threat characteristics to help analysts understand the evolving threat landscape
Timely, Correlated, & Contextualized intelligence that helps reduce mean-time-to-detect (MTTD) and mean-time-to-respond (MTTR) for security analysts
Improved threat detection for existing security products such as SIEM and SOAR solutions

Feeds included in Cyren Threat InDepth content pack

The Cyren Threat InDepth content pack includes access to these streams of indicators:

IP Reputation Intelligence
Phishing & Fraud URL Intelligence
Malware URL Intelligence
Malware File Intelligence

Configure Cyren Threat InDepth Threat Intelligence Feed In Cortex

Parameter	Description	Required
apikey	API JWT token that has been issued to you	True
feed_name	Name of the particular feed that matches your API JWT token	True
max_indicators	The maximum number of indicators to fetch	False
feed	Fetch indicators.	False
feedIncremental	Is incremental or not	False
feedReputation	The reputation to apply to the fetched indicators.	False
feedReliability	The reliability of the this feed.	True
tlp_color	The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed. More information about the protocol can be found at https://us-cert.cisa.gov/tlp	False
feedExpirationPolicy		False
feedExpirationInterval		False
feedFetchInterval	Feed Fetch Interval	False
feedBypassExclusionList	Bypass exclusion list	False

The underlying Cyren Threat InDepth API provides you with an incremental feed, meaning it provides new
or modified indicators. It also works with an offset value that keeps track of your currently processed
indicators. Your current offset defaults at the globally known maximum offset on your first setup and
is being stored and updated for you in the integration instance context. The integration then uses the
“Maximum number of indicators” parameter as the count in each request. It is recommended to set it to
a high enough value so that you get all the feed indicators for maximum product value, to handle bursts
etc. (the value cannot be higher than 100.000 and it will be capped at that value if you set a higher one).

In case you want to want to reset the offset value, use the cyren-threat-indepth-reset-client-offset command.

You can retrieve the current offset value using the cyren-threat-indepth-get-client-offset command.

Commands

You can execute these commands from the XSOAR CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

Fetch indicators

Fetching Cyren Threat InDepth indicators

Required Permissions

A valid API JWT token and a matching feed name

Base Command

cyren-threat-indepth-get-indicators

Input

Argument Name	Description	Required
max_indicators	The maximum number of results to return.	True

Context Output

There is no context output for this command.

Command Example

!cyren-threat-indepth-get-indicators max_indicators=2

Human Readable Output

Indicators from Cyren Threat InDepth:

value	type	rawJSON	score
http://nu4vs0m.u5jkzm4r.i2wd30t.bpbp9c7d.b7ni2cio.auz8x15h.freshoff.eu	URL	payload: {"action": "+", "type": "url", "identifier": "f59ef036-a790-5193-b942-24a8618c936a", "first_seen": "2020-10-25T13:41:36.000Z", "last_seen": "2021-01-05T13:54:41.000Z", "detection": {"category": ["phishing"], "detection_ts": "2020-10-25T13:41:36.000Z"}, "meta": {"port": 80, "protocol": "http"}, "relationships": [{"relationship_type": "resolves to", "relationship_ts": "2020-10-25T13:41:36.000Z", "ip": "217.70.142.108", "related_entity_category": "phishing", "relationship_description": "resolves to phishing ip"}], "detection_methods": ["URL Categorization"], "url": "http://nu4vs0m.u5jkzm4r.i2wd30t.bpbp9c7d.b7ni2cio.auz8x15h.freshoff.eu"} offset: 57006380 timestamp: 2021-01-05T14:00:48.919Z	3

Reset Client Offset

This command allows you to update the stored client offset for the feed API.

Required Permissions

A valid API JWT token and a matching feed name

Base Command

cyren-threat-indepth-reset-client-offset

Input

Argument Name	Description	Required
offset	Optional The offset you want to use as your baseline for future fetches (if not provided, the global max offset from the API is used)	False

Context Output

There is no context output for this command.

Command Example

!cyren-threat-indepth-reset-client-offset
!cyren-threat-indepth-reset-client-offset offset = 34234234

Human Readable Output

Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 (API provided max offset of 1000, was 500).

Get Client Offset

This command allows you to retrieve the stored client offset for the feed API.

Required Permissions

A valid API JWT token and a matching feed name

Base Command

cyren-threat-indepth-get-client-offset

Input

There is not input for this command.

Context Output

There is no context output for this command.

Command Example

!cyren-threat-indepth-get-client-offset

Human Readable Output

Cyren Threat InDepth ip_reputation feed client offset is 500 (API provided max offset of 1000).

Additional Information

Configuration parameters

feedReputation — Indicator Reputation
feedReliability — Source Reliability (required)
feedExpirationPolicy —
feedExpirationInterval —
feedIncremental — Incremental Feed
feedFetchInterval — Feed Fetch Interval
feedBypassExclusionList — Bypass exclusion list
tlp_color — Traffic Light Protocol Color
feedTags — Tags
feed — Fetch indicators
apikey — API Token (required)
feed_name — Feed Name (required)
max_indicators — Maximum number of indicators
insecure — Trust any certificate (not secure)
proxy — Use system proxy settings

Commands (3)

cyren-threat-indepth-get-client-offset
Prints the max client offset stored in the integration context.
cyren-threat-indepth-get-indicators
Fetching Cyren Threat InDepth indicators
cyren-threat-indepth-reset-client-offset
Resets the max client offset stored in the integration context.

Source

CyrenThreatInDepth.py CyrenThreatInDepth_test.py CyrenThreatInDepth.yml CyrenThreatInDepth_description.md

Download History Raw

import os
import pathlib

import pytest
from CommonServerPython import Common, DemistoException, FeedIndicatorType, get_integration_context, set_integration_context
from CyrenThreatInDepth import (
    BASE_URL,
    Client,
    fetch_indicators_command,
    get_indicators_command,
    get_offset_command,
    reset_offset_command,
)
from CyrenThreatInDepth import test_module_command as _test_module_command

pytestmark = pytest.mark.usefixtures("clean_integration_context")

API_TOKEN = "12345"
VERSION = "1.5.0"


def _load_file(file_name):
    full_path = os.path.join(pathlib.Path(__file__).parent.absolute(), "test_data", file_name)
    with open(full_path) as f:
        return f.read()


@pytest.fixture(name="malware_files", scope="session")
def fixture_malware_files():
    return _load_file("malware_files.jsonl")


@pytest.fixture(name="malware_urls", scope="session")
def fixture_malware_urls():
    return _load_file("malware_urls.jsonl")


@pytest.fixture(name="phishing_urls", scope="session")
def fixture_phishing_urls():
    return _load_file("phishing_urls.jsonl")


@pytest.fixture(name="ip_reputation", scope="session")
def fixture_ip_reputation():
    return _load_file("ip_reputation.jsonl")


@pytest.fixture(name="response_429", scope="session")
def fixture_response_429():
    return _load_file("429.html")


@pytest.fixture(name="clean_integration_context", scope="function")
def fixture_clean_integration_context():
    set_integration_context({})
    yield
    set_integration_context({})


def _create_client(feed):
    return Client(feed_name=feed, api_token=API_TOKEN, base_url=BASE_URL, verify=False, proxy=False)


def _expected_headers():
    return {
        "Authorization": f"Bearer {API_TOKEN}",
        "Cyren-Client-Name": "Palo Alto Cortex XSOAR",
        "Cyren-Client-Version": VERSION,
    }


def _create_instance(requests_mock, feed, feed_data, offset_data, offset=0, count=2):
    expected_headers = _expected_headers()
    requests_mock.get(
        BASE_URL + f"/data?format=jsonl&feedId={feed}_v2&offset={offset}&count={count}",
        text=feed_data,
        request_headers=expected_headers,
    )
    requests_mock.get(BASE_URL + f"/info?format=jsonl&feedId={feed}_v2", json=offset_data, request_headers=expected_headers)
    client = _create_client(feed)

    def fetch_command(initial_count=0, max_indicators=2, update_context=False):
        return fetch_indicators_command(client, initial_count, max_indicators, update_context)

    def get_command(max_indicators):
        args = {"max_indicators": max_indicators}
        return get_indicators_command(client, args)

    return fetch_command, get_command


@pytest.mark.parametrize(
    "context_data, offsets, initial_count, max_indicators, expected_offset, expected_count",
    [
        # Given:
        #   - first run
        #   - no previous offset stored
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with no initial import wanted and wanting 2
        # Then:
        #   - the API is asked for 2 from offset 10000
        ({}, {"startOffset": 1, "endOffset": 10000}, 0, 2, 10000, 2),
        # Given:
        #   - first run
        #   - no previous offset stored
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with initial import of 1000 and wanting 2
        # Then:
        #   - the API is asked for 1002 from offset 9001
        ({}, {"startOffset": 1, "endOffset": 10000}, 1000, 2, 9001, 1002),
        # Given:
        #   - not the first run
        #   - previous offset of 9001
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with no initial import wanted and wanting 2
        # Then:
        #   - the API is asked for 2 from offset 9001
        ({"offset": 9001}, {"startOffset": 1, "endOffset": 10000}, 0, 2, 9001, 2),
        # Given:
        #   - not the first run
        #   - previous offset of 9001
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with initial import of 10000 and wanting 2
        # Then:
        #   - the API is asked for 2 from offset 9001
        ({"offset": 9001}, {"startOffset": 1, "endOffset": 10000}, 10000, 2, 9001, 2),
        # Given:
        #   - not the first run
        #   - previous offset of 9001
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with no initial import wanted and wanting 2000
        # Then:
        #   - the API is asked for 2000 from offset 9001
        ({"offset": 9001}, {"startOffset": 1, "endOffset": 10000}, 0, 2000, 9001, 2000),
        # Given:
        #   - not the first run
        #   - previous offset of 9001
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with initial import of 10000 and wanting 2000
        # Then:
        #   - the API is asked for 2000 from offset 9001
        ({"offset": 9001}, {"startOffset": 1, "endOffset": 10000}, 10000, 2000, 9001, 2000),
        # Given:
        #   - first run
        #   - no previous offset stored
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with no initial import wanted and wanting 100001
        # Then:
        #   - the API is asked for a max of 100000 from offset 10000
        ({}, {"startOffset": 1, "endOffset": 10000}, 0, 100001, 10000, 100000),
        # Given:
        #   - not the first run
        #   - previous offset of 9001
        #   - an end offset of 10000
        # When:
        #   - running fetch-indicators with no initial import wanted and wanting 100001
        # Then:
        #   - the API is asked for a max of 100000 from offset 9001
        ({"offset": 9001}, {"startOffset": 1, "endOffset": 10000}, 0, 100001, 9001, 100000),
    ],
)
def test_fetch_indicators_offsets(
    requests_mock, ip_reputation, context_data, offsets, initial_count, max_indicators, expected_offset, expected_count
):
    """
    Given:
        - the IP reputation feed

    When:
        - running fetch-indicators

    Then:
        - the new offset in the integration context is the max offset from the
          entries + 1
        - the number of imported indicators is the number of IP's in the feed

    """

    set_integration_context(context_data)
    fetch, _ = _create_instance(requests_mock, "ip_reputation", ip_reputation, offsets, expected_offset, expected_count)
    created = fetch(initial_count, max_indicators, True)

    assert len(created) == 8
    assert get_integration_context() == {"offset": 50007}


def test_fetch_indicators_parsing_errors(requests_mock, ip_reputation):
    """
    Given:
        - the IP reputation feed

    When:
        - running fetch-indicators
        - some non-JSON lines in the response

    Then:
        - still imported the number of good JSON lines in the response

    """

    ip_reputation_with_errors = f"\nbla\n{ip_reputation}\n\nno json, too\n"
    fetch, _ = _create_instance(requests_mock, "ip_reputation", ip_reputation_with_errors, {"startOffset": 0, "endOffset": 0})
    created = fetch()

    assert len(created) == 8


def test_fetch_indicators_rate_limiting(requests_mock, response_429):
    """
    Given:
        - the IP reputation feed

    When:
        - running fetch-indicators
        - a 429 Rate Limited response from the API

    Then:
        - a DemistoException is raised

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10",
        request_headers=_expected_headers(),
        text=response_429,
        status_code=429,
    )
    requests_mock.get(
        BASE_URL + "/info?format=jsonl&feedId=ip_reputation_v2",
        json={"startOffset": 0, "endOffset": 0},
        request_headers=_expected_headers(),
    )
    client = _create_client("ip_reputation")

    with pytest.raises(DemistoException, match=f".*{response_429}.*"):
        fetch_indicators_command(client, 0, 10, False)


def test_fetch_indicators_output_ip_reputation(requests_mock, ip_reputation):
    """
    Given:
        - the IP reputation feed
        - no relationship information in the feed

    When:
        - running fetch-indicators

    Then:
        - the indicator type and value are being set
        - the DBot score is set to
          - BAD on spam category
          - SUSPICIOUS on malware, phishing category
          - NONE on feed removal and confirmed clean category
        - basic indicator fields are filled from the feed meta data
        - Cyren-specific indicator fields are filled

    """

    fetch, _ = _create_instance(requests_mock, "ip_reputation", ip_reputation, {"startOffset": 0, "endOffset": 0})
    created = fetch()

    assert len(created) == 8

    assert created[0]["fields"] == {"updateddate": "2020-10-29T05:15:29.062Z", "indicatoridentification": "45.193.212.54"}
    assert created[0]["score"] == Common.DBotScore.SUSPICIOUS
    assert created[0]["rawJSON"]["tags"] == ["spam", "Botnet detection"]
    assert created[0]["rawJSON"]["source_tag"] == "primary"
    assert created[0]["type"] == FeedIndicatorType.IP
    assert created[0]["value"] == "45.193.212.54"

    assert created[1]["fields"] == {"updateddate": "2020-10-29T05:15:29.062Z", "indicatoridentification": "45.193.216.182"}
    assert created[1]["score"] == Common.DBotScore.SUSPICIOUS
    assert created[1]["rawJSON"]["tags"] == ["malware", "Botnet detection"]
    assert created[1]["rawJSON"]["source_tag"] == "primary"
    assert created[1]["type"] == FeedIndicatorType.IP
    assert created[1]["value"] == "45.193.216.182"

    assert created[2]["fields"] == {
        "updateddate": "2020-10-29T05:15:29.062Z",
        "published": "2020-10-29T05:15:29.062Z",
        "indicatoridentification": "45.193.216.183",  # noqa: E501
    }
    assert created[2]["score"] == Common.DBotScore.SUSPICIOUS
    assert created[2]["rawJSON"]["tags"] == ["phishing", "Botnet detection"]
    assert created[2]["rawJSON"]["source_tag"] == "primary"
    assert created[2]["type"] == FeedIndicatorType.IP
    assert created[2]["value"] == "45.193.216.183"

    assert created[3]["fields"] == {"updateddate": "2020-10-29T05:15:29.062Z", "indicatoridentification": "45.193.216.184"}
    assert created[3]["score"] == Common.DBotScore.NONE
    assert created[3]["rawJSON"]["tags"] == ["spam", "Botnet detection"]
    assert created[3]["rawJSON"]["source_tag"] == "primary"
    assert created[3]["type"] == FeedIndicatorType.IP
    assert created[3]["value"] == "45.193.216.184"

    assert created[4]["fields"] == {"updateddate": "2020-10-29T05:15:29.062Z", "indicatoridentification": "45.193.216.185"}
    assert created[4]["score"] == Common.DBotScore.NONE
    assert created[4]["rawJSON"]["tags"] == ["confirmed clean", "Botnet detection"]
    assert created[4]["rawJSON"]["source_tag"] == "primary"
    assert created[4]["type"] == FeedIndicatorType.IP
    assert created[4]["value"] == "45.193.216.185"

    assert created[5]["fields"] == {
        "updateddate": "2020-10-29T05:15:29.062Z",
        "published": "2020-10-29T05:15:29.062Z",
        "indicatoridentification": "45.193.212.55",  # noqa: E501
    }
    assert created[5]["score"] == Common.DBotScore.SUSPICIOUS
    assert created[5]["rawJSON"]["tags"] == ["spam", "Botnet detection"]
    assert created[5]["rawJSON"]["source_tag"] == "primary"
    assert created[5]["type"] == FeedIndicatorType.IP
    assert created[5]["value"] == "45.193.212.55"

    assert created[6]["fields"] == {"updateddate": "2020-10-29T05:15:29.062Z", "indicatoridentification": "45.193.212.56"}
    assert created[6]["score"] == Common.DBotScore.BAD
    assert created[6]["rawJSON"]["tags"] == ["spam", "Botnet detection"]
    assert created[6]["rawJSON"]["source_tag"] == "primary"
    assert created[6]["type"] == FeedIndicatorType.IP
    assert created[6]["value"] == "45.193.212.56"

    assert created[7]["fields"] == {
        "updateddate": "2020-10-29T05:15:29.062Z",
        "published": "2020-10-29T05:15:29.062Z",
        "indicatoridentification": "45.193.212.57",  # noqa: E501
    }
    assert created[7]["score"] == Common.DBotScore.BAD
    assert created[7]["rawJSON"]["tags"] == ["spam", "Botnet detection"]
    assert created[7]["rawJSON"]["source_tag"] == "primary"
    assert created[7]["type"] == FeedIndicatorType.IP
    assert created[7]["value"] == "45.193.212.57"


def test_fetch_indicators_output_malware_files(requests_mock, malware_files):
    """
    Given:
        - the malware file feed
        - some relationship information in the feed

    When:
        - running fetch-indicators

    Then:
        - the indicator type and value are being set
        - the DBot score is set to
          - BAD on non confirmed-clean category
          - NONE on feed removal and confirmed clean category
        - basic indicator fields are filled from the feed meta data
        - Cyren-specific indicator fields are filled
        - feed related indicator field is filled with IP and SHA-256 relationships

    """

    fetch, _ = _create_instance(requests_mock, "malware_files", malware_files, {"startOffset": 0, "endOffset": 0})
    created = fetch()

    assert len(created) == 8

    assert created[0]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "SHA-256",
                "relationshiptype": "downloaded from",
                "value": "0f6dbfb291ba1b84601b0372f70db3430df636c631d074c1c2463f9e5a033f21",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T14:42:14.000Z",
                "entitycategory": "malware",
            }
        ]
    }
    assert created[0]["score"] == Common.DBotScore.NONE
    assert created[0]["rawJSON"]["source_tag"] == "related"
    assert created[0]["type"] == FeedIndicatorType.IP
    assert created[0]["value"] == "172.217.4.65"

    assert created[1]["fields"] == {
        "updateddate": "2020-10-28T14:45:24.921Z",
        "published": "2020-10-28T14:45:24.921Z",
        "indicatoridentification": ("0f6dbfb291ba1b84601b0372f70db3430df636c631d074c1c2463f9e5a033f21"),
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "downloaded from",
                "value": "172.217.4.65",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T14:42:14.000Z",
                "entitycategory": "malware",
            }
        ],
    }
    assert created[1]["score"] == Common.DBotScore.BAD
    assert created[1]["rawJSON"]["tags"] == ["malware", "Malware detection", "js/clickjack.d"]
    assert created[1]["rawJSON"]["source_tag"] == "primary"
    assert created[1]["type"] == FeedIndicatorType.File
    assert created[1]["value"] == "0f6dbfb291ba1b84601b0372f70db3430df636c631d074c1c2463f9e5a033f21"

    assert created[2]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "SHA-256",
                "relationshiptype": "downloaded from",
                "value": "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0a",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T11:50:21.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[2]["score"] == Common.DBotScore.NONE
    assert created[2]["rawJSON"]["source_tag"] == "related"
    assert created[2]["type"] == FeedIndicatorType.IP
    assert created[2]["value"] == "62.149.142.116"

    assert created[3]["fields"] == {
        "updateddate": "2020-10-28T14:45:24.921Z",
        "published": "2020-10-28T14:45:24.921Z",
        "indicatoridentification": "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0a",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "downloaded from",
                "value": "62.149.142.116",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T11:50:21.000Z",
                "entitycategory": "malware",
            }
        ],
    }

    assert created[3]["score"] == Common.DBotScore.BAD
    assert created[3]["rawJSON"]["tags"] == ["malware", "Malware detection", "js/coinhive.a!eldorado"]
    assert created[3]["rawJSON"]["source_tag"] == "primary"
    assert created[3]["type"] == FeedIndicatorType.File
    assert created[3]["value"] == "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0a"

    assert created[4]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "SHA-256",
                "relationshiptype": "downloaded from",
                "value": "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0b",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T11:50:21.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[4]["score"] == Common.DBotScore.NONE
    assert created[4]["rawJSON"]["source_tag"] == "related"
    assert created[4]["type"] == FeedIndicatorType.IP
    assert created[4]["value"] == "62.149.142.116"

    assert created[5]["fields"] == {
        "updateddate": "2020-10-28T14:45:24.921Z",
        "indicatoridentification": ("243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0b"),
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "downloaded from",
                "value": "62.149.142.116",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T11:50:21.000Z",
                "entitycategory": "malware",
            }
        ],
    }

    assert created[5]["score"] == Common.DBotScore.NONE
    assert created[5]["rawJSON"]["tags"] == ["confirmed clean", "Malware detection", "js/coinhive.a!eldorado"]
    assert created[5]["rawJSON"]["source_tag"] == "primary"
    assert created[5]["type"] == FeedIndicatorType.File
    assert created[5]["value"] == "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0b"

    assert created[6]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "SHA-256",
                "relationshiptype": "downloaded from",
                "value": "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0c",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T11:50:21.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[6]["score"] == Common.DBotScore.NONE
    assert created[6]["rawJSON"]["source_tag"] == "related"
    assert created[6]["type"] == FeedIndicatorType.IP
    assert created[6]["value"] == "62.149.142.116"

    assert created[7]["fields"] == {
        "updateddate": "2020-10-28T14:45:24.921Z",
        "indicatoridentification": ("243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0c"),
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "downloaded from",
                "value": "62.149.142.116",
                "description": "downloaded from malware ip",
                "timestamp": "2020-10-28T11:50:21.000Z",
                "entitycategory": "malware",
            }
        ],
    }

    assert created[7]["score"] == Common.DBotScore.NONE
    assert created[7]["rawJSON"]["tags"] == ["malware", "Malware detection", "js/coinhive.a!eldorado"]
    assert created[7]["rawJSON"]["source_tag"] == "primary"
    assert created[7]["type"] == FeedIndicatorType.File
    assert created[7]["value"] == "243f68c5fffe1e868c012b7fcf20bd8c9025ec199b18d569a497a2e3f1aaca0c"


def test_fetch_indicators_output_malware_urls(requests_mock, malware_urls):
    """
    Given:
        - the malware URL feed
        - some relationship information in the feed

    When:
        - running fetch-indicators

    Then:
        - the indicator type and value are being set
        - the DBot score is set to
          - BAD on non confirmed-clean category
          - NONE on feed removal and confirmed clean category
        - basic indicator fields are filled from the feed meta data
        - Cyren-specific indicator fields are filled
        - feed related indicator field is filled with IP and SHA-256 relationships

    """

    fetch, _ = _create_instance(requests_mock, "malware_urls", malware_urls, {"startOffset": 0, "endOffset": 0})
    created = fetch()

    assert len(created) == 8

    assert created[0]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "resolves to",
                "value": "http://radiobarreradigitall.blogspot.com",
                "description": "resolves to malware ip",
                "timestamp": "2020-11-01T16:20:57.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[0]["score"] == Common.DBotScore.NONE
    assert created[0]["type"] == FeedIndicatorType.IP
    assert created[0]["value"] == "172.217.4.65"

    assert created[1]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "serves",
                "value": "http://radiobarreradigitall.blogspot.com",
                "description": "serves malware file",
                "timestamp": "2020-11-01T16:11:54.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[1]["score"] == Common.DBotScore.BAD
    assert created[1]["rawJSON"]["source_tag"] == "related"
    assert created[1]["type"] == FeedIndicatorType.File
    assert created[1]["value"] == "a18c43948195abd429ba42ef66b26483a097d987e55289010bc8f935fc950515"

    assert created[2]["fields"] == {
        "indicatoridentification": "045541ea-fd19-5c08-bb60-437ce08cc08f",
        "updateddate": "2020-11-01T17:45:16.268Z",
        "published": "2020-11-01T17:45:16.268Z",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "resolves to",
                "value": "172.217.4.65",
                "description": "resolves to malware ip",
                "timestamp": "2020-11-01T16:20:57.000Z",
                "entitycategory": "malware",
            },
            {
                "indicatortype": "SHA-256",
                "relationshiptype": "serves",
                "value": "a18c43948195abd429ba42ef66b26483a097d987e55289010bc8f935fc950515",
                "description": "serves malware file",
                "timestamp": "2020-11-01T16:11:54.000Z",
                "entitycategory": "malware",
            },
        ],
    }

    assert created[2]["score"] == Common.DBotScore.BAD
    assert created[2]["rawJSON"]["tags"] == ["malware", "Malware detection", "finance"]
    assert created[2]["rawJSON"]["source_tag"] == "primary"
    assert created[2]["type"] == FeedIndicatorType.URL
    assert created[2]["value"] == "http://radiobarreradigitall.blogspot.com"

    assert created[3]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "resolves to",
                "value": "https://wizkhalifanoticias.blogspot.com/2014/01/wiz-khalifa-adormece-durante.html",
                "description": "resolves to malware ip",
                "timestamp": "2020-11-01T17:39:16.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[3]["score"] == Common.DBotScore.NONE
    assert created[3]["rawJSON"]["source_tag"] == "related"
    assert created[3]["type"] == FeedIndicatorType.IP
    assert created[3]["value"] == "172.217.4.193"

    assert created[4]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "serves",
                "value": "https://wizkhalifanoticias.blogspot.com/2014/01/wiz-khalifa-adormece-durante.html",
                "description": "serves malware file",
                "timestamp": "2020-11-01T17:39:10.000Z",
                "entitycategory": "malware",
            }
        ]
    }

    assert created[4]["score"] == Common.DBotScore.BAD
    assert created[4]["rawJSON"]["source_tag"] == "related"
    assert created[4]["type"] == FeedIndicatorType.File
    assert created[4]["value"] == "2bbeeaa4139b8e033fc1e114f55917e7180b305e75ac56701a0b6dcda4495494"

    assert created[5]["fields"] == {
        "indicatoridentification": "05040e64-a035-5014-8564-9c8faaf4da83",
        "updateddate": "2020-11-01T17:45:16.268Z",
        "published": "2020-11-01T17:45:16.268Z",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "resolves to",
                "value": "172.217.4.193",
                "description": "resolves to malware ip",
                "timestamp": "2020-11-01T17:39:16.000Z",
                "entitycategory": "malware",
            },
            {
                "indicatortype": "SHA-256",
                "relationshiptype": "serves",
                "value": "2bbeeaa4139b8e033fc1e114f55917e7180b305e75ac56701a0b6dcda4495494",
                "description": "serves malware file",
                "timestamp": "2020-11-01T17:39:10.000Z",
                "entitycategory": "malware",
            },
        ],
    }

    assert created[5]["score"] == Common.DBotScore.BAD
    assert created[5]["rawJSON"]["tags"] == ["malware", "Malware detection"]
    assert created[5]["rawJSON"]["source_tag"] == "primary"
    assert created[5]["type"] == FeedIndicatorType.URL
    assert created[5]["value"] == "https://wizkhalifanoticias.blogspot.com/2014/01/wiz-khalifa-adormece-durante.html"

    assert created[6]["fields"] == {
        "indicatoridentification": "05040e64-a035-5014-8564-9c8faaf4da84",
        "updateddate": "2020-11-01T17:45:16.268Z",
    }
    assert created[6]["score"] == Common.DBotScore.NONE
    assert created[6]["rawJSON"]["tags"] == ["confirmed clean", "Malware detection"]
    assert created[6]["rawJSON"]["source_tag"] == "primary"
    assert created[6]["type"] == FeedIndicatorType.URL
    assert created[6]["value"] == "https://wizkhalifanoticias.blogspot.com/2014/01/wiz-khalifa-adormece-durante-2.html"

    assert created[7]["fields"] == {
        "updateddate": "2020-11-01T17:45:16.268Z",
        "indicatoridentification": "05040e64-a035-5014-8564-9c8faaf4da85",
    }
    assert created[7]["score"] == Common.DBotScore.NONE
    assert created[7]["rawJSON"]["tags"] == ["malware", "Malware detection"]
    assert created[7]["rawJSON"]["source_tag"] == "primary"
    assert created[7]["type"] == FeedIndicatorType.URL
    assert created[7]["value"] == "https://wizkhalifanoticias.blogspot.com/2014/01/wiz-khalifa-adormece-durante-3.html"


def test_fetch_indicators_output_phishing_urls(requests_mock, phishing_urls):
    """
    Given:
        - the phishing URL feed
        - some relationship information in the feed

    When:
        - running fetch-indicators

    Then:
        - the indicator type and value are being set
        - the DBot score is set to
          - BAD on non confirmed-clean category
          - NONE on feed removal and confirmed clean category
        - basic indicator fields are filled from the feed meta data
        - Cyren-specific indicator fields are filled
        - feed related indicator field is filled with IP and SHA-256 relationships

    """

    fetch, _ = _create_instance(requests_mock, "phishing_urls", phishing_urls, {"startOffset": 0, "endOffset": 0})
    created = fetch()

    assert len(created) == 8

    assert created[0]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "resolves to",
                "value": "https://verify.paypalc.o.m.accoun.t-updates.info",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:01:45.000Z",
                "entitycategory": "phishing",
            }
        ]
    }

    assert created[0]["score"] == Common.DBotScore.NONE
    assert created[0]["rawJSON"]["source_tag"] == "related"
    assert created[0]["type"] == FeedIndicatorType.IP
    assert created[0]["value"] == "195.201.98.73"

    assert created[1]["fields"] == {
        "indicatoridentification": "025859f4-4b07-58de-953b-0ed2bdc7ee0f",
        "updateddate": "2020-11-01T17:05:26.347Z",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "resolves to",
                "value": "195.201.98.73",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:01:45.000Z",
                "entitycategory": "phishing",
            }
        ],
    }

    assert created[1]["score"] == Common.DBotScore.BAD
    assert created[1]["rawJSON"]["tags"] == ["phishing", "URL Categorization", "Active URL inspection", "finance", "apple"]
    assert created[1]["rawJSON"]["source_tag"] == "primary"
    assert created[1]["type"] == FeedIndicatorType.URL
    assert created[1]["value"] == "https://verify.paypalc.o.m.accoun.t-updates.info"

    assert created[2]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "resolves to",
                "value": "http://secureapplelock.servebeer.com/manage",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:03:40.000Z",
                "entitycategory": "phishing",
            }
        ]
    }

    assert created[2]["score"] == Common.DBotScore.NONE
    assert created[2]["rawJSON"]["source_tag"] == "related"
    assert created[2]["type"] == FeedIndicatorType.IP
    assert created[2]["value"] == "192.163.194.76"

    assert created[3]["fields"] == {
        "indicatoridentification": "054f305a-f39c-51b7-b2c3-9f8c281ff1ea",
        "updateddate": "2020-11-01T17:05:26.347Z",
        "published": "2020-11-01T17:05:26.347Z",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "resolves to",
                "value": "192.163.194.76",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:03:40.000Z",
                "entitycategory": "phishing",
            }
        ],
    }

    assert created[3]["score"] == Common.DBotScore.BAD
    assert created[3]["rawJSON"]["tags"] == ["phishing", "Active URL inspection", "cloudapp"]
    assert created[3]["rawJSON"]["source_tag"] == "primary"
    assert created[3]["type"] == FeedIndicatorType.URL
    assert created[3]["value"] == "http://secureapplelock.servebeer.com/manage"

    assert created[4]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "resolves to",
                "value": "http://secureapplelock.servebeer.com/manage-2",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:03:40.000Z",
                "entitycategory": "phishing",
            }
        ]
    }

    assert created[4]["score"] == Common.DBotScore.NONE
    assert created[4]["rawJSON"]["source_tag"] == "related"
    assert created[4]["type"] == FeedIndicatorType.IP
    assert created[4]["value"] == "192.163.194.76"

    assert created[5]["fields"] == {
        "indicatoridentification": "054f305a-f39c-51b7-b2c3-9f8c281ff1eb",
        "updateddate": "2020-11-01T17:05:26.347Z",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "resolves to",
                "value": "192.163.194.76",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:03:40.000Z",
                "entitycategory": "phishing",
            }
        ],
    }

    assert created[5]["score"] == Common.DBotScore.NONE
    assert created[5]["rawJSON"]["tags"] == ["confirmed clean", "Active URL inspection", "cloudapp"]
    assert created[5]["rawJSON"]["source_tag"] == "primary"
    assert created[5]["type"] == FeedIndicatorType.URL
    assert created[5]["value"] == "http://secureapplelock.servebeer.com/manage-2"

    assert created[6]["fields"] == {
        "cyrenfeedrelationships": [
            {
                "indicatortype": "URL",
                "relationshiptype": "resolves to",
                "value": "http://secureapplelock.servebeer.com/manage-3",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:03:40.000Z",
                "entitycategory": "phishing",
            }
        ]
    }

    assert created[6]["score"] == Common.DBotScore.NONE
    assert created[6]["rawJSON"]["source_tag"] == "related"
    assert created[6]["type"] == FeedIndicatorType.IP
    assert created[6]["value"] == "192.163.194.76"

    assert created[7]["fields"] == {
        "updateddate": "2020-11-01T17:05:26.347Z",
        "indicatoridentification": "054f305a-f39c-51b7-b2c3-9f8c281ff1ec",
        "cyrenfeedrelationships": [
            {
                "indicatortype": "IP",
                "relationshiptype": "resolves to",
                "value": "192.163.194.76",
                "description": "resolves to phishing ip",
                "timestamp": "2020-11-01T17:03:40.000Z",
                "entitycategory": "phishing",
            }
        ],
    }

    assert created[7]["score"] == Common.DBotScore.NONE
    assert created[7]["rawJSON"]["tags"] == ["phishing", "Active URL inspection", "cloudapp"]
    assert created[7]["rawJSON"]["source_tag"] == "primary"
    assert created[7]["type"] == FeedIndicatorType.URL
    assert created[7]["value"] == "http://secureapplelock.servebeer.com/manage-3"


@pytest.mark.parametrize(
    "context_data, offsets, max_indicators, expected_offset, expected_count",
    [
        # Given:
        #   - first run
        #   - no previous offset stored
        #   - an end offset of 1000
        # When:
        #   - running get-indicators with count 10
        # Then:
        #   - the API is asked for 10 from offset 991
        ({}, {"startOffset": 1, "endOffset": 1000}, 10, 991, 10),
        # Given:
        #   - not the first run
        #   - previous offset is 900
        #   - an end offset of 1000
        # When:
        #   - running get-indicators with count 20
        # Then:
        #   - the API is asked for 20 from offset 981
        ({"offset": 900}, {"startOffset": 1, "endOffset": 1000}, 20, 981, 20),
    ],
)
def test_get_indicators(requests_mock, phishing_urls, context_data, offsets, max_indicators, expected_offset, expected_count):
    """
    Given:
        - the phishing URL feed

    When:
        - running get-indicators

    Then:
        - no adjustments made to the integration context
        - the number of indicators is taken from the response, meaning 4 entries

    """

    set_integration_context(context_data)
    _, get = _create_instance(requests_mock, "phishing_urls", phishing_urls, offsets, expected_offset, expected_count)
    result = get(max_indicators)

    assert len(result.raw_response) == 8
    assert get_integration_context() == context_data


def test_test_module_server_error(requests_mock):
    """
    Given:
        - the IP reputation feed

    When:
        - running test-module with a 500 Server Error

    Then:
        - it tells you the test failed

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10",
        status_code=500,
        request_headers=_expected_headers(),
    )
    client = _create_client("ip_reputation")

    assert "Test failed because of: Error in API call [500] - None" in _test_module_command(client)


def test_test_module_invalid_token(requests_mock):
    """
    Given:
        - the IP reputation feed

    When:
        - running test-module with a 400 on an invalid claim

    Then:
        - it tells you the test failed

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10",
        status_code=400,
        request_headers=_expected_headers(),
        json={"statusCode": 400, "error": "unable to parse claims from token: ..."},
    )
    client = _create_client("ip_reputation")

    assert "Test failed because of an invalid API token!" in _test_module_command(client)


def test_test_module_other_400(requests_mock):
    """
    Given:
        - the IP reputation feed

    When:
        - running test-module with an unknown 400

    Then:
        - it tells you the test failed

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10",
        status_code=400,
        request_headers=_expected_headers(),
    )
    client = _create_client("ip_reputation")

    assert "Test failed because of: 400 Client Error:" in _test_module_command(client)


def test_test_module_404(requests_mock):
    """
    Given:
        - the IP reputation feed

    When:
        - running test-module with a 404

    Then:
        - it tells you the test failed

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10",
        status_code=404,
        request_headers=_expected_headers(),
    )
    client = _create_client("ip_reputation")

    assert "Test failed because of an invalid API URL!" in _test_module_command(client)


def test_test_module_no_entries(requests_mock):
    """
    Given:
        - the IP reputation feed

    When:
        - running test-module with no entries being returned

    Then:
        - it tells you the test failed

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10", text="", request_headers=_expected_headers()
    )
    client = _create_client("ip_reputation")

    assert "Test failed because no indicators could be fetched!" in _test_module_command(client)


def test_test_module_ok(requests_mock, ip_reputation):
    """
    Given:
        - the IP reputation feed

    When:
        - running test-module with good result

    Then:
        - it tells you the test did not fail

    """

    requests_mock.get(
        BASE_URL + "/data?format=jsonl&feedId=ip_reputation_v2&offset=0&count=10",
        text=ip_reputation,
        request_headers=_expected_headers(),
    )
    client = _create_client("ip_reputation")

    assert _test_module_command(client) == "ok"


@pytest.mark.parametrize(
    "offset_data, context_data, offset, expected_text, expected_offset",
    [
        (
            {"startOffset": 1, "endOffset": 1000},
            {},
            None,
            (
                "Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 "
                "(API provided max offset of 1000, was not set before)."
            ),
            1000,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {},
            900,
            (
                "Reset Cyren Threat InDepth ip_reputation feed client offset to 900 "
                "(API provided max offset of 1000, was not set before)."
            ),
            900,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {},
            1000,
            (
                "Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 "
                "(API provided max offset of 1000, was not set before)."
            ),
            1000,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {},
            1001,
            (
                "Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 "
                "(API provided max offset of 1000, was not set before)."
            ),
            1000,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {"offset": 500},
            None,
            "Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 (API provided max offset of 1000, was 500).",
            1000,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {"offset": 500},
            900,
            "Reset Cyren Threat InDepth ip_reputation feed client offset to 900 (API provided max offset of 1000, was 500).",
            900,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {"offset": 500},
            1000,
            "Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 (API provided max offset of 1000, was 500).",
            1000,
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {"offset": 500},
            1001,
            "Reset Cyren Threat InDepth ip_reputation feed client offset to 1000 (API provided max offset of 1000, was 500).",
            1000,
        ),
    ],
)
def test_reset_offset_command(requests_mock, offset_data, context_data, offset, expected_text, expected_offset):
    """
    Given:
        - different stored offset configurations and desired offset parameters

    When:
        - running the reset offset command

    Then:
        - I am told what happened in a human-readable form
        - the new context has been stored in the integration context

    """

    set_integration_context(context_data)
    feed = "ip_reputation"
    requests_mock.get(BASE_URL + f"/info?format=jsonl&feedId={feed}_v2", json=offset_data, request_headers=_expected_headers())
    client = _create_client(feed)

    args = {}
    if offset is not None:
        args["offset"] = offset

    result = reset_offset_command(client, args)

    assert result.readable_output == expected_text
    assert get_integration_context() == {"offset": expected_offset}


@pytest.mark.parametrize(
    "offset_data, context_data, expected_text",
    [
        (
            {"startOffset": 1, "endOffset": 1000},
            {},
            ("Cyren Threat InDepth ip_reputation feed client offset has not been set yet (API provided max offset of 1000)."),
        ),
        (
            {"startOffset": 1, "endOffset": 1000},
            {"offset": 500},
            ("Cyren Threat InDepth ip_reputation feed client offset is 500 (API provided max offset of 1000)."),
        ),
    ],
)
def test_get_offset_command(requests_mock, offset_data, context_data, expected_text):
    """
    Given:
        - different stored offset configurations

    When:
        - running the get offset command

    Then:
        - I am told what the offset is

    """

    set_integration_context(context_data)
    feed = "ip_reputation"
    requests_mock.get(BASE_URL + f"/info?format=jsonl&feedId={feed}_v2", json=offset_data, request_headers=_expected_headers())
    client = _create_client(feed)

    result = get_offset_command(client, {})

    assert result.readable_output == expected_text