URLhaus

URLhaus has the goal of sharing malicious URLs that are being used for malware distribution.

Data Enrichment & Threat Intelligence · URLhaus

Details

IDURLhaus
ProviderOpen Source
CategoryData Enrichment & Threat Intelligence
From Version5.0.0
Docker Imagedemisto/python3:3.12.13.10116658
Supported ModulesAgentix XSIAM

README

URLhaus leverages abuse.ch and shares malicious URLs that are being used for malware distribution.
This integration was integrated and tested with version v1 of URLhaus.

Configure URLhaus in Cortex

Parameter Description Required
Server URL (e.g. https://192.168.0.1)   False
Auth Key Auth Key for authentication with abuse.ch True
Source Reliability Reliability of the source providing the intelligence data. True
Trust any certificate (not secure)   False
Use system proxy settings   False
Create relationships   False
Maximum number of relationships to fetch per indicator Maximal value is 1000. False
Blacklists appearances threshold   False
Compromised (is malicious)   False
Number of retries Determines how many times a command should be retried before raising an error. False

Create a required Auth Key for abuse.ch

Note: If you already have a profile, you can skip steps 1 and 2.

  1. Sign up for an abuse.ch account. You can do this easily by using an existing account that you may already have on X, LinkedIn, Google or Github. Just log in with the authentication provider of your choice here: https://auth.abuse.ch/

  2. Once you are authenticated on abuse.ch, ensure that you connect at least one additional authentication provider. This will ensure that you have access to abuse.ch platforms, even if one of the authentication providers you use shuts down (yes, it happened with Twitter!)

  3. Ensure that you hit the “Save profile” button. In the “Optional” section, you can now generate an “Auth-Key”. This is your personal Auth-Key that you can now use in the integration.

Commands

You can execute these commands from the CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

url


Retrieves URL information from URLhaus.

Notice: Submitting indicators using this command might make the indicator data publicly available. See the vendor’s documentation for more details.

Base Command

url

Input

Argument Name Description Required
url A comma-separated list of URLs to query. Required

Context Output

Path Type Description
URL.Data string The URL.
URL.Malicious.Vendor string Vendor that reported the URL as malicious.
URL.Malicious.Description string Description of the malicious URL.
URL.Tags string A list of tags associated with the queried malware URL.
URL.Relationships.EntityA String The source of the relationship.
URL.Relationships.EntityB String The destination of the relationship.
URL.Relationships.Relationship String The name of the relationship.
URL.Relationships.EntityAType String The type of the source of the relationship.
URL.Relationships.EntityBType String The type of the destination of the relationship.
URLhaus.URL.ID String Unique identifier of the URLhaus database entry.
URLhaus.URL.Status String The current status of the URL.
URLhaus.URL.Host String The extracted host of the malware URL (IP address or domain name/FQDN).
URLhaus.URL.DateAdded date Date the URL was added to URLhaus.
URLhaus.URL.Threat String The threat corresponding to this malware URL.
URLhaus.URL.Blacklist.Name String Name of the block list.
URLhaus.URL.Tags String A list of tags associated with the queried malware URL.
URLhaus.URL.Payload.Name String Payload file name.
URLhaus.URL.Payload.Type String Payload file type.
URLhaus.URL.Payload.MD5 String MD5 hash of the HTTP response body (payload).
URLhaus.URL.Payload.VT.Result Number VirusTotal results for the payload.
DBotScore.Indicator String The indicator that was tested.
DBotScore.Type String The indicator type.
DBotScore.Vendor String The vendor used to calculate the score.
DBotScore.Score Number The actual score.
DBotScore.Reliability String Reliability of the source providing the intelligence data.
URLhaus.URL.Blacklist.Status String Status of the URL in the block list.
URLhaus.URL.Payload.VT.Link String Link to the VirusTotal report.

Command example

!url using-brand=URLhaus url=http://example.com/anklet/WQG1/?i=1

Context Example

{
    "DBotScore": {
        "Indicator": "http://example.com/anklet/WQG1/?i=1",
        "Reliability": "C - Fairly reliable",
        "Score": 2,
        "Type": "url",
        "Vendor": "URLhaus"
    },
    "URL": {
        "Data": "http://example.com/anklet/WQG1/?i=1",
        "Relationships": [
            {
                "EntityA": "http://example.com/anklet/WQG1/?i=1",
                "EntityAType": "URL",
                "EntityB": "example.com",
                "EntityBType": "Domain",
                "Relationship": "hosted-on"
            }
        ],
        "Tags": [
            "doc",
            "emotet",
            "epoch5",
            "heodo",
            "malware_download"
        ]
    },
    "URLhaus": {
        "URL": {
            "Blacklist": [
                {
                    "Name": "spamhaus_dbl",
                    "Status": "not listed"
                },
                {
                    "Name": "surbl",
                    "Status": "not listed"
                }
            ],
            "DateAdded": "2022-01-20T14:11:09",
            "Host": "example.com",
            "ID": "1992762",
            "Payload": [
                {
                    "MD5": "716c3aa1e0da98b6e99cadd60363ae7e",
                    "Name": "BC-77388.xlsm",
                    "SHA256": "64c6ba33444e5db3cc9c99613d04fd163ec1971ee5eb90041a17068e37578fc0",
                    "Type": "xls",
                    "VT": null
              }
            ],
            "Status": "offline",
            "Tags": [
                "doc",
                "emotet",
                "epoch5",
                "heodo",
                "malware_download"
            ],
            "Threat": "malware_download"
        }
    }
}

Human Readable Output

URLhaus reputation for http://example.com/anklet/WQG1/?i=1

Date added Description Status Threat URLhaus ID URLhaus link
2022-01-20T14:11:09 The URL is inactive (offline) and serving no payload offline malware_download 1992762 https://urlhaus.abuse.ch/url/1992762/

domain


Retrieves domain information from URLhaus.

Notice: Submitting indicators using this command might make the indicator data publicly available. See the vendor’s documentation for more details.

Base Command

domain

Input

Argument Name Description Required
domain A comma-separated list of domains to query. Required

Context Output

Path Type Description
Domain.Name String The domain name, for example, google.com.
Domain.Tags string A list of tags associated with the queried malware Domain.
Domain.Relationships.EntityA String The source of the relationship.
Domain.Relationships.EntityB String The destination of the relationship.
Domain.Relationships.Relationship String The name of the relationship.
Domain.Relationships.EntityAType String The type of the source of the relationship.
Domain.Relationships.EntityBType String The type of the destination of the relationship.
DBotScore.Indicator String The indicator that was tested.
DBotScore.Type String The indicator type.
DBotScore.Vendor String The vendor used to calculate the score.
DBotScore.Score Number The actual score.
DBotScore.Reliability String Reliability of the source providing the intelligence data.
URLhaus.Domain.FirstSeen Date Date that the IP was seen for the first time (UTC).
URLhaus.Domain.Blacklist.Name String The status of the domain in different block lists.
URLhaus.Domain.URL String URLs observed on this domain.
Domain.Malicious.Vendor String Vendor that reported the domain as malicious.
Domain.Malicious.Description String Description of the malicious domain.
URLhaus.Domain.Blacklist.Status String Status of the URL in the block list.

Command example

!domain using-brand=URLhaus domain=example.com

Context Example

{
    "DBotScore": {
        "Indicator": "example.com",
        "Reliability": "C - Fairly reliable",
        "Score": 1,
        "Type": "domain",
        "Vendor": "URLhaus"
    },
    "Domain": {
        "Name": "example.com",
        "Relationships": [
            {
                "EntityA": "example.com",
                "EntityAType": "Domain",
                "EntityB": "http://example.com:443/wp-content/plugins/wp-roilbask/includes/",
                "EntityBType": "URL",
                "Relationship": "hosts"
            }
        ],
        "Tags": [
            "abused_legit_malware"
        ]
    },
    "URLhaus": {
        "Domain": {
            "Blacklist": {
                "spamhaus_dbl": "abused_legit_malware",
                "surbl": "not listed"
            },
            "FirstSeen": "2022-01-27T12:51:03",
            "URL": [
                {
                    "date_added": "2022-01-28 04:41:03 UTC",
                    "id": "2010874",
                    "larted": "false",
                    "reporter": "Cryptolaemus1",
                    "tags": [
                        "IcedID"
                    ],
                    "takedown_time_seconds": null,
                    "threat": "malware_download",
                    "url": "http://example.com:443/wp-content/plugins/wp-roilbask/includes/",
                    "url_status": "offline",
                    "urlhaus_reference": "https://urlhaus.abuse.ch/url/2010874/"
                }
            ]
        }
    }
}

Human Readable Output

URLhaus reputation for example.com

Description First seen URLhaus link
There is no information about Domain in the blacklist 2022-01-27T12:51:03 https://urlhaus.abuse.ch/host/example.com/

file


Retrieves file information from URLhaus.

Base Command

file

Input

Argument Name Description Required
file A comma-separated list of MD5 or SHA256 hashes of the file to query. Required

Context Output

Path Type Description
File.Size Number File size (in bytes).
File.MD5 String MD5 hash of the file.
File.SHA256 String SHA256 hash of the file.
File.SSDeep String SSDeep of the file.
File.Type String Type of the file.
File.Relationships.EntityA String The source of the relationship.
File.Relationships.EntityB String The destination of the relationship.
File.Relationships.Relationship String The name of the relationship.
File.Relationships.EntityAType String The type of the source of the relationship.
File.Relationships.EntityBType String The type of the destination of the relationship.
URLhaus.File.MD5 String MD5 hash of the file.
URLhaus.File.SHA256 String SHA256 hash of the file.
URLhaus.File.Type String File type guessed by URLhaus, for example: .exe, .doc.
URLhaus.File.Size Number File size (in bytes).
URLhaus.File.Signature String Malware family.
URLhaus.File.FirstSeen Date Date and time (UTC) that URLhaus first saw this file (payload).
URLhaus.File.LastSeen Date Date and time (UTC) that URLhaus last saw this file (payload).
URLhaus.File.DownloadLink String Location (URL) where you can download a copy of this file.
URLhaus.File.VirusTotal.Percent Number AV detection (percentage), for example: 24.14.
URLhaus.File.VirusTotal.Link String Link to the VirusTotal report.
URLhaus.File.URL Unknown A list of malware URLs associated with this payload (max. 100).
DBotScore.Indicator String The indicator that was tested.
DBotScore.Type String The indicator type.
DBotScore.Vendor String The vendor used to calculate the score.
DBotScore.Score Number The actual score.
DBotScore.Reliability String Reliability of the source providing the intelligence data.

Command example

!file using-brand=URLhaus file=7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89

Context Example

{
    "DBotScore": {
        "Indicator": "7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89",
        "Reliability": "C - Fairly reliable",
        "Score": 3,
        "Type": "file",
        "Vendor": "URLhaus"
    },
    "File": {
        "Malicious": {
            "Description": "This file is malicious",
            "Vendor": "URLhaus"
        },
        "Relationships": [
            {
                "EntityA": "7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89",
                "EntityAType": "File",
                "EntityB": "BazaLoader",
                "EntityBType": "Malware",
                "Relationship": "indicator-of"
            }
        ],
        "SHA256": "7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89",
        "SSDeep": "24576:la1QHwgJMrQqj/wAc6QORNx2nAjwkaMm0GV9igWwlnwXQBwfalj21X4GtZ+FdnZ8:vH5qloBMd8A",
        "Type": "dll"
    },
    "URLhaus": {
        "File": {
            "DownloadLink": "https://urlhaus-api.abuse.ch/v1/download/7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89/",
            "FirstSeen": "2022-01-18T11:18:31",
            "LastSeen": "2022-01-28T09:36:21",
            "MD5": "2ff9cce7a08215ded0945de5965d2a0a",
            "SHA256": "7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89",
            "Signature": "BazaLoader",
            "Size": 1816064,
            "Type": "dll",
            "URL": [
                {
                    "filename": "DH-1643319814.xll",
                    "firstseen": "2022-01-27",
                    "lastseen": null,
                    "url": "http://www.example.com/wp-content/plugins/wp-roilbask/includes/",
                    "url_id": "2009726",
                    "url_status": "online",
                    "urlhaus_reference": "https://urlhaus.abuse.ch/url/2009726/"
                }
            ]
        }
    }
}

Human Readable Output

URLhaus reputation for SHA256 : 7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89

First seen Last seen MD5 SHA256 Signature URLhaus link
2022-01-18T11:18:31 2022-01-28T09:36:21 2ff9cce7a08215ded0945de5965d2a0a 7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89 BazaLoader https://urlhaus-api.abuse.ch/v1/download/7855068e0cfb093ab9be9ec172676e3c119e16511f3d631d715a4e77ddad9d89/

urlhaus-download-sample


Downloads a malware sample from URLhaus.

Base Command

urlhaus-download-sample

Input

Argument Name Description Required
file SHA256 hash of the file to download. Required

Context Output

Path Type Description
File.Size number File size.
File.SHA1 string SHA1 hash of the file.
File.SHA256 string SHA256 hash of the file.
File.Name string File name.
File.SSDeep string SSDeep hash of the file.
File.EntryID string File entry ID.
File.Info string File information.
File.Type string File type.
File.MD5 string MD5 hash of the file.
File.Extension string File extension.

Command example

!urlhaus-download-sample file=254ca6a7a7ef7f17d9884c4a86f88b5d5fd8fe5341c0996eaaf1d4bcb3b2337b

Human Readable Output

{
   "HumanReadable": "No results for SHA256: 254ca6a7a7ef7f17d9884c4a86f88b5d5fd8fe5341c0996eaaf1d4bcb3b2337b",
   "HumanReadableFormat": "markdown",
   "Type": 1
}

Configuration parameters

  • url — Server URL (e.g. https://192.168.0.1)
  • credentials
  • integrationReliability — Source Reliability (required)
  • insecure — Trust any certificate (not secure)
  • proxy — Use system proxy settings
  • create_relationships — Create relationships
  • max_num_of_relationships — Maximum number of relationships to fetch per indicator
  • threshold — Blacklists appearances threshold
  • compromised_is_malicious — Compromised (is malicious)
  • retry — Number of retries
  • should_error — Return Error

Commands (4)

  • domain

    Retrieves domain information from URLhaus.

  • file

    Retrieves file information from URLhaus.

  • url

    Retrieves URL information from URLhaus.

  • urlhaus-download-sample

    Downloads a malware sample from URLhaus.

import urllib3
from CommonServerPython import *

""" IMPORTS """
import io
import re
import traceback
import zipfile
from datetime import datetime as dt

import requests

# Disable insecure warnings
urllib3.disable_warnings()

# disable-secrets-detection-start
# Whether compromised websites are considered malicious or not. See the blacklists output in
# https://urlhaus-api.abuse.ch/
# disable-secrets-detection-end
COMPROMISED_IS_MALICIOUS = demisto.params().get("compromised_is_malicious", False)

# Headers to be sent in requests
HEADERS = {"Content-Type": "application/x-www-form-urlencoded", "Accept": "application/json"}

""" HELPER FUNCTIONS """


def http_request(method, command, api_url, use_ssl, data=None):
    retry = int(demisto.params().get("retry", 3))
    try_num = 0

    while try_num < retry:
        try_num += 1
        url = f"{api_url}/{command}/"
        res = requests.request(method, url, verify=use_ssl, data=data, headers=HEADERS)

        if res.status_code == 200:
            return res

    raise Exception(f"Error in API call {url} [{res.status_code}] - {res.reason}")


def reformat_date(date):
    try:
        return dt.strptime(date.rstrip(" UTC"), "%Y-%m-%d %H:%M:%S").strftime("%Y-%m-%dT%H:%M:%S")
    except Exception:
        return "Unknown"


def extract_zipped_buffer(buffer):
    with io.BytesIO() as bio:
        bio.write(buffer)
        with zipfile.ZipFile(bio) as z:
            return z.read(z.namelist()[0])


def query_url_information(url, api_url, use_ssl):
    return http_request("POST", "url", api_url, use_ssl, f"url={url}")


def query_host_information(host, api_url, use_ssl):
    return http_request("POST", "host", api_url, use_ssl, f"host={host}")


def query_payload_information(hash_type, api_url, use_ssl, hash):
    return http_request("POST", "payload", api_url, use_ssl, f"{hash_type}_hash={hash}")


def download_malware_sample(sha256, api_url, use_ssl):
    return http_request("GET", f"download/{sha256}", api_url=api_url, use_ssl=use_ssl)


""" COMMANDS + REQUESTS FUNCTIONS """


def test_module(**kwargs):
    """
    Performs basic get request to get item samples
    """
    http_request("POST", "url", kwargs.get("api_url"), kwargs.get("use_ssl"))


def url_calculate_score(status: str) -> tuple[int, str]:
    """
    Calculate DBot Score for the url command using url status.

    Args:
        status (str): A URL status.

    Returns:
        dbot_score,description (tuple): The DBot Score and the description associated with it.
    """

    status_dict = {
        "online": (Common.DBotScore.BAD, "The URL is active (online) and currently serving a payload"),
        "offline": (Common.DBotScore.SUSPICIOUS, "The URL is inadctive (offline) and serving no payload"),
        "unknown": (Common.DBotScore.NONE, "The URL status could not be determined"),
    }
    if status_dict.get(status):
        return status_dict[status]
    raise Exception("Got bad url status")


def domain_calculate_score(blacklist: dict) -> tuple[int, str]:
    """
    Calculate DBot Score for the domain command using blacklist.

    Args:
        blacklist (dict): Containing spamhaus_dbl and surbl.

    Returns:
        dbot_score,description (tuple): The DBot Score and the description associated with it.
    """
    spamhaus_dbl = blacklist.get("spamhaus_dbl", "")
    surbl = blacklist.get("surbl", "")

    if spamhaus_dbl:
        if spamhaus_dbl == "spammer_domain":
            return Common.DBotScore.BAD, "The queried Domain is a known spammer domain"
        if spamhaus_dbl == "phishing_domain":
            return Common.DBotScore.BAD, "The queried Domain is a known phishing domain"
        if spamhaus_dbl == "botnet_cc_domain":
            return Common.DBotScore.BAD, "The queried Domain is a known botnet C&C domain"
    if surbl and surbl == "listed":
        return Common.DBotScore.BAD, "The queried Domain is listed on SURBL"
    if spamhaus_dbl and spamhaus_dbl == "not listed":
        return Common.DBotScore.NONE, "The queried Domain is not listed on Spamhaus DBL"
    if surbl and surbl == "not listed":
        return Common.DBotScore.NONE, "The queried Domain is not listed on SURBL"
    return Common.DBotScore.GOOD, "There is no information about Domain in the blacklist"


def file_calculate_score() -> tuple[int, str]:
    """
    Calculate DBot Score for the file command (always malicious).

    Args:
       -

    Returns:
        dbot_score,description (tuple): The DBot Score and the description associated with it.
    """
    return Common.DBotScore.BAD, "This file is malicious"


def determine_host_ioc_type(host: str) -> str:
    """
    Determine the host ioc type.

    Args:
        host (str): The host.

    Returns:
        type (str): The type of the host.
    """
    return "ip" if is_ip_valid(host) else "domain"


def url_create_relationships(
    uri: str, host: str, files: List[dict], create_relationships: bool, max_num_of_relationships: int
) -> List[EntityRelationship]:
    """
    Returns a list of relationships if create_relationships is true (limited to max_num_of_relationships).

    Args:
        uri (str): The queried URL.
        host (str): A host associated with the URL.
        files (list): Files associated with the URL.
        create_relationships (bool): Indicator for create relationships table.
        max_num_of_relationships (int): Indicator for how many relationships to display.

    Returns:
        relationships (list): The EntityRelationship objects representing the URL relationships.
    """
    relationships = []
    if create_relationships and max_num_of_relationships is not None:
        if host:
            parsed_host = determine_host_ioc_type(host)
            if parsed_host == "domain":
                relationships.append(
                    EntityRelationship(
                        name=EntityRelationship.Relationships.HOSTED_ON,
                        entity_a=uri,
                        entity_a_type=FeedIndicatorType.URL,
                        entity_b=host,
                        entity_b_type=FeedIndicatorType.Domain,
                        reverse_name=EntityRelationship.Relationships.HOSTS,
                    )
                )
            if parsed_host == "ip":
                relationships.append(
                    EntityRelationship(
                        name=EntityRelationship.Relationships.RELATED_TO,
                        entity_a=uri,
                        entity_a_type=FeedIndicatorType.URL,
                        entity_b=host,
                        entity_b_type=FeedIndicatorType.IP,
                        reverse_name=EntityRelationship.Relationships.RELATED_TO,
                    )
                )
        if files:
            for file in files:
                if len(relationships) >= max_num_of_relationships:
                    break

                file_sh256 = file.get("SHA256")
                if file_sh256:
                    relationships.append(
                        EntityRelationship(
                            name=EntityRelationship.Relationships.RELATED_TO,
                            entity_a=uri,
                            entity_a_type=FeedIndicatorType.URL,
                            entity_b=file_sh256,
                            entity_b_type=FeedIndicatorType.File,
                            reverse_name=EntityRelationship.Relationships.RELATED_TO,
                        )
                    )
    return relationships


def url_create_tags(urlhaus_data: dict) -> List[str]:
    """
    Create url tags.

    Args:
        urlhaus_data (dict): The data retrieved from URLHaus db.

    Returns:
        tags (list): a list of tags to add.
    """
    tags = urlhaus_data.get("Tags", [])
    if urlhaus_data.get("Threat"):
        tags.append(urlhaus_data.get("Threat"))
    return tags


def url_create_payloads(url_information: dict) -> List[dict]:
    """
    Returns a list of payloads.

    Args:
        url_information (dict): The data retrieved from URLHaus db.

    Returns:
        payloads (list): list of payloads associated with the URL.
    """
    payloads = []
    for payload in url_information.get("payloads") or []:
        vt_data = payload.get("virustotal", None)
        vt_information = None
        if vt_data:
            vt_information = {"Result": float(vt_data.get("percent", 0)), "Link": vt_data.get("link", "")}
        payloads.append(
            {
                "Name": payload.get("filename", "unknown"),
                "Type": payload.get("file_type", ""),
                "MD5": payload.get("response_md5", ""),
                "SHA256": payload.get("response_sha256", ""),
                "VT": vt_information,
            }
        )
    return payloads


def url_create_blacklist(url_information: dict) -> List[dict]:
    """
    Create blacklist for url command.

    Args:
        url_information(dict).

    Returns:
        Blacklist(list).
    """
    blacklist_information = []
    blacklists = url_information.get("blacklists", {})
    for bl_name, bl_status in blacklists.items():
        blacklist_information.append({"Name": bl_name, "Status": bl_status})
    return blacklist_information


def build_context_url_ok_status(url_information: dict, uri: str, params: dict) -> CommandResults:
    """
    Build the output context if the status is ok.

    Args:
       url_information (dict): The data retrieved from URLHaus db.
       uri (str): The queried URL.
       params (dict): The integration params.

    Returns:
        result (CommandResults): The CommandResults object representing the url command results.
    """
    blacklist_information = url_create_blacklist(url_information)
    date_added = reformat_date(url_information.get("date_added"))
    payloads = url_create_payloads(url_information)
    urlhaus_data = {
        "ID": url_information.get("id", ""),
        "Status": url_information.get("url_status", ""),
        "Host": url_information.get("host", ""),
        "DateAdded": date_added,
        "Threat": url_information.get("threat", ""),
        "Blacklist": blacklist_information,
        "Tags": url_information.get("tags", []),
        "Payload": payloads,
    }

    # DBot score calculation
    score, description = url_calculate_score(url_information.get("url_status", {}))
    dbot_score = Common.DBotScore(
        indicator=uri,
        integration_name="URLhaus",
        indicator_type=DBotScoreType.URL,
        reliability=params.get("reliability"),
        score=score,
        malicious_description=description,
    )
    relationships = url_create_relationships(
        uri,
        url_information.get("host", ""),
        payloads,
        params.get("create_relationships", True),
        params.get("max_num_of_relationships", 10),
    )
    url_indicator = Common.URL(url=uri, dbot_score=dbot_score, tags=url_create_tags(urlhaus_data), relationships=relationships)
    human_readable = tableToMarkdown(
        f"URLhaus reputation for {uri}",
        {
            "URLhaus link": url_information.get("urlhaus_reference", "None"),
            "Description": description,
            "URLhaus ID": urlhaus_data["ID"],
            "Status": urlhaus_data["Status"],
            "Threat": url_information.get("threat", ""),
            "Date added": date_added,
        },
    )

    return CommandResults(
        readable_output=human_readable,
        outputs_prefix="URLhaus.URL",
        outputs_key_field="ID",
        outputs=urlhaus_data,
        raw_response=url_information,
        indicator=url_indicator,
        relationships=relationships,
    )


def process_query_info(url_information: dict, uri: str, params: dict) -> CommandResults:
    """
    Process the response.

    Args:
       url_information (dict): The data retrieved from URLHaus db.
       uri (str): The queried URL.
       params (dict): The integration params.

    Returns:
        result (CommandResults): The CommandResults object representing the url command results.
    """
    if url_information["query_status"] == "ok":
        return build_context_url_ok_status(url_information, uri, params)

    elif url_information["query_status"] == "no_results" or url_information["query_status"] == "invalid_url":
        if re.match(urlRegex, uri):
            return create_indicator_result_with_dbotscore_unknown(
                indicator=uri, indicator_type=DBotScoreType.URL, reliability=params.get("reliability")
            )
        human_readable = f"## URLhaus reputation for {uri}\nInvalid URL!"
        return CommandResults(
            readable_output=human_readable,
            raw_response=url_information,
        )
    else:
        raise DemistoException(f'Query results = {url_information["query_status"]}', res=url_information)


def run_url_command(url: str, params: dict) -> CommandResults:
    """
    Query the url_information from URLHaus db.

    Args:
       params (dict): The integration params.
       url (str): The queried URL.

    Returns:
        result (CommandResults): The CommandResults object representing the url command resultsgit .
    """
    try:
        url_information = query_url_information(url, params.get("api_url"), params.get("use_ssl")).json()
    except UnicodeEncodeError:
        return CommandResults(
            readable_output="Service Does not support special characters.",
        )
    return process_query_info(url_information, url, params)


def url_command(params: dict):
    """
    Split urls and call run_url_command on each of them.

     Args:
        params (dict): The integration params.
    """
    urls = demisto.args().get("url", "")
    for url in argToList(urls):
        return_results(results=run_url_command(url, params))


def domain_create_relationships(
    urls: List[dict], domain: str, create_relationships: bool, max_num_of_relationships: int
) -> List[EntityRelationship]:
    """
    Returns a list of relationships if create_relationships is true (limited to max_num_of_relationships).

    Args:
        domain (str): The queried Domain.
        urls (list): Urls associated with the Domain.
        create_relationships (bool): Indicator for create relationships table.
        max_num_of_relationships (int): Indicator for how many relationships to display.

    Returns:
        relationships (list): The EntityRelationship objects representing the Domain relationships.
    """
    relationships: list = []
    if create_relationships and max_num_of_relationships is not None:
        for url in urls:
            if len(relationships) >= max_num_of_relationships:
                break
            relationships.append(
                EntityRelationship(
                    name=EntityRelationship.Relationships.HOSTS,
                    entity_a=domain,
                    entity_a_type=FeedIndicatorType.Domain,
                    entity_b=url.get("url"),
                    entity_b_type=FeedIndicatorType.URL,
                    reverse_name=EntityRelationship.Relationships.HOSTED_ON,
                )
            )
    return relationships


def domain_add_tags(bl_status: str, tags: List[str]) -> None:
    """
    Create tags associated to the domain.

    Args:
        bl_status (str): The Blacklist status associated with the Domain.
        tags (list): A list of tags to return.

    """
    if bl_status:
        tag_to_add = (
            bl_status.replace("_domain", "")
            if bl_status.endswith("domain")
            else bl_status
            if bl_status.startswith("abused")
            else ""
        )
        if tag_to_add:
            tags.append(tag_to_add)


def run_domain_command(domain: str, params: dict) -> CommandResults:
    """
    Query the domain_information from URLHaus db.

    Args:
       domain (str): Domain to query.
       params (dict): The integration params.

    Returns:
        result (CommandResults): The CommandResults object representing the domain command results.
    """

    domain_information = query_host_information(domain, params.get("api_url"), params.get("use_ssl")).json()

    tags: list = []
    if domain_information["query_status"] == "ok":
        # URLHaus output
        blacklist_information = []
        blacklists = domain_information.get("blacklists", {})
        for bl_name, bl_status in blacklists.items():
            blacklist_information.append({"Name": bl_name, "Status": bl_status})
            domain_add_tags(bl_status, tags)
        first_seen = reformat_date(domain_information.get("firstseen"))

        urlhaus_data = {"FirstSeen": first_seen, "Blacklist": blacklists, "URL": domain_information.get("urls", [])}

        # DBot score calculation
        score, description = domain_calculate_score(domain_information.get("blacklists", {}))
        dbot_score = Common.DBotScore(
            indicator=domain,
            integration_name="URLhaus",
            indicator_type=DBotScoreType.DOMAIN,
            reliability=params.get("reliability"),
            score=score,
            malicious_description=description,
        )
        relationships = domain_create_relationships(
            urlhaus_data.get("URL", ""),
            domain,
            params.get("create_relationships", True),
            params.get("max_num_of_relationships", False),
        )
        domain_indicator = Common.Domain(domain=domain, dbot_score=dbot_score, tags=tags, relationships=relationships)

        human_readable = tableToMarkdown(
            f"URLhaus reputation for {domain}",
            {
                "URLhaus link": domain_information.get("urlhaus_reference", "None"),
                "Description": description,
                "First seen": first_seen,
            },
        )
        return CommandResults(
            readable_output=human_readable,
            outputs_prefix="URLhaus.Domain",
            outputs=urlhaus_data,
            raw_response=domain_information,
            indicator=domain_indicator,
            relationships=relationships,
        )
    elif domain_information["query_status"] == "no_results":
        return create_indicator_result_with_dbotscore_unknown(
            indicator=domain, indicator_type=DBotScoreType.DOMAIN, reliability=params.get("reliability")
        )

    elif domain_information["query_status"] == "invalid_host":
        human_readable = f"## URLhaus reputation for {domain}\nInvalid domain!"
        return CommandResults(readable_output=human_readable, raw_response=domain_information)
    else:
        raise DemistoException(f'Query results = {domain_information["query_status"]}', res=domain_information)


def domain_command(params: dict):
    """
    Split domains and call run_domain_command on each of them.

     Args:
        params (dict): The integration params.

    """
    domains = demisto.args().get("domain", "")
    for domain in argToList(domains):
        return_results(results=run_domain_command(domain, params))


def file_create_relationships(
    urls: List[dict], sig: str, file: str, create_relationships: bool, max_num_of_relationships: int
) -> List[EntityRelationship]:
    """
    Returns a list of relationships if create_relationships is true (limited to max_num_of_relationships).

    Args:
        urls (list): Urls associated with the Domain.
        sig (str): The signature of the File.
        file (str): The queried File.
        create_relationships (bool): Indicator for create relationships table.
        max_num_of_relationships (int): Indicator for how many relationships to display.

    Returns:
        relationships (list): The EntityRelationship objects representing the File relationships.
    """
    relationships = []
    if create_relationships and max_num_of_relationships is not None:
        if sig:
            relationships.append(
                EntityRelationship(
                    name=EntityRelationship.Relationships.INDICATOR_OF,
                    entity_a=file,
                    entity_a_type=FeedIndicatorType.File,
                    entity_b=sig,
                    entity_b_type=ThreatIntel.ObjectsNames.MALWARE,
                    reverse_name=EntityRelationship.Relationships.INDICATED_BY,
                )
            )
        for url in urls:
            if len(relationships) >= max_num_of_relationships:
                break
            relationships.append(
                EntityRelationship(
                    name=EntityRelationship.Relationships.RELATED_TO,
                    entity_a=file,
                    entity_a_type=FeedIndicatorType.File,
                    entity_b=url.get("url"),
                    entity_b_type=FeedIndicatorType.URL,
                    reverse_name=EntityRelationship.Relationships.RELATED_TO,
                )
            )
    return relationships


def run_file_command(hash: str, params: dict) -> CommandResults:
    """
    Query the file_information from URLHaus db.

    Args:
       hash (str): file to query.
       params (dict): The integration params.

    Returns:
        result (CommandResults): The CommandResults object representing the file command results.
    """
    hash_type = ""
    if len(hash) == 32:
        hash_type = "md5"
    elif len(hash) == 64:
        hash_type = "sha256"
    elif params.get("should_error", True):
        return_error("Only accepting MD5 (32 bytes) or SHA256 (64 bytes) hash types")
    else:
        return_warning("Only accepting MD5 (32 bytes) or SHA256 (64 bytes) hash types", exit=True)

    file_information = query_payload_information(hash_type, params.get("api_url"), params.get("use_ssl"), hash).json()

    if file_information["query_status"] == "ok" and file_information["md5_hash"]:
        # URLhaus output
        first_seen = reformat_date(file_information.get("firstseen"))
        last_seen = reformat_date(file_information.get("lastseen"))

        urlhaus_data = {
            "MD5": file_information.get("md5_hash", ""),
            "SHA256": file_information.get("sha256_hash", ""),
            "Type": file_information.get("file_type", ""),
            "Size": int(file_information.get("file_size", "")),
            "Signature": file_information.get("signature", ""),
            "FirstSeen": first_seen,
            "LastSeen": last_seen,
            "DownloadLink": file_information.get("urlhaus_download", ""),
            "URL": file_information.get("urls", []),
        }

        virus_total_data = file_information.get("virustotal")
        if virus_total_data:
            urlhaus_data["VirusTotal"] = {
                "Percent": float(file_information.get("virustotal", {"percent": 0})["percent"]),
                "Link": file_information.get("virustotal", {"link": ""})["link"],
            }

        score, description = file_calculate_score()
        dbot_score = Common.DBotScore(
            indicator=hash,
            integration_name="URLhaus",
            indicator_type=DBotScoreType.FILE,
            reliability=params.get("reliability"),
            score=score,
            malicious_description=description,
        )

        relationships = file_create_relationships(
            urlhaus_data["URL"],
            urlhaus_data.get("Signature", ""),
            hash,
            params.get("create_relationships", True),
            params.get("max_num_of_relationships", 10),
        )

        file_indicator = Common.File(
            sha256=hash,
            dbot_score=dbot_score,
            relationships=relationships,
            ssdeep=file_information.get("ssdeep"),
            file_type=file_information.get("file_type"),
        )

        human_readable = tableToMarkdown(
            f"URLhaus reputation for {hash_type.upper()} : {hash}",
            {
                "URLhaus link": urlhaus_data.get("DownloadLink", ""),
                "Signature": urlhaus_data.get("Signature", ""),
                "MD5": urlhaus_data.get("MD5", ""),
                "SHA256": urlhaus_data.get("SHA256", ""),
                "First seen": first_seen,
                "Last seen": last_seen,
                "SSDeep": file_information.get("ssdeep"),
                "Type": file_information.get("file_type"),
            },
        )
        return CommandResults(
            readable_output=human_readable,
            outputs_prefix="URLhaus.File",
            outputs=urlhaus_data,
            raw_response=file_information,
            indicator=file_indicator,
            relationships=relationships,
        )

    elif (file_information["query_status"] == "ok" and not file_information["md5_hash"]) or file_information[
        "query_status"
    ] == "no_results":
        return create_indicator_result_with_dbotscore_unknown(
            indicator=hash, indicator_type=DBotScoreType.FILE, reliability=params.get("reliability")
        )

    elif file_information["query_status"] in ["invalid_md5", "invalid_sha256"]:
        human_readable = (
            f'## URLhaus reputation for {hash_type.upper()} : {hash}\n'
            f'Invalid {file_information["query_status"].lstrip("invalid_").upper()}!'  # noqa: B005
        )
        return CommandResults(readable_output=human_readable, raw_response=file_information)
    else:
        raise DemistoException(f'Query results = {file_information["query_status"]}', res=file_information)


def file_command(params: dict):
    """
    Split domains and call run_domain_command on each of them.

     Args:
        params (dict): The integration params.

    """
    files = demisto.args().get("file", "")
    for file in argToList(files):
        return_results(results=run_file_command(file, params))


def urlhaus_download_sample_command(**kwargs):
    """
    The response can be either the zipped sample (content-type = application/zip), or JSON (content-type = text/html)
    containing the query status.
    """
    file_sha256 = demisto.args()["file"]
    res = download_malware_sample(file_sha256, kwargs.get("api_url"), kwargs.get("use_ssl"))

    try:
        if len(res.content) == 0:
            demisto.results(
                {
                    "Type": entryTypes["note"],
                    "HumanReadable": f"No results for SHA256: {file_sha256}",
                    "HumanReadableFormat": formats["markdown"],
                }
            )
        elif res.headers["content-type"] in ["text/html", "application/json"] and res.json()["query_status"] == "not_found":
            demisto.results(
                {
                    "Type": entryTypes["note"],
                    "ContentsFormat": formats["json"],
                    "Contents": res.json(),
                    "HumanReadable": f"No results for SHA256: {file_sha256}",
                    "HumanReadableFormat": formats["markdown"],
                }
            )
        elif res.headers["content-type"] == "application/zip":
            demisto.results(fileResult(file_sha256, extract_zipped_buffer(res.content)))
        else:
            raise Exception
            # Handle like an exception
    except Exception:
        demisto.results({"Type": entryTypes["error"], "ContentsFormat": formats["text"], "Contents": str(res.content)})


""" COMMANDS MANAGER / SWITCH PANEL """

LOG(f"Command being called is {demisto.command()}")


def main():
    try:
        demisto_params = demisto.params()
        command = demisto.command()

        auth_key = demisto_params.get("credentials", {}).get("password")
        if not auth_key:
            raise ValueError("Missing required parameter Auth Key. Please set this parameter in the instance configuration.")

        HEADERS["Auth-Key"] = auth_key

        params = {
            "api_url": demisto_params["url"].rstrip("/"),
            "use_ssl": not demisto_params.get("insecure", False),
            "threshold": int(demisto_params.get("threshold", 1)),
            "create_relationships": demisto_params.get("create_relationships", True),
            "max_num_of_relationships": min(1000, int(demisto_params.get("max_num_of_relationships", 10))),
        }

        reliability = demisto_params.get("integrationReliability", DBotScoreReliability.C)

        if DBotScoreReliability.is_valid_type(reliability):
            params["reliability"] = DBotScoreReliability.get_dbot_score_reliability_from_str(reliability)
        else:
            Exception("Please provide a valid value for the Source Reliability parameter.")

        # Remove proxy if not set to true in params
        handle_proxy()

        if command == "test-module":
            # This is the call made when pressing the integration test button.
            test_module(**params)
            demisto.results("ok")
        elif command == "url":
            url_command(params)
        elif command == "domain":
            domain_command(params)
        elif command == "file":
            params["should_error"] = argToBoolean(demisto.params().get("should_error", True))
            file_command(params)
        elif command == "urlhaus-download-sample":
            urlhaus_download_sample_command(**params)

    # Log exceptions
    except Exception as exc:
        demisto.debug(traceback.format_exc())
        return_error(f'Failed to execute command "{command}".\nError: {exc}', error=exc)


if __name__ in ["__main__", "__builtin__", "builtins"]:
    main()