TAXII2 Server

This integration provides TAXII2 Services for system indicators (Outbound feed).

Data Enrichment & Threat Intelligence · TAXII Server

Details

IDTAXII2 Server
ProviderOpen Source
CategoryData Enrichment & Threat Intelligence
From Version6.1.0
Docker Imagedemisto/flask-nginx:1.0.0.11047721
Supported ModulesAgentix XSIAM

README

TAXII2 Server Integration

This integration provides TAXII2 Services for system indicators (outbound feed).
You can choose to use TAXII v2.0 or TAXII v2.1.

The TAXII2 Server integration is a long-running integration. For more information about long-running integrations, see the Cortex XSOAR 8 Cloud, Cortex XSOAR 8 On-prem or Cortex XSIAM documentation.

Configure Collections

Each TAXII collection in the integration is represented by a Cortex XSOAR indicator query.

The collections are defined by a JSON object in the following format:

{
  "collection1_name":{
    "query": "<Cortex XSOAR indicator query>",
    "description": "<Custom collection description>"
  },
  "collection2_name": "<Cortex XSOAR indicator query>"
}

You can add a collection description as is done in collection1_name, or enter only a collection query, as in collection2_name.

How to Access the TAXII2 Server

For Cortex XSOAR 6.x

Use one of the following options:

  • https://<xsoar_address>/instance/execute/<instance_name>/<taxii2_api_endpoint>/
  • http://<xsoar_address>:<listen_port>/<taxii2_api_endpoint>/

For Cortex XSOAR 8 On-prem, Cortex XSOAR 8 Cloud, or Cortex XSIAM

Use https://ext-<tenant>/xsoar/instance/execute/<instance-name>/<taxii2_api_endpoint>/

Note:

  • For Cortex XSOAR 8 On-prem, you need to add the ext- FQDN DNS record to map the Cortex XSOAR DNS name to the external IP address.
    For example, ext-xsoar.mycompany.com.
  • The instance name cannot be changed after saving the integration configuration.

Access the TAXII Service by Instance Name

To access the TAXII service by instance name, make sure Instance execute external is enabled.

In Cortex XSOAR 6.x:

  1. Navigate to Settings > About > Troubleshooting.
  2. In the Server Configuration section, verify that the instance.execute.external key is set to true. If this key does not exist, click + Add Server Configuration, add the instance.execute.external and set the value to true.

How to Use HTTPS

To use HTTPS, a certificate and private key have to be supplied in the integration configuration.

Set up Authentication

For Cortex XSOAR 8 Cloud Tenant or Cortex XSIAM Tenant

The TAXII2 Server integration running on a Cortex XSOAR 8 Cloud tenant or Cortex XSIAM tenant enables using basic authentication in the requests.
To enable basic authentication, a user and password must be supplied in the Credentials parameters in the integration configuration.
The server will then authenticate the requests by the Authorization header, expecting basic authentication encrypted in base64 to match the given credentials.

For Cortex XSOAR On-prem (6.x or 8) or When Using Engines

For Cortex XSOAR On-prem (6.x or 8) or when using engines, you can set up authentication using custom certificates. For more information on setting up a custom certificate for Cortex XSOAR 8 On-prem, see HTTPS with a signed certificate. For more information on setting up a custom certificate for Cortex XSOAR 6.x, see HTTPS with a Signed Certificate.

TAXII v2.0 API Endpoints

URL Method Response TAXII2 Documentation
/taxii/ GET Server Discovery Information. Server Discovery
/{api_root}/ GET Cortex XSOAR API root is threatintel. API Root Information
/{api_root}/collections/ GET All Cortex XSOAR collections that configure in Collection JSON parameter. Collections Resource
/{api_root}/collections/{collection_id}/ GET Cortex XSOAR Collection with given collection_id. Collection Response
/{api_root}/collections/{collection_id}/manifest/ GET Object manifests from the given collection. Objects Manifest Resource
/{api_root}/collections/{collection_id}/objects/ GET Objects (Cortex XSOAR Indicators) from the given collection. Object Resource

For more information, visit TAXII2 Documentation.

TAXII v2.1 API Endpoints

URL Method Response TAXII2 Documentation
/taxii2/ GET Server Discovery Information. Server Discovery
/{api_root}/ GET XSOAR API root is threatintel. API Root Information
/{api_root}/collections/ GET All Cortex XSOAR collections that configure in Collection JSON parameter. Collections Resource
/{api_root}/collections/{collection_id}/ GET Cortex XSOAR Collection with given collection_id. Collection Response
/{api_root}/collections/{collection_id}/manifest/ GET Object manifests from the given collection. Objects Manifest Resource
/{api_root}/collections/{collection_id}/objects/ GET Objects (Cortex XSOAR Indicators and Relationships) from the given collection. Object Resource

For more information, visit TAXII2 Documentation.

  • When retrieving Objects (Cortex XSOAR Indicators and Relationships) from a collection, use the next parameter to paginate through additional records.

Known Limitations

  • GET objects by ID is not allowed.
  • Filtering objects by ID or version not allowed.
  • POST and DELETE objects are not allowed. Cannot add or delete indicators using TAXII2 Server.

How UUIDs Work for TAXII2 in Cortex XSOAR


STIX Cyber Objects (SCO)

All STIX SCOs UUIDs follow STIX 2.1 guidelines and use UUID5 with STIX unique namespace
(00abedb4-aa42-466c-9c01-fed23315a9b7). This is used so all SCOs created have persistent UUID across all producers.

STIX Domain Objects (SDO)

Unlike SCOs, STIX 2.1 specs for SDOs require a UUID4. While this solution works if the UUID is part of the database,
it is not the case in Cortex XSOAR. If the SDO already has a unique UUID stored it will use it, if not it will generate a unique and persistent UUID using the following method.

A general UUID5 is created using the NameSpace_URL as follows:

PAWN_UUID = uuid.uuid5(uuid.NAMESPACE_URL, 'https://www.paloaltonetworks.com')

The generated UUID is then used to create a unique UUID5 per customer:

UNIQUE_UUID = uuid.uuid5(PAWN_UUID, <UniqueCostumerString>)

We then use this UUID as a base namespace to generate UUIDs for SDOs following the STIX 2.1 specs. Using this method,
we create unique and persistent UUIDs per customer.

Cortex XSOAR TIM Extension Fields


When selected in the integration settings (Cortex XSOAR Extension Fields) the TAXII2 integration will generate an extension object and an extension attribute that holds Cortex XSOAR additional
TIM fields (system generated and custom). An example of these two related objects:

{
  "id": "extension-definition--<UUID>",
  "type": "extension-definition",
  "spec_version": "2.1",
  "name": "XSOAR TIM <Cortex XSOAR Type>",
  "description": "This schema adds TIM data to the object",
  "created": "<creation date>",
  "modified": "<modification date>",
  "created_by_ref": "identity--<UUID of creator>",
  "schema": "https://github.com/demisto/content/blob/4265bd5c71913cd9d9ed47d9c37d0d4d3141c3eb/Packs/TAXIIServer/doc_files/XSOAR_indicator_schema.json",
  "version": "1.0",
  "extension_types": ["property-extension"]
},
{
    "type": "ipv4-addr",
    "spec_version": "2.1",
    "id": "ipv4-addr--2f689bf9-0ff2-545f-aa61-e495eb8cecc7",
    "value": "8.8.8.8",
    "extensions":{
        "extension-definition--<UUID>": {
           "Extension_type": "property_extension",
           "Field_1": "Value1",
           "Field_2": "Value2",
           "Field_3": "Value3"
        }
    }
}

Performance Benchmark

Indicators Amount Request time (seconds)
10,000 5-10
50,000 30-40
100,000 50-90

Microsoft Sentinel Configuration Guide

Configure the TAXII2 Server Instance

  1. Set TAXII2 Server version to 2.0 (The integration currently doesn’t work with Microsoft Sentinel in TAXII Version 2.1).

  2. Under STIX types for STIX indicator Domain Object select the indicator types you want to ingest.

  3. Set the Listen Port and Collection JSON to your linking.

Find the Information Required for the Sentinel TAXII Connector

For Cortex XSOAR 6.x

  1. All your server info can be found by running !taxii-server-info, the default API root for you server will usually be - https://<xsoar-server>/instance/execute/<instance_name>/threatintel/
  2. You can use the !taxii-server-list-collections command in order to get a list of your server’s collections and their IDs. You can also do it manually by running curl https://<xsoar-server>/instance/execute/<instance_name>/threatintel/collections/ | jq . to get a list of the collections available and on your TAXII server. From the list, copy the correct ID of the collection you want to ingest.

For Cortex XSOAR 8 On-prem, Cortex XSOAR Cloud, or Cortex XSIAM

  1. All your server info can be found by running !taxii-server-info, the default API root for you server will usually be - https://ext-<tenant>.crtx.<region>.paloaltonetworks.com/xsoar/instance/execute/<instance-name>/threatintel/
  2. You can use the !taxii-server-list-collections command in order to get a list of your server’s collections and their IDs. You can also do it manually by running curl https://ext-<tenant>.crtx.<region>.paloaltonetworks.com/xsoar/instance/execute/<instance-name>/threatintel/collections/ | jq . to get a list of the collections available and on your TAXII server. From the list, copy the correct ID of the collection you want to ingest.

Response Example:

  {
    "collections": [
      {
        "can_read": true,
        "can_write": false,
        "description": "",
        "id": "3709e6df-bbe1-5ece-b683-e99a42f31fce",
        "media_types": [
          "application/vnd.oasis.stix+json; version=2.0"
        ],
        "query": "type:IP",
        "title": "AllIPs"
      },
      {
        "can_read": true,
        "can_write": false,
        "description": "",
        "id": "16763019-5ee6-59bc-b4d9-be586235b308",
        "media_types": [
          "application/vnd.oasis.stix+json; version=2.0"
        ],
        "query": "type:File",
        "title": "AllHashes"
      }
    ]
  }

Set up the Microsoft Sentinel TAXII Connector

Now that we have the API root URL and the collection ID we can configure the Threat intelligence - TAXII Connector in Microsoft Sentinel.

Paste your API root URL in the field marked API Root URL and the desired collection ID we got in step (2) under Collection ID.

Example:
Microsoft Sentinel TI Configuration

Commands

You can execute these commands from the Cortex XSOAR CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

taxii-server-list-collections


Returns all the collections.

Base Command

taxii-server-list-collections

Input

There are no input arguments for this command.

Context Output

Path Type Description
TAXIIServer.Collection.id String The collection ID.
TAXIIServer.Collection.query String The collection query.
TAXIIServer.Collection.title String The collection title.
TAXIIServer.Collection.description String The collection description.

Command example


#### Context Example

```json
{
    "TAXIIServer": {
        "Collection": {
            "can_read": true,
            "can_write": false,
            "description": "",
            "id": "2eb7bfae-7739-5863-9b00-1681309c3d8c",
            "media_types": [
                "application/stix+json;version=2.1"
            ],
            "query": "",
            "title": "ALL"
        }
    }
}

Human Readable Output

Collections

id title query description
2eb7bfae-7739-5863-9b00-1681309c3d8c ALL    

taxii-server-info


Returns the TAXII server info, default URL, title, etc.

Base Command

taxii-server-info

Input

There are no input arguments for this command.

Context Output

Path Type Description
TAXIIServer.ServerInfo.title String The server title
TAXIIServer.ServerInfo.api_roots Unknown The server API roots URLs.
TAXIIServer.ServerInfo.default String The default URL.
TAXIIServer.ServerInfo.description String The server description

Command Example


#### Context Example

```json
{
    "TAXIIServer": {
        "ServerInfo": {
            "api_roots": [
                "https://foo.cooo.com/inc/threatintel/"
            ],
            "default": "https://foo.cooo.com/inc/threatintel/",
            "description": "This integration provides TAXII Services for system indicators (Outbound feed).",
            "title": "Cortex XSOAR TAXII2 Server"
        }
    }
}

Human Readable Output

In case the default URL is incorrect, you can override it by setting the “TAXII2 Service URL Address” field in the integration configuration

Server Info

api_roots default description title
https://foo.cooo.com/inc/threatintel/ https://foo.cooo.com/inc/threatintel/ This integration provides TAXII Services for system indicators (Outbound feed). Cortex XSOAR TAXII2 Server

Troubleshooting

429 Too Many Requests error

NGINX prevents concurrent builds of the same cache entry. If multiple requests for the same cache entry (matching the URL and parameters) arrive simultaneously, NGINX builds the entry for the first request and rejects the others with an HTTP 429 Too Many Requests rather than queuing them. Requests for different entries are still processed in parallel. If a request triggers a refresh of an existing cache entry, the previous data is served (HTTP 200) with no 429.

This is expected behavior. Retry the request after a short delay; once the initial build finishes populating the cache, retries are served from the cache (HTTP 200).

[Errno 98] Address in use error

Each instance uses three consecutive ports: the configured Listen Port, port + 1, and port + 2. NGINX listens on the configured port (public), the Python process listens on port + 1, and NGINX uses port + 2 internally for its fail-fast cache fetch tier. For example, if configured for port 9009, ports 9009, 9010, and 9011 must all be free. Ensure no other instance uses a Listen Port within 2 of another, or an [Errno 98] Address in use error will occur. When running without --network=host, ports + 1 and + 2 are not exposed to the machine.

Configuration parameters

  • longRunning — Long Running Instance
  • version — TAXII2 Server version (required)
  • credentials — Username
  • collections — Collection JSON (required)
  • fields_filter — Cortex XSOAR Extension fields
  • res_size — Response Size (required)
  • certificate — Certificate (Required for HTTPS)
  • key — Private Key (Required for HTTPS)
  • hsts_header — Add HSTS header
  • service_address — TAXII2 Service URL Address
  • longRunningPort — Listen Port (required)
  • nginx_global_directives — NGINX Global Directives
  • nginx_server_conf — NGINX Server Conf
  • cache_refresh_rate — Refresh Rate
  • cache_lock_timeout — Cache Lock Timeout (Deprecated)
  • cache_lock_age — Cache Lock Age (Deprecated)
  • cache_404_ttl — Cache 404 TTL
  • cache_default_ttl — Cache Default TTL
  • provide_as_indicator — STIX types for STIX indicator Domain Object
  • cache_duration_hours — Search-after cache duration hours (required)

Commands (2)

  • taxii-server-info

    Returns the TAXII server info, default URL, title, etc.

  • taxii-server-list-collections

    Returns all the collections.

import copy
import json
from freezegun import freeze_time

import demistomock as demisto
import pytest
from requests.auth import _basic_auth_str
from TAXII2Server import APP, MEDIA_TYPE_STIX_V20, MEDIA_TYPE_TAXII_V20, TAXII2Server, create_fields_list, create_query, uuid

HEADERS = {
    "Authorization": _basic_auth_str("username", "password"),
    "Accept": "application/taxii+json",
}


@pytest.fixture
def taxii2_server_v20(mocker):
    mocker.patch.object(demisto, "getLicenseID", return_value="test")
    server = TAXII2Server(
        url_scheme="http",
        host="demisto",
        port=7000,
        collections={"Collection1": "type:IP", "Collection2": 'sourceBrands:"Some Feed"'},
        certificate="",
        private_key="",
        http_server=True,
        credentials={"identifier": "username", "password": "password"},
        version="2.0",
        service_address=None,
        fields_to_present=set(),
    )

    return server


@pytest.fixture
def taxii2_server_v21(mocker):
    mocker.patch.object(demisto, "getLicenseID", return_value="test")
    server = TAXII2Server(
        url_scheme="http",
        host="demisto",
        port=7000,
        collections={"Collection1": "type:IP", "Collection2": {"query": 'sourceBrands:"Some Feed"', "description": "Test desc"}},
        certificate="",
        private_key="",
        http_server=True,
        credentials={"identifier": "username", "password": "password"},
        version="2.1",
        service_address=None,
        fields_to_present=set(),
    )

    return server


def util_load_json(path):
    with open(path, encoding="utf-8") as f:
        return json.loads(f.read())


@pytest.mark.parametrize(
    "fields, result",
    [
        ("", {"name", "type"}),
        ("all", set()),
        ("name,type,sha1", {"name", "type", "sha1"}),
        ("value,type,sha1", {"name", "type", "sha1"}),
        ("value,indicator_type,createdTime", {"name", "type", "createdTime"}),
    ],
)
def test_create_fields_list(fields, result):
    """
    Given
        fields list parameter, expected result
    When
        User enters filter_field param
    Then
        Validate right result returned
    """
    assert result == create_fields_list(fields)


@pytest.mark.parametrize("headers", [{"Authorization": _basic_auth_str("user", "pwd")}, {}])
def test_taxii_wrong_auth(mocker, headers, taxii2_server_v20):
    """
    Given
        Taxii server v2.0
    When
        Getting server discovery, with wrong auth
    Then
        Validate that the error and status code right
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch.object(demisto, "error")
    mocker.patch.object(demisto, "updateModuleHealth")
    with APP.test_client() as test_client:
        response = test_client.get("/taxii/", headers=headers)
        assert response.status_code == 401
        assert response.json == {"title": "Authorization failed"}


@pytest.mark.parametrize(
    "headers",
    [
        {"Authorization": _basic_auth_str("username", "password")},
        {"Authorization": _basic_auth_str("username", "password"), "Accept": "wrong_type"},
    ],
)
def test_taxii_wrong_accept(mocker, headers, taxii2_server_v20):
    """
    Given
        Taxii server v2.0
    When
        Getting server discovery, with wrong accept header
    Then
        Validate that the error and status code right
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch.object(demisto, "error")
    mocker.patch.object(demisto, "updateModuleHealth")
    with APP.test_client() as test_client:
        response = test_client.get("/taxii/", headers=headers)
        assert response.status_code == 406


def test_taxii20_server_discovery(mocker, taxii2_server_v20):
    """
    Given
        Taxii server v2.0
    When
        Getting server discovery
    Then
        Validate that the discovery output as expected
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    with APP.test_client() as test_client:
        response = test_client.get("/taxii/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0"
        assert response.json.get("default") == "http://demisto:7000/threatintel/"


def test_taxii21_server_discovery(mocker, taxii2_server_v21):
    """
    Given
        Taxii server v2.1
    When
        Call server discovery api request
    Then
        Validate that the discovery output as expected
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    with APP.test_client() as test_client:
        response = test_client.get("/taxii/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json.get("default") == "http://demisto:7000/threatintel/"


def test_taxii20_api_root(mocker, taxii2_server_v20):
    """
    Given
        TAXII v2.0 server, api_root
    When
        Call api_root api request
    Then
        Validate that the api_root information returned as expected
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0"
        assert response.json.get("title") == "Cortex XSOAR TAXII2 Server ThreatIntel"


def test_taxii_wrong_api_root(mocker, taxii2_server_v20):
    """
    Given
        Taxii server v2.0, Not exiting api_root
    When
        Getting api root information, for wrong api_root
    Then
        Validate that the error and status code right
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch.object(demisto, "error")
    mocker.patch.object(demisto, "updateModuleHealth")
    with APP.test_client() as test_client:
        response = test_client.get("/not_exsisting_api_root/", headers=HEADERS)
        assert response.status_code == 404
        assert response.json.get("title") == "Unknown API Root"


def test_taxii20_status(mocker, taxii2_server_v20):
    """
    Given
        Status api call
    When
        Calling a status request
    Then
        Validate the error returned.
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/status/1223456/", headers=HEADERS)
        assert response.status_code == 404


def test_taxii20_collections(mocker, taxii2_server_v20):
    """
    Given
        TAXII Server v2.0
    When
        Calling collections api request
    Then
        Validate that collections returned as expected
    """
    collections = util_load_json("test_data/collections20.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0"
        assert response.json == collections


def test_taxii21_collections(mocker, taxii2_server_v21):
    """
    Given
        TAXII Server v2.1
    When
        Calling collections api request
    Then
        Validate that collections returned as expected
    """
    collections = util_load_json("test_data/collections21.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json == collections


def test_taxii20_collection(mocker, taxii2_server_v20):
    """
    Given
        TAXII Server v2.0, collection_id
    When
        Calling collection by id api request
    Then
        Validate that right collection returned
    """
    collections = util_load_json("test_data/collections20.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0"
        assert response.json == collections.get("collections")[0]


def test_taxii20_get_collections(mocker, taxii2_server_v20):
    from TAXII2Server import get_server_collections_command

    collections = taxii2_server_v20.get_collections()

    integration_context = {"collections": collections["collections"]}
    result = get_server_collections_command(integration_context=integration_context)

    assert result.outputs == integration_context["collections"]


def test_taxii20_get_server_info(mocker, taxii2_server_v20):
    from TAXII2Server import get_server_info_command

    integration_context = {}
    integration_context["server_info"] = taxii2_server_v20.get_discovery_service(instance_execute=True)
    default_url = integration_context["server_info"]["default"]
    assert default_url == "https://demisto/instance/execute/threatintel/"

    result = get_server_info_command(integration_context=integration_context)

    assert result.outputs == integration_context["server_info"]


def test_taxii20_get_server_info_demisto_version(mocker):
    """
    Given
        TAXII Server v2.0, Integration context.
    When
        Calling get-server-info command
    Then
        Validate that the correct default URL is returned

    """
    from TAXII2Server import get_server_info_command

    integration_context = {}
    integration_context["server_info"] = {
        "api_roots": ["https://www.example.com/path/to/resource"],
        "default": "https://www.example.com/path/to/resource",
    }
    mocker.patch("CommonServerPython.get_demisto_version", return_value={"version": "8.1.0", "buildNumber": "12345"})

    results = get_server_info_command(integration_context=integration_context)

    assert results.outputs["default"] == "https://ext-www.example.com/xsoar/path/to/resource"


def test_taxii21_collection(mocker, taxii2_server_v21):
    """
    Given
        TAXII Server v2.1, collection_id
    When
        Calling collection by id api request
    Then
        Validate that right collection returned
    """
    collections = util_load_json("test_data/collections21.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json == collections.get("collections")[0]


def test_taxii_wrong_collection_id(mocker, taxii2_server_v21):
    """
    Given
        Taxii server v2.1, Not exiting collection_id
    When
        Getting collection information, for wrong collection_id
    Then
        Validate that the error and status code right
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch.object(demisto, "error")
    mocker.patch.object(demisto, "updateModuleHealth")
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/not_exsisting_collection_id/", headers=HEADERS)
        assert response.status_code == 404
        assert response.json.get("title") == "Unknown Collection"


def test_taxii20_manifest(mocker, taxii2_server_v20):
    """
    Given
        TAXII Server v2.0, collection_id, range
    When
        Calling manifest api request for given collection
    Then
        Validate that right manifest returned.
    """
    iocs = util_load_json("test_data/ip_iocs.json")
    manifest = util_load_json("test_data/manifest20.json")
    headers = copy.deepcopy(HEADERS)
    headers["Range"] = "items 0-4"
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    mocker.patch.object(demisto, "params", return_value={"res_size": "100"})
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/", headers=headers)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0"
        assert response.json == manifest


def test_taxii21_manifest(mocker, taxii2_server_v21):
    """
    Given
        TAXII Server v2.1, collection_id
    When
        Calling manifest api request for given collection
    Then
        Validate that right manifest returned.
    """
    iocs = util_load_json("test_data/ip_iocs.json")
    manifest = util_load_json("test_data/manifest21.json")
    mocker.patch.object(demisto, "params", return_value={"res_size": "100"})
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    with APP.test_client() as test_client:
        response = test_client.get(
            "/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/?limit=4", headers=HEADERS
        )
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json == manifest


def test_taxii20_objects(mocker, taxii2_server_v20):
    """
    Given
        TAXII Server v2.0, collection_id, content-range
    When
        Calling get objects api request for given collection
    Then
        Validate that right objects are returned.
    """
    iocs = util_load_json("test_data/ip_iocs.json")
    objects = util_load_json("test_data/objects20.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777")
    headers = copy.deepcopy(HEADERS)
    headers["Content-Range"] = "items 0-2/5"
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    mocker.patch.object(demisto, "params", return_value={"res_size": "100"})
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/objects/", headers=headers)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.stix+json; version=2.0"
        assert response.json == objects
        assert response.headers.get("Content-Range") == "items 0-3/5"


def test_taxii20_indicators_objects(mocker, taxii2_server_v20):
    """
    Given
        TAXII Server v2.0, collection_id, content-range, types_for_indicator_sdo with all types included.
    When
        Calling get objects api request for given collection
    Then
        Validate that right objects are returned and no extensions are returned.
    """
    iocs = util_load_json("test_data/ip_iocs.json")
    objects = util_load_json("test_data/objects20-indicators.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch(
        "TAXII2Server.SERVER.types_for_indicator_sdo",
        ["ipv4-addr", "domain-name", "ipv6-addr", "user-account", "email-addr", "windows-registry-key", "file", "url"],
    )
    mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777")
    headers = copy.deepcopy(HEADERS)
    headers["Content-Range"] = "items 0-2/5"
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    mocker.patch.object(demisto, "params", return_value={"res_size": "100"})
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/objects/", headers=headers)
        assert response.status_code == 200
        assert response.content_type == "application/vnd.oasis.stix+json; version=2.0"
        assert response.json == objects
        assert response.headers.get("Content-Range") == "items 0-3/5"


@pytest.mark.parametrize(
    "demisto_iocs_file,res_file,query_type",
    [
        ("malware_iocs", "objects21_malware", "malware"),
        ("file_iocs", "objects21_file", "file"),
        ("domain_iocs", "objects21_domain", "domain-name,attack-pattern"),
    ],
)
def test_taxii21_objects(mocker, taxii2_server_v21, demisto_iocs_file, res_file, query_type):
    """
    Given
        TAXII Server v2.1, collection_id, limit, next, type parameter
    When
        Calling get objects api request for given collection
    Then
        Validate that right objects are returned.
    """
    iocs = util_load_json(f"test_data/{demisto_iocs_file}.json")
    objects = util_load_json(f"test_data/{res_file}.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777")
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    mocker.patch.object(demisto, "params", return_value={"res_size": "100"})
    with APP.test_client() as test_client:
        response = test_client.get(
            f"/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/objects/?match[type]={query_type}&limit=2&next=1",
            headers=HEADERS,
        )
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json == objects


@pytest.mark.parametrize("api_request", ["objects", "manifest"])
def test_taxii21_bad_request(mocker, taxii2_server_v21, api_request):
    """
    Given
        TAXII Server v2.1, non-supported filter.
    When
        Calling get objects or manifest api request for given collection
    Then
        Validate that right error returned.
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch.object(demisto, "error")
    mocker.patch.object(demisto, "params", return_value={"res_size": "2500"})
    mocker.patch.object(demisto, "updateModuleHealth")
    with APP.test_client() as test_client:
        response = test_client.get(
            f"/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/{api_request}/?match[version]=3", headers=HEADERS
        )
        assert response.status_code == 404
        assert response.content_type == "application/taxii+json;version=2.1"
        assert "Filtering by ID or version is not supported." in response.json.get("description")


@pytest.mark.parametrize("api_request", ["objects", "manifest"])
def test_taxii20_bad_content_range(mocker, taxii2_server_v20, api_request):
    """
    Given
        TAXII Server v2.0, non-supported range.
    When
        Calling get objects or manifest api request for given collection
    Then
        Validate that right error returned.
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v20)
    mocker.patch.object(demisto, "params", return_value={"res_size": "2500"})
    headers = copy.deepcopy(HEADERS)
    headers["Content-Range"] = "items 8-2/10"
    with APP.test_client() as test_client:
        response = test_client.get(
            f"/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/{api_request}/", headers=headers
        )
        assert response.status_code == 416


@pytest.mark.parametrize(
    "res_file,fields,has_extension",
    [("objects21_no_extention_file", {"name", "type"}, False), ("objects21_spec_fields_file", {"sha1"}, True)],
)
def test_taxii21_objects_filtered_params(mocker, taxii2_server_v21, res_file, fields, has_extension):
    """
    Given
        TAXII Server v2.1, collection_id, type parameter, filtered_fields params
    When
        Calling get objects api request for given collection
    Then
        Validate that right objects are returned.
    """
    iocs = util_load_json("test_data/file_iocs_filter_test.json")
    objects = util_load_json(f"test_data/{res_file}.json")
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch("TAXII2Server.SERVER.fields_to_present", fields)
    mocker.patch("TAXII2Server.SERVER.has_extension", has_extension)
    mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777")
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    mocker.patch.object(demisto, "params", return_value={"res_size": "7"})
    with APP.test_client() as test_client:
        response = test_client.get(
            "/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/objects/?match[type]=file", headers=HEADERS
        )
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json == objects


@pytest.mark.parametrize("header", (MEDIA_TYPE_TAXII_V20, MEDIA_TYPE_STIX_V20))
def test_taxii21_with_taxii20_header(mocker, taxii2_server_v21, header: str):
    """
    Given
        a TAXII 2.1 server
    When
        calling /taxii2/ with TAXII 2.0 header
    Then
        validate that an appropriate error is returned
    """
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    with APP.test_client() as test_client:
        response = test_client.get("/taxii2/", headers=HEADERS | {"Accept": header})
        assert response.status_code == 406


@pytest.mark.parametrize(
    "query,types,added_after,expected_response",
    (
        ("my custom query", [], "2023-07-06T10:57:15.133309Z", 'my custom query and modified:>="2023-07-06T10:57:15.133309Z"'),
        (
            "my custom query",
            ["file"],
            "2023-07-06T10:57:15.133309Z",
            '(my custom query) and (type:"File") and modified:>="2023-07-06T10:57:15.133309Z"',
        ),
        (
            "my custom query",
            ["file", "domain"],
            "2023-07-06T10:57:15.133309Z",
            '(my custom query) and (type:"File" or type:"domain") and modified:>="2023-07-06T10:57:15.133309Z"',
        ),
    ),
)
def test_create_query(query: str, types: list[str], added_after, expected_response: str):
    """
    Given
        a query, types to match, and added after.
    When
        calling create_query
    Then
        Validate that right query is returned.
    """
    assert create_query(query, types, added_after) == expected_response


@pytest.mark.parametrize(
    "endpoint",
    [
        ("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/?limit=4&added_after=2022-06-03T00:00:00Z"),
        (
            "/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/?limit=4&added_after=2022-06-03T13:54:27.234765Z"
        ),
    ],
)
def test_parse_manifest_and_object_args_with_valid_date(mocker, taxii2_server_v21, endpoint):
    """
    Given
        case 1: endpoint with utc date format.
        case 2: endpoint with stix date format.
    When
        testing parse_manifest_and_object_args.
    Then
        Ensure that Should parsing was done correctly and a valid results message was returned.
    """
    iocs = util_load_json("test_data/ip_iocs.json")
    manifest = util_load_json("test_data/manifest21.json")
    mocker.patch.object(demisto, "params", return_value={"res_size": "100"})
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch.object(demisto, "searchIndicators", return_value=iocs)
    with APP.test_client() as test_client:
        response = test_client.get(endpoint, headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        assert response.json == manifest


def test_taxii21_objects_with_relationships(mocker, taxii2_server_v21):
    """
    Given
        TAXII Server v2.1, collection_id, no_extension
    When
        Calling get objects api request for given collection
    Then
        Validate that right objects are returned.
        Ensure that searchRelationships is called with the expected arguments.

    """
    from CommonServerPython import get_demisto_version

    get_demisto_version._version = None  # clear cache between runs of the test
    mocker.patch.object(demisto, "demistoVersion", return_value={"version": "6.6.0"})
    mocker.patch("TAXII2Server.SERVER", taxii2_server_v21)
    mocker.patch("TAXII2Server.SERVER.has_extension", False)
    mocker.patch("TAXII2Server.SERVER.fields_to_present", {"name", "type"})
    mock_search_relationships_response = util_load_json("test_data/searchRelationships-response.json")
    mocker.patch.object(demisto, "searchRelationships", return_value=mock_search_relationships_response)

    objects = util_load_json("test_data/objects21_ip_with_relationships.json")
    mock_iocs = util_load_json("test_data/sort_ip_iocs.json")
    mock_entity_b_iocs = util_load_json("test_data/entity_b_iocs.json")
    mocker.patch.object(demisto, "searchIndicators", side_effect=[mock_iocs, mock_entity_b_iocs])

    mocker.patch.object(demisto, "params", return_value={"res_size": "20"})
    with APP.test_client() as test_client:
        response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/objects/", headers=HEADERS)
        assert response.status_code == 200
        assert response.content_type == "application/taxii+json;version=2.1"
        demisto.searchRelationships.assert_called_once_with(
            {
                "entities": [
                    "1.1.1.1",
                    "3.3.3.3",
                    "f1412386aa8db2579aff2636cb9511cacc5fd9880ecab60c048508fbe26ee4d9",
                    "2.2.2.2",
                    "4.4.4.4",
                    "bad-domain.com",
                ]
            }
        )
        assert response.json == objects


@freeze_time("2025-01-01 12:00:00 UTC")
def test_remove_old_cache(mocker):
    """
    Given
        An integration context with search_after_cache entries.
    When
        remove_old_cache is called.
    Then
        Validate that old cache entries are removed and new ones are kept.
    """
    from TAXII2Server import remove_old_cache

    mocker.patch.object(demisto, "params", return_value={"cache_duration_hours": 24})

    integration_context = {
        "search_after_cache": {
            "collection1": {
                "0": {"search_after": ["val1"], "last_updated": "2024-12-31T10:00:00Z"},  # Expired (26 hours old)
                "10": {"search_after": ["val2"], "last_updated": "2025-01-01T10:00:00Z"},  # Not expired (2 hours old)
            },
            "collection2": {
                "0": {"search_after": ["val3"], "last_updated": "2024-12-30T10:00:00Z"},  # Expired (50 hours old)
                "20": {"search_after": ["val4"], "last_updated": "2025-01-01T11:00:00Z"},  # Not expired (1 hour old)
            },
            "collection3": {
                "0": {"search_after": ["val5"], "last_updated": "invalid-date"},  # Invalid date, should be kept
            },
            "collection4": {
                "0": {"search_after": ["val6"]},  # Missing last_updated, should be kept
            },
        }
    }

    remove_old_cache(integration_context)

    expected_context = {
        "search_after_cache": {
            "collection1": {
                "10": {"search_after": ["val2"], "last_updated": "2025-01-01T10:00:00Z"},
            },
            "collection2": {
                "20": {"search_after": ["val4"], "last_updated": "2025-01-01T11:00:00Z"},
            },
            "collection3": {
                "0": {"search_after": ["val5"], "last_updated": "invalid-date"},
            },
            "collection4": {
                "0": {"search_after": ["val6"]},
            },
        }
    }
    assert integration_context == expected_context