TAXII2 Server
This integration provides TAXII2 Services for system indicators (Outbound feed).
Data Enrichment & Threat Intelligence · TAXII Server
Details
| ID | TAXII2 Server |
|---|---|
| Provider | Open Source |
| Category | Data Enrichment & Threat Intelligence |
| From Version | 6.1.0 |
| Docker Image | demisto/flask-nginx:1.0.0.11047721 |
| Supported Modules | Agentix XSIAM |
README
TAXII2 Server Integration
This integration provides TAXII2 Services for system indicators (outbound feed).
You can choose to use TAXII v2.0 or TAXII v2.1.
The TAXII2 Server integration is a long-running integration. For more information about long-running integrations, see the Cortex XSOAR 8 Cloud, Cortex XSOAR 8 On-prem or Cortex XSIAM documentation.
Configure Collections
Each TAXII collection in the integration is represented by a Cortex XSOAR indicator query.
The collections are defined by a JSON object in the following format:
{
"collection1_name":{
"query": "<Cortex XSOAR indicator query>",
"description": "<Custom collection description>"
},
"collection2_name": "<Cortex XSOAR indicator query>"
}
You can add a collection description as is done in collection1_name, or enter only a collection query, as in collection2_name.
How to Access the TAXII2 Server
For Cortex XSOAR 6.x
Use one of the following options:
https://<xsoar_address>/instance/execute/<instance_name>/<taxii2_api_endpoint>/http://<xsoar_address>:<listen_port>/<taxii2_api_endpoint>/
For Cortex XSOAR 8 On-prem, Cortex XSOAR 8 Cloud, or Cortex XSIAM
Use https://ext-<tenant>/xsoar/instance/execute/<instance-name>/<taxii2_api_endpoint>/
Note:
- For Cortex XSOAR 8 On-prem, you need to add the
ext-FQDN DNS record to map the Cortex XSOAR DNS name to the external IP address.
For example,ext-xsoar.mycompany.com. - The instance name cannot be changed after saving the integration configuration.
Access the TAXII Service by Instance Name
To access the TAXII service by instance name, make sure Instance execute external is enabled.
In Cortex XSOAR 6.x:
- Navigate to Settings > About > Troubleshooting.
- In the Server Configuration section, verify that the instance.execute.external key is set to true. If this key does not exist, click + Add Server Configuration, add the instance.execute.external and set the value to true.
How to Use HTTPS
To use HTTPS, a certificate and private key have to be supplied in the integration configuration.
Set up Authentication
For Cortex XSOAR 8 Cloud Tenant or Cortex XSIAM Tenant
The TAXII2 Server integration running on a Cortex XSOAR 8 Cloud tenant or Cortex XSIAM tenant enables using basic authentication in the requests.
To enable basic authentication, a user and password must be supplied in the Credentials parameters in the integration configuration.
The server will then authenticate the requests by the Authorization header, expecting basic authentication encrypted in base64 to match the given credentials.
For Cortex XSOAR On-prem (6.x or 8) or When Using Engines
For Cortex XSOAR On-prem (6.x or 8) or when using engines, you can set up authentication using custom certificates. For more information on setting up a custom certificate for Cortex XSOAR 8 On-prem, see HTTPS with a signed certificate. For more information on setting up a custom certificate for Cortex XSOAR 6.x, see HTTPS with a Signed Certificate.
TAXII v2.0 API Endpoints
| URL | Method | Response | TAXII2 Documentation |
|---|---|---|---|
| /taxii/ | GET | Server Discovery Information. | Server Discovery |
| /{api_root}/ | GET | Cortex XSOAR API root is threatintel. | API Root Information |
| /{api_root}/collections/ | GET | All Cortex XSOAR collections that configure in Collection JSON parameter. | Collections Resource |
| /{api_root}/collections/{collection_id}/ | GET | Cortex XSOAR Collection with given collection_id. | Collection Response |
| /{api_root}/collections/{collection_id}/manifest/ | GET | Object manifests from the given collection. | Objects Manifest Resource |
| /{api_root}/collections/{collection_id}/objects/ | GET | Objects (Cortex XSOAR Indicators) from the given collection. | Object Resource |
For more information, visit TAXII2 Documentation.
TAXII v2.1 API Endpoints
| URL | Method | Response | TAXII2 Documentation |
|---|---|---|---|
| /taxii2/ | GET | Server Discovery Information. | Server Discovery |
| /{api_root}/ | GET | XSOAR API root is threatintel. | API Root Information |
| /{api_root}/collections/ | GET | All Cortex XSOAR collections that configure in Collection JSON parameter. | Collections Resource |
| /{api_root}/collections/{collection_id}/ | GET | Cortex XSOAR Collection with given collection_id. | Collection Response |
| /{api_root}/collections/{collection_id}/manifest/ | GET | Object manifests from the given collection. | Objects Manifest Resource |
| /{api_root}/collections/{collection_id}/objects/ | GET | Objects (Cortex XSOAR Indicators and Relationships) from the given collection. | Object Resource |
For more information, visit TAXII2 Documentation.
- When retrieving Objects (Cortex XSOAR Indicators and Relationships) from a collection, use the next parameter to paginate through additional records.
Known Limitations
- GET objects by ID is not allowed.
- Filtering objects by ID or version not allowed.
- POST and DELETE objects are not allowed. Cannot add or delete indicators using TAXII2 Server.
How UUIDs Work for TAXII2 in Cortex XSOAR
STIX Cyber Objects (SCO)
All STIX SCOs UUIDs follow STIX 2.1 guidelines and use UUID5 with STIX unique namespace
(00abedb4-aa42-466c-9c01-fed23315a9b7). This is used so all SCOs created have persistent UUID across all producers.
STIX Domain Objects (SDO)
Unlike SCOs, STIX 2.1 specs for SDOs require a UUID4. While this solution works if the UUID is part of the database,
it is not the case in Cortex XSOAR. If the SDO already has a unique UUID stored it will use it, if not it will generate a unique and persistent UUID using the following method.
A general UUID5 is created using the NameSpace_URL as follows:
PAWN_UUID = uuid.uuid5(uuid.NAMESPACE_URL, 'https://www.paloaltonetworks.com')
The generated UUID is then used to create a unique UUID5 per customer:
UNIQUE_UUID = uuid.uuid5(PAWN_UUID, <UniqueCostumerString>)
We then use this UUID as a base namespace to generate UUIDs for SDOs following the STIX 2.1 specs. Using this method,
we create unique and persistent UUIDs per customer.
Cortex XSOAR TIM Extension Fields
When selected in the integration settings (Cortex XSOAR Extension Fields) the TAXII2 integration will generate an extension object and an extension attribute that holds Cortex XSOAR additional
TIM fields (system generated and custom). An example of these two related objects:
{
"id": "extension-definition--<UUID>",
"type": "extension-definition",
"spec_version": "2.1",
"name": "XSOAR TIM <Cortex XSOAR Type>",
"description": "This schema adds TIM data to the object",
"created": "<creation date>",
"modified": "<modification date>",
"created_by_ref": "identity--<UUID of creator>",
"schema": "https://github.com/demisto/content/blob/4265bd5c71913cd9d9ed47d9c37d0d4d3141c3eb/Packs/TAXIIServer/doc_files/XSOAR_indicator_schema.json",
"version": "1.0",
"extension_types": ["property-extension"]
},
{
"type": "ipv4-addr",
"spec_version": "2.1",
"id": "ipv4-addr--2f689bf9-0ff2-545f-aa61-e495eb8cecc7",
"value": "8.8.8.8",
"extensions":{
"extension-definition--<UUID>": {
"Extension_type": "property_extension",
"Field_1": "Value1",
"Field_2": "Value2",
"Field_3": "Value3"
}
}
}
Performance Benchmark
| Indicators Amount | Request time (seconds) |
|---|---|
| 10,000 | 5-10 |
| 50,000 | 30-40 |
| 100,000 | 50-90 |
Microsoft Sentinel Configuration Guide
Configure the TAXII2 Server Instance
-
Set TAXII2 Server version to 2.0 (The integration currently doesn’t work with Microsoft Sentinel in TAXII Version 2.1).
-
Under STIX types for STIX indicator Domain Object select the indicator types you want to ingest.
-
Set the Listen Port and Collection JSON to your linking.
Find the Information Required for the Sentinel TAXII Connector
For Cortex XSOAR 6.x
- All your server info can be found by running
!taxii-server-info, the default API root for you server will usually be -https://<xsoar-server>/instance/execute/<instance_name>/threatintel/ - You can use the
!taxii-server-list-collectionscommand in order to get a list of your server’s collections and their IDs. You can also do it manually by runningcurl https://<xsoar-server>/instance/execute/<instance_name>/threatintel/collections/ | jq .to get a list of the collections available and on your TAXII server. From the list, copy the correct ID of the collection you want to ingest.
For Cortex XSOAR 8 On-prem, Cortex XSOAR Cloud, or Cortex XSIAM
- All your server info can be found by running
!taxii-server-info, the default API root for you server will usually be -https://ext-<tenant>.crtx.<region>.paloaltonetworks.com/xsoar/instance/execute/<instance-name>/threatintel/ - You can use the
!taxii-server-list-collectionscommand in order to get a list of your server’s collections and their IDs. You can also do it manually by runningcurl https://ext-<tenant>.crtx.<region>.paloaltonetworks.com/xsoar/instance/execute/<instance-name>/threatintel/collections/ | jq .to get a list of the collections available and on your TAXII server. From the list, copy the correct ID of the collection you want to ingest.
Response Example:
{
"collections": [
{
"can_read": true,
"can_write": false,
"description": "",
"id": "3709e6df-bbe1-5ece-b683-e99a42f31fce",
"media_types": [
"application/vnd.oasis.stix+json; version=2.0"
],
"query": "type:IP",
"title": "AllIPs"
},
{
"can_read": true,
"can_write": false,
"description": "",
"id": "16763019-5ee6-59bc-b4d9-be586235b308",
"media_types": [
"application/vnd.oasis.stix+json; version=2.0"
],
"query": "type:File",
"title": "AllHashes"
}
]
}
Set up the Microsoft Sentinel TAXII Connector
Now that we have the API root URL and the collection ID we can configure the Threat intelligence - TAXII Connector in Microsoft Sentinel.
Paste your API root URL in the field marked API Root URL and the desired collection ID we got in step (2) under Collection ID.
Example:

Commands
You can execute these commands from the Cortex XSOAR CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.
taxii-server-list-collections
Returns all the collections.
Base Command
taxii-server-list-collections
Input
There are no input arguments for this command.
Context Output
| Path | Type | Description |
|---|---|---|
| TAXIIServer.Collection.id | String | The collection ID. |
| TAXIIServer.Collection.query | String | The collection query. |
| TAXIIServer.Collection.title | String | The collection title. |
| TAXIIServer.Collection.description | String | The collection description. |
Command example
#### Context Example
```json
{
"TAXIIServer": {
"Collection": {
"can_read": true,
"can_write": false,
"description": "",
"id": "2eb7bfae-7739-5863-9b00-1681309c3d8c",
"media_types": [
"application/stix+json;version=2.1"
],
"query": "",
"title": "ALL"
}
}
}
Human Readable Output
Collections
id title query description 2eb7bfae-7739-5863-9b00-1681309c3d8c ALL
taxii-server-info
Returns the TAXII server info, default URL, title, etc.
Base Command
taxii-server-info
Input
There are no input arguments for this command.
Context Output
| Path | Type | Description |
|---|---|---|
| TAXIIServer.ServerInfo.title | String | The server title |
| TAXIIServer.ServerInfo.api_roots | Unknown | The server API roots URLs. |
| TAXIIServer.ServerInfo.default | String | The default URL. |
| TAXIIServer.ServerInfo.description | String | The server description |
Command Example
#### Context Example
```json
{
"TAXIIServer": {
"ServerInfo": {
"api_roots": [
"https://foo.cooo.com/inc/threatintel/"
],
"default": "https://foo.cooo.com/inc/threatintel/",
"description": "This integration provides TAXII Services for system indicators (Outbound feed).",
"title": "Cortex XSOAR TAXII2 Server"
}
}
}
Human Readable Output
In case the default URL is incorrect, you can override it by setting the “TAXII2 Service URL Address” field in the integration configuration
Server Info
api_roots default description title https://foo.cooo.com/inc/threatintel/ https://foo.cooo.com/inc/threatintel/ This integration provides TAXII Services for system indicators (Outbound feed). Cortex XSOAR TAXII2 Server
Troubleshooting
429 Too Many Requests error
NGINX prevents concurrent builds of the same cache entry. If multiple requests for the same cache entry (matching the URL and parameters) arrive simultaneously, NGINX builds the entry for the first request and rejects the others with an HTTP 429 Too Many Requests rather than queuing them. Requests for different entries are still processed in parallel. If a request triggers a refresh of an existing cache entry, the previous data is served (HTTP 200) with no 429.
This is expected behavior. Retry the request after a short delay; once the initial build finishes populating the cache, retries are served from the cache (HTTP 200).
[Errno 98] Address in use error
Each instance uses three consecutive ports: the configured Listen Port, port + 1, and port + 2. NGINX listens on the configured port (public), the Python process listens on port + 1, and NGINX uses port + 2 internally for its fail-fast cache fetch tier. For example, if configured for port 9009, ports 9009, 9010, and 9011 must all be free. Ensure no other instance uses a Listen Port within 2 of another, or an [Errno 98] Address in use error will occur. When running without --network=host, ports + 1 and + 2 are not exposed to the machine.
Configuration parameters
longRunning— Long Running Instanceversion— TAXII2 Server version (required)credentials— Usernamecollections— Collection JSON (required)fields_filter— Cortex XSOAR Extension fieldsres_size— Response Size (required)certificate— Certificate (Required for HTTPS)key— Private Key (Required for HTTPS)hsts_header— Add HSTS headerservice_address— TAXII2 Service URL AddresslongRunningPort— Listen Port (required)nginx_global_directives— NGINX Global Directivesnginx_server_conf— NGINX Server Confcache_refresh_rate— Refresh Ratecache_lock_timeout— Cache Lock Timeout (Deprecated)cache_lock_age— Cache Lock Age (Deprecated)cache_404_ttl— Cache 404 TTLcache_default_ttl— Cache Default TTLprovide_as_indicator— STIX types for STIX indicator Domain Objectcache_duration_hours— Search-after cache duration hours (required)
Commands (2)
-
taxii-server-infoReturns the TAXII server info, default URL, title, etc.
-
taxii-server-list-collectionsReturns all the collections.
import copy import json from freezegun import freeze_time import demistomock as demisto import pytest from requests.auth import _basic_auth_str from TAXII2Server import APP, MEDIA_TYPE_STIX_V20, MEDIA_TYPE_TAXII_V20, TAXII2Server, create_fields_list, create_query, uuid HEADERS = { "Authorization": _basic_auth_str("username", "password"), "Accept": "application/taxii+json", } @pytest.fixture def taxii2_server_v20(mocker): mocker.patch.object(demisto, "getLicenseID", return_value="test") server = TAXII2Server( url_scheme="http", host="demisto", port=7000, collections={"Collection1": "type:IP", "Collection2": 'sourceBrands:"Some Feed"'}, certificate="", private_key="", http_server=True, credentials={"identifier": "username", "password": "password"}, version="2.0", service_address=None, fields_to_present=set(), ) return server @pytest.fixture def taxii2_server_v21(mocker): mocker.patch.object(demisto, "getLicenseID", return_value="test") server = TAXII2Server( url_scheme="http", host="demisto", port=7000, collections={"Collection1": "type:IP", "Collection2": {"query": 'sourceBrands:"Some Feed"', "description": "Test desc"}}, certificate="", private_key="", http_server=True, credentials={"identifier": "username", "password": "password"}, version="2.1", service_address=None, fields_to_present=set(), ) return server def util_load_json(path): with open(path, encoding="utf-8") as f: return json.loads(f.read()) @pytest.mark.parametrize( "fields, result", [ ("", {"name", "type"}), ("all", set()), ("name,type,sha1", {"name", "type", "sha1"}), ("value,type,sha1", {"name", "type", "sha1"}), ("value,indicator_type,createdTime", {"name", "type", "createdTime"}), ], ) def test_create_fields_list(fields, result): """ Given fields list parameter, expected result When User enters filter_field param Then Validate right result returned """ assert result == create_fields_list(fields) @pytest.mark.parametrize("headers", [{"Authorization": _basic_auth_str("user", "pwd")}, {}]) def test_taxii_wrong_auth(mocker, headers, taxii2_server_v20): """ Given Taxii server v2.0 When Getting server discovery, with wrong auth Then Validate that the error and status code right """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch.object(demisto, "error") mocker.patch.object(demisto, "updateModuleHealth") with APP.test_client() as test_client: response = test_client.get("/taxii/", headers=headers) assert response.status_code == 401 assert response.json == {"title": "Authorization failed"} @pytest.mark.parametrize( "headers", [ {"Authorization": _basic_auth_str("username", "password")}, {"Authorization": _basic_auth_str("username", "password"), "Accept": "wrong_type"}, ], ) def test_taxii_wrong_accept(mocker, headers, taxii2_server_v20): """ Given Taxii server v2.0 When Getting server discovery, with wrong accept header Then Validate that the error and status code right """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch.object(demisto, "error") mocker.patch.object(demisto, "updateModuleHealth") with APP.test_client() as test_client: response = test_client.get("/taxii/", headers=headers) assert response.status_code == 406 def test_taxii20_server_discovery(mocker, taxii2_server_v20): """ Given Taxii server v2.0 When Getting server discovery Then Validate that the discovery output as expected """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) with APP.test_client() as test_client: response = test_client.get("/taxii/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0" assert response.json.get("default") == "http://demisto:7000/threatintel/" def test_taxii21_server_discovery(mocker, taxii2_server_v21): """ Given Taxii server v2.1 When Call server discovery api request Then Validate that the discovery output as expected """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) with APP.test_client() as test_client: response = test_client.get("/taxii/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json.get("default") == "http://demisto:7000/threatintel/" def test_taxii20_api_root(mocker, taxii2_server_v20): """ Given TAXII v2.0 server, api_root When Call api_root api request Then Validate that the api_root information returned as expected """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) with APP.test_client() as test_client: response = test_client.get("/threatintel/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0" assert response.json.get("title") == "Cortex XSOAR TAXII2 Server ThreatIntel" def test_taxii_wrong_api_root(mocker, taxii2_server_v20): """ Given Taxii server v2.0, Not exiting api_root When Getting api root information, for wrong api_root Then Validate that the error and status code right """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch.object(demisto, "error") mocker.patch.object(demisto, "updateModuleHealth") with APP.test_client() as test_client: response = test_client.get("/not_exsisting_api_root/", headers=HEADERS) assert response.status_code == 404 assert response.json.get("title") == "Unknown API Root" def test_taxii20_status(mocker, taxii2_server_v20): """ Given Status api call When Calling a status request Then Validate the error returned. """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) with APP.test_client() as test_client: response = test_client.get("/threatintel/status/1223456/", headers=HEADERS) assert response.status_code == 404 def test_taxii20_collections(mocker, taxii2_server_v20): """ Given TAXII Server v2.0 When Calling collections api request Then Validate that collections returned as expected """ collections = util_load_json("test_data/collections20.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0" assert response.json == collections def test_taxii21_collections(mocker, taxii2_server_v21): """ Given TAXII Server v2.1 When Calling collections api request Then Validate that collections returned as expected """ collections = util_load_json("test_data/collections21.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json == collections def test_taxii20_collection(mocker, taxii2_server_v20): """ Given TAXII Server v2.0, collection_id When Calling collection by id api request Then Validate that right collection returned """ collections = util_load_json("test_data/collections20.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0" assert response.json == collections.get("collections")[0] def test_taxii20_get_collections(mocker, taxii2_server_v20): from TAXII2Server import get_server_collections_command collections = taxii2_server_v20.get_collections() integration_context = {"collections": collections["collections"]} result = get_server_collections_command(integration_context=integration_context) assert result.outputs == integration_context["collections"] def test_taxii20_get_server_info(mocker, taxii2_server_v20): from TAXII2Server import get_server_info_command integration_context = {} integration_context["server_info"] = taxii2_server_v20.get_discovery_service(instance_execute=True) default_url = integration_context["server_info"]["default"] assert default_url == "https://demisto/instance/execute/threatintel/" result = get_server_info_command(integration_context=integration_context) assert result.outputs == integration_context["server_info"] def test_taxii20_get_server_info_demisto_version(mocker): """ Given TAXII Server v2.0, Integration context. When Calling get-server-info command Then Validate that the correct default URL is returned """ from TAXII2Server import get_server_info_command integration_context = {} integration_context["server_info"] = { "api_roots": ["https://www.example.com/path/to/resource"], "default": "https://www.example.com/path/to/resource", } mocker.patch("CommonServerPython.get_demisto_version", return_value={"version": "8.1.0", "buildNumber": "12345"}) results = get_server_info_command(integration_context=integration_context) assert results.outputs["default"] == "https://ext-www.example.com/xsoar/path/to/resource" def test_taxii21_collection(mocker, taxii2_server_v21): """ Given TAXII Server v2.1, collection_id When Calling collection by id api request Then Validate that right collection returned """ collections = util_load_json("test_data/collections21.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json == collections.get("collections")[0] def test_taxii_wrong_collection_id(mocker, taxii2_server_v21): """ Given Taxii server v2.1, Not exiting collection_id When Getting collection information, for wrong collection_id Then Validate that the error and status code right """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch.object(demisto, "error") mocker.patch.object(demisto, "updateModuleHealth") with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/not_exsisting_collection_id/", headers=HEADERS) assert response.status_code == 404 assert response.json.get("title") == "Unknown Collection" def test_taxii20_manifest(mocker, taxii2_server_v20): """ Given TAXII Server v2.0, collection_id, range When Calling manifest api request for given collection Then Validate that right manifest returned. """ iocs = util_load_json("test_data/ip_iocs.json") manifest = util_load_json("test_data/manifest20.json") headers = copy.deepcopy(HEADERS) headers["Range"] = "items 0-4" mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch.object(demisto, "searchIndicators", return_value=iocs) mocker.patch.object(demisto, "params", return_value={"res_size": "100"}) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/", headers=headers) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.taxii+json; version=2.0" assert response.json == manifest def test_taxii21_manifest(mocker, taxii2_server_v21): """ Given TAXII Server v2.1, collection_id When Calling manifest api request for given collection Then Validate that right manifest returned. """ iocs = util_load_json("test_data/ip_iocs.json") manifest = util_load_json("test_data/manifest21.json") mocker.patch.object(demisto, "params", return_value={"res_size": "100"}) mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch.object(demisto, "searchIndicators", return_value=iocs) with APP.test_client() as test_client: response = test_client.get( "/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/?limit=4", headers=HEADERS ) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json == manifest def test_taxii20_objects(mocker, taxii2_server_v20): """ Given TAXII Server v2.0, collection_id, content-range When Calling get objects api request for given collection Then Validate that right objects are returned. """ iocs = util_load_json("test_data/ip_iocs.json") objects = util_load_json("test_data/objects20.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777") headers = copy.deepcopy(HEADERS) headers["Content-Range"] = "items 0-2/5" mocker.patch.object(demisto, "searchIndicators", return_value=iocs) mocker.patch.object(demisto, "params", return_value={"res_size": "100"}) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/objects/", headers=headers) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.stix+json; version=2.0" assert response.json == objects assert response.headers.get("Content-Range") == "items 0-3/5" def test_taxii20_indicators_objects(mocker, taxii2_server_v20): """ Given TAXII Server v2.0, collection_id, content-range, types_for_indicator_sdo with all types included. When Calling get objects api request for given collection Then Validate that right objects are returned and no extensions are returned. """ iocs = util_load_json("test_data/ip_iocs.json") objects = util_load_json("test_data/objects20-indicators.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch( "TAXII2Server.SERVER.types_for_indicator_sdo", ["ipv4-addr", "domain-name", "ipv6-addr", "user-account", "email-addr", "windows-registry-key", "file", "url"], ) mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777") headers = copy.deepcopy(HEADERS) headers["Content-Range"] = "items 0-2/5" mocker.patch.object(demisto, "searchIndicators", return_value=iocs) mocker.patch.object(demisto, "params", return_value={"res_size": "100"}) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/objects/", headers=headers) assert response.status_code == 200 assert response.content_type == "application/vnd.oasis.stix+json; version=2.0" assert response.json == objects assert response.headers.get("Content-Range") == "items 0-3/5" @pytest.mark.parametrize( "demisto_iocs_file,res_file,query_type", [ ("malware_iocs", "objects21_malware", "malware"), ("file_iocs", "objects21_file", "file"), ("domain_iocs", "objects21_domain", "domain-name,attack-pattern"), ], ) def test_taxii21_objects(mocker, taxii2_server_v21, demisto_iocs_file, res_file, query_type): """ Given TAXII Server v2.1, collection_id, limit, next, type parameter When Calling get objects api request for given collection Then Validate that right objects are returned. """ iocs = util_load_json(f"test_data/{demisto_iocs_file}.json") objects = util_load_json(f"test_data/{res_file}.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777") mocker.patch.object(demisto, "searchIndicators", return_value=iocs) mocker.patch.object(demisto, "params", return_value={"res_size": "100"}) with APP.test_client() as test_client: response = test_client.get( f"/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/objects/?match[type]={query_type}&limit=2&next=1", headers=HEADERS, ) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json == objects @pytest.mark.parametrize("api_request", ["objects", "manifest"]) def test_taxii21_bad_request(mocker, taxii2_server_v21, api_request): """ Given TAXII Server v2.1, non-supported filter. When Calling get objects or manifest api request for given collection Then Validate that right error returned. """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch.object(demisto, "error") mocker.patch.object(demisto, "params", return_value={"res_size": "2500"}) mocker.patch.object(demisto, "updateModuleHealth") with APP.test_client() as test_client: response = test_client.get( f"/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/{api_request}/?match[version]=3", headers=HEADERS ) assert response.status_code == 404 assert response.content_type == "application/taxii+json;version=2.1" assert "Filtering by ID or version is not supported." in response.json.get("description") @pytest.mark.parametrize("api_request", ["objects", "manifest"]) def test_taxii20_bad_content_range(mocker, taxii2_server_v20, api_request): """ Given TAXII Server v2.0, non-supported range. When Calling get objects or manifest api request for given collection Then Validate that right error returned. """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v20) mocker.patch.object(demisto, "params", return_value={"res_size": "2500"}) headers = copy.deepcopy(HEADERS) headers["Content-Range"] = "items 8-2/10" with APP.test_client() as test_client: response = test_client.get( f"/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/{api_request}/", headers=headers ) assert response.status_code == 416 @pytest.mark.parametrize( "res_file,fields,has_extension", [("objects21_no_extention_file", {"name", "type"}, False), ("objects21_spec_fields_file", {"sha1"}, True)], ) def test_taxii21_objects_filtered_params(mocker, taxii2_server_v21, res_file, fields, has_extension): """ Given TAXII Server v2.1, collection_id, type parameter, filtered_fields params When Calling get objects api request for given collection Then Validate that right objects are returned. """ iocs = util_load_json("test_data/file_iocs_filter_test.json") objects = util_load_json(f"test_data/{res_file}.json") mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch("TAXII2Server.SERVER.fields_to_present", fields) mocker.patch("TAXII2Server.SERVER.has_extension", has_extension) mocker.patch.object(uuid, "uuid4", return_value="1ffe4bee-95e7-4e36-9a17-f56dbab3c777") mocker.patch.object(demisto, "searchIndicators", return_value=iocs) mocker.patch.object(demisto, "params", return_value={"res_size": "7"}) with APP.test_client() as test_client: response = test_client.get( "/threatintel/collections/e46189b5-c5c8-5c7f-b947-183e0302b4d3/objects/?match[type]=file", headers=HEADERS ) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json == objects @pytest.mark.parametrize("header", (MEDIA_TYPE_TAXII_V20, MEDIA_TYPE_STIX_V20)) def test_taxii21_with_taxii20_header(mocker, taxii2_server_v21, header: str): """ Given a TAXII 2.1 server When calling /taxii2/ with TAXII 2.0 header Then validate that an appropriate error is returned """ mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) with APP.test_client() as test_client: response = test_client.get("/taxii2/", headers=HEADERS | {"Accept": header}) assert response.status_code == 406 @pytest.mark.parametrize( "query,types,added_after,expected_response", ( ("my custom query", [], "2023-07-06T10:57:15.133309Z", 'my custom query and modified:>="2023-07-06T10:57:15.133309Z"'), ( "my custom query", ["file"], "2023-07-06T10:57:15.133309Z", '(my custom query) and (type:"File") and modified:>="2023-07-06T10:57:15.133309Z"', ), ( "my custom query", ["file", "domain"], "2023-07-06T10:57:15.133309Z", '(my custom query) and (type:"File" or type:"domain") and modified:>="2023-07-06T10:57:15.133309Z"', ), ), ) def test_create_query(query: str, types: list[str], added_after, expected_response: str): """ Given a query, types to match, and added after. When calling create_query Then Validate that right query is returned. """ assert create_query(query, types, added_after) == expected_response @pytest.mark.parametrize( "endpoint", [ ("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/?limit=4&added_after=2022-06-03T00:00:00Z"), ( "/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/manifest/?limit=4&added_after=2022-06-03T13:54:27.234765Z" ), ], ) def test_parse_manifest_and_object_args_with_valid_date(mocker, taxii2_server_v21, endpoint): """ Given case 1: endpoint with utc date format. case 2: endpoint with stix date format. When testing parse_manifest_and_object_args. Then Ensure that Should parsing was done correctly and a valid results message was returned. """ iocs = util_load_json("test_data/ip_iocs.json") manifest = util_load_json("test_data/manifest21.json") mocker.patch.object(demisto, "params", return_value={"res_size": "100"}) mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch.object(demisto, "searchIndicators", return_value=iocs) with APP.test_client() as test_client: response = test_client.get(endpoint, headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" assert response.json == manifest def test_taxii21_objects_with_relationships(mocker, taxii2_server_v21): """ Given TAXII Server v2.1, collection_id, no_extension When Calling get objects api request for given collection Then Validate that right objects are returned. Ensure that searchRelationships is called with the expected arguments. """ from CommonServerPython import get_demisto_version get_demisto_version._version = None # clear cache between runs of the test mocker.patch.object(demisto, "demistoVersion", return_value={"version": "6.6.0"}) mocker.patch("TAXII2Server.SERVER", taxii2_server_v21) mocker.patch("TAXII2Server.SERVER.has_extension", False) mocker.patch("TAXII2Server.SERVER.fields_to_present", {"name", "type"}) mock_search_relationships_response = util_load_json("test_data/searchRelationships-response.json") mocker.patch.object(demisto, "searchRelationships", return_value=mock_search_relationships_response) objects = util_load_json("test_data/objects21_ip_with_relationships.json") mock_iocs = util_load_json("test_data/sort_ip_iocs.json") mock_entity_b_iocs = util_load_json("test_data/entity_b_iocs.json") mocker.patch.object(demisto, "searchIndicators", side_effect=[mock_iocs, mock_entity_b_iocs]) mocker.patch.object(demisto, "params", return_value={"res_size": "20"}) with APP.test_client() as test_client: response = test_client.get("/threatintel/collections/4c649e16-2bb7-50f5-8826-2a2d0a0b9631/objects/", headers=HEADERS) assert response.status_code == 200 assert response.content_type == "application/taxii+json;version=2.1" demisto.searchRelationships.assert_called_once_with( { "entities": [ "1.1.1.1", "3.3.3.3", "f1412386aa8db2579aff2636cb9511cacc5fd9880ecab60c048508fbe26ee4d9", "2.2.2.2", "4.4.4.4", "bad-domain.com", ] } ) assert response.json == objects @freeze_time("2025-01-01 12:00:00 UTC") def test_remove_old_cache(mocker): """ Given An integration context with search_after_cache entries. When remove_old_cache is called. Then Validate that old cache entries are removed and new ones are kept. """ from TAXII2Server import remove_old_cache mocker.patch.object(demisto, "params", return_value={"cache_duration_hours": 24}) integration_context = { "search_after_cache": { "collection1": { "0": {"search_after": ["val1"], "last_updated": "2024-12-31T10:00:00Z"}, # Expired (26 hours old) "10": {"search_after": ["val2"], "last_updated": "2025-01-01T10:00:00Z"}, # Not expired (2 hours old) }, "collection2": { "0": {"search_after": ["val3"], "last_updated": "2024-12-30T10:00:00Z"}, # Expired (50 hours old) "20": {"search_after": ["val4"], "last_updated": "2025-01-01T11:00:00Z"}, # Not expired (1 hour old) }, "collection3": { "0": {"search_after": ["val5"], "last_updated": "invalid-date"}, # Invalid date, should be kept }, "collection4": { "0": {"search_after": ["val6"]}, # Missing last_updated, should be kept }, } } remove_old_cache(integration_context) expected_context = { "search_after_cache": { "collection1": { "10": {"search_after": ["val2"], "last_updated": "2025-01-01T10:00:00Z"}, }, "collection2": { "20": {"search_after": ["val4"], "last_updated": "2025-01-01T11:00:00Z"}, }, "collection3": { "0": {"search_after": ["val5"], "last_updated": "invalid-date"}, }, "collection4": { "0": {"search_after": ["val6"]}, }, } } assert integration_context == expected_context