CSVFeed

Fetch indicators from a CSV feed.

Data Enrichment & Threat Intelligence · CSV Feed · Feed

Details

ID	`CSVFeed`
Provider	Open Source
Category	Data Enrichment & Threat Intelligence
From Version	5.5.0
Docker Image	`demisto/py3-tools:1.0.0.114656`
Supported Modules	Agentix XSIAM

README

Overview

Fetch indicators from a CSV feed. The integration allows a great amount of user configuration to support different types of CSV feeds.

Configure CSV Feed on Cortex XSOAR

Navigate to Settings > Integrations > Servers & Services.
Search for CSVFeed.
Click Add instance to create and configure a new integration instance.
- Name: a textual name for the integration instance.
- URL: Server URL where the feed is.
- Fetch indicators: boolean flag. If set to true will fetch indicators.
- Fetch Interval: Interval of the fetches.
- Reliability: Reliability of the feed.
- Traffic Light Protocol Color: The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed. More information about the protocol can be found at https://us-cert.cisa.gov/tlp
- Username + Password - Credentials to access feeds that require basic authentication.
  These fields also support the use of API key headers. To use API key headers, specify the header name and value in the following format:
  _header:<header_name> in the Username field and the header value in the Password field.
- Trust any certificate (not secure)
- Use system proxy settings
- Request Timeout: Time (in seconds) before HTTP requests timeout.
- Ignore Regex: Python regular expression for lines that should be ignored.
- Field Names: Name of the field names in the CSV. If several are given, will use
  “indicator” as the indicator value field.
- Delimiter: A one-character string used to separate fields.
- Double quote: Controls how instances of quote character appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar. It defaults to True.
- Escape character: A one-character string used by the writer to escape the delimiter.
- Quote Character: A one-character string used to quote fields containing special characters.
- Skip Initial Space: When True, whitespace immediately following the delimiter is ignored.
Click Test to validate the URLs, token, and connection.

Step by step configuration

As an example, we’ll be looking at the SSL BL feed by Abuse. This feed will ingest indicators of type IP. These are the feed isntance configuration parameters for our example.

Indicator Type - IP.

Server URL: https://sslbl.abuse.ch/blacklist/sslipblacklist.csv.

Credentials - This feed does not require authentication.

From a quick look at the feed in the web browser, we are going to configure the rest of the parameters:

Ignore Regex - We are going to need to ignore all the text inside the part enclosed by the # character (included)
so we’ll configure ^# as the regular expression to use to ignore this text.

Field Names - We have 3 fields in this feed - Firstseen,DstIP,DstPort. The integration ignores these headers and we have to configure the field names for each indicator.
Note that the field for the indicator value itself (the IP) must be value. So we will configure these field names: date,value,name, so that the indicator will be created with these fields.

Double quote - No need to double the quote characters, we’ll leave this option unchecked.

Delimiter - The delimiter between the fields in this feed is ,, we’ll use that as the value for this field.

Quote Character - No need to change the quote character, we’ll leave that as the default (").

Escape Character - No need to change the escape character, we’ll leave that empty.

Skip Initial Space - No whitespaces between the delimiter and the value, we’ll leave the unchecked.

Now we have successfully configured an instance for the Abuse SSL BL feed, once we enable Fetches indicators the instance will start pulling indicators.

By clicking Mapping in the integration instance, we can map the field names we previously configured to actual indicator fields (except value which is the indicator value).
We can use Set up a new classification rule using actual data from the feed.

Demo Video

Sorry, your browser doesn't support embedded videos. You can download the video at: https://github.com/demisto/content-assets/blob/7982404664dc68c2035b7c701d093ec026628802/Assets/FeedCSV/CSVFeed_Video.mp4

Configuration parameters

feed — Fetch indicators
feedReputation — Indicator Reputation
feedReliability — Source Reliability (required)
tlp_color — Traffic Light Protocol Color
feedExpirationPolicy —
feedExpirationInterval —
feedFetchInterval — Feed Fetch Interval
feedBypassExclusionList — Bypass exclusion list
auto_detect_type — Auto detect indicator type
indicator_type — Indicator Type
url — URL (required)
credentials — Username
polling_timeout — Request Timeout (required)
ignore_regex — Ignore Regex
fieldnames — Field Names
delimiter — Delimiter
doublequote — Double quote
escapechar — Escape character
quotechar — Quote Character
skipinitialspace — Skip Initial Space
encoding — Text Encoding (required)
feedTags — Tags
insecure — Trust any certificate (not secure)
proxy — Use system proxy settings
enrichmentExcluded — Enrichment Excluded

Commands (1)

csv-get-indicators
Gets indicators from the feed.

Source

FeedCSV.py FeedCSV.yml FeedCSV_description.md

Download History Raw

import demistomock as demisto  # noqa: F401
from CommonServerPython import *  # noqa: F401


def main():
    params = demisto.params()
    # when auto_detect is not selected
    if params.get("auto_detect_type") is False and not params.get("indicator_type"):
        return_error("Indicator Type cannot be empty when Auto Detect Indicator Type is unchecked")
    # when auto_detect does not exist - for previous integration instances
    if params.get("auto_detect_type") is None and not params.get("indicator_type"):
        return_error("Indicator Type cannot be empty")
    feed_main("CSV", prefix="csv")


from CSVFeedApiModule import *  # noqa: E402

if __name__ in ("__builtin__", "builtins", "__main__"):
    main()