CSVFeed

Fetch indicators from a CSV feed.

Data Enrichment & Threat Intelligence · CSV Feed · Feed

Details

ID	`CSVFeed`
Provider	Open Source
Category	Data Enrichment & Threat Intelligence
From Version	5.5.0
Docker Image	`demisto/py3-tools:1.0.0.114656`
Supported Modules	Agentix XSIAM

README

Overview

Fetch indicators from a CSV feed. The integration allows a great amount of user configuration to support different types of CSV feeds.

Configure CSV Feed on Cortex XSOAR

Navigate to Settings > Integrations > Servers & Services.
Search for CSVFeed.
Click Add instance to create and configure a new integration instance.
- Name: a textual name for the integration instance.
- URL: Server URL where the feed is.
- Fetch indicators: boolean flag. If set to true will fetch indicators.
- Fetch Interval: Interval of the fetches.
- Reliability: Reliability of the feed.
- Traffic Light Protocol Color: The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed. More information about the protocol can be found at https://us-cert.cisa.gov/tlp
- Username + Password - Credentials to access feeds that require basic authentication.
  These fields also support the use of API key headers. To use API key headers, specify the header name and value in the following format:
  _header:<header_name> in the Username field and the header value in the Password field.
- Trust any certificate (not secure)
- Use system proxy settings
- Request Timeout: Time (in seconds) before HTTP requests timeout.
- Ignore Regex: Python regular expression for lines that should be ignored.
- Field Names: Name of the field names in the CSV. If several are given, will use
  “indicator” as the indicator value field.
- Delimiter: A one-character string used to separate fields.
- Double quote: Controls how instances of quote character appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar. It defaults to True.
- Escape character: A one-character string used by the writer to escape the delimiter.
- Quote Character: A one-character string used to quote fields containing special characters.
- Skip Initial Space: When True, whitespace immediately following the delimiter is ignored.
Click Test to validate the URLs, token, and connection.

Step by step configuration

As an example, we’ll be looking at the SSL BL feed by Abuse. This feed will ingest indicators of type IP. These are the feed isntance configuration parameters for our example.

Indicator Type - IP.

Server URL: https://sslbl.abuse.ch/blacklist/sslipblacklist.csv.

Credentials - This feed does not require authentication.

From a quick look at the feed in the web browser, we are going to configure the rest of the parameters:

Ignore Regex - We are going to need to ignore all the text inside the part enclosed by the # character (included)
so we’ll configure ^# as the regular expression to use to ignore this text.

Field Names - We have 3 fields in this feed - Firstseen,DstIP,DstPort. The integration ignores these headers and we have to configure the field names for each indicator.
Note that the field for the indicator value itself (the IP) must be value. So we will configure these field names: date,value,name, so that the indicator will be created with these fields.

Double quote - No need to double the quote characters, we’ll leave this option unchecked.

Delimiter - The delimiter between the fields in this feed is ,, we’ll use that as the value for this field.

Quote Character - No need to change the quote character, we’ll leave that as the default (").

Escape Character - No need to change the escape character, we’ll leave that empty.

Skip Initial Space - No whitespaces between the delimiter and the value, we’ll leave the unchecked.

Now we have successfully configured an instance for the Abuse SSL BL feed, once we enable Fetches indicators the instance will start pulling indicators.

By clicking Mapping in the integration instance, we can map the field names we previously configured to actual indicator fields (except value which is the indicator value).
We can use Set up a new classification rule using actual data from the feed.

Demo Video

Sorry, your browser doesn't support embedded videos. You can download the video at: https://github.com/demisto/content-assets/blob/7982404664dc68c2035b7c701d093ec026628802/Assets/FeedCSV/CSVFeed_Video.mp4

Configuration parameters

feed — Fetch indicators
feedReputation — Indicator Reputation
feedReliability — Source Reliability (required)
tlp_color — Traffic Light Protocol Color
feedExpirationPolicy —
feedExpirationInterval —
feedFetchInterval — Feed Fetch Interval
feedBypassExclusionList — Bypass exclusion list
auto_detect_type — Auto detect indicator type
indicator_type — Indicator Type
url — URL (required)
credentials — Username
polling_timeout — Request Timeout (required)
ignore_regex — Ignore Regex
fieldnames — Field Names
delimiter — Delimiter
doublequote — Double quote
escapechar — Escape character
quotechar — Quote Character
skipinitialspace — Skip Initial Space
encoding — Text Encoding (required)
feedTags — Tags
insecure — Trust any certificate (not secure)
proxy — Use system proxy settings
enrichmentExcluded — Enrichment Excluded

Commands (1)

csv-get-indicators
Gets indicators from the feed.

Source

FeedCSV.py FeedCSV.yml FeedCSV_description.md

Download History Raw

category: Data Enrichment & Threat Intelligence
provider: Open Source
commonfields:
  id: CSVFeed
  version: -1
configuration:
- display: Fetch indicators
  name: feed
  defaultvalue: 'true'
  type: 8
  required: false
  section: Collect
- additionalinfo: Indicators from this integration instance will be marked with this reputation
  defaultvalue: feedInstanceReputationNotSet
  display: Indicator Reputation
  name: feedReputation
  options:
  - None
  - Good
  - Suspicious
  - Bad
  type: 18
  required: false
  section: Collect
- additionalinfo: Reliability of the source providing the intelligence data
  defaultvalue: F - Reliability cannot be judged
  display: Source Reliability
  name: feedReliability
  options:
  - A - Completely reliable
  - B - Usually reliable
  - C - Fairly reliable
  - D - Not usually reliable
  - E - Unreliable
  - F - Reliability cannot be judged
  required: true
  type: 15
  section: Collect
- additionalinfo: The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed
  display: Traffic Light Protocol Color
  name: tlp_color
  options:
  - RED
  - AMBER
  - GREEN
  - WHITE
  type: 15
  required: false
  section: Collect
- defaultvalue: indicatorType
  display: ""
  name: feedExpirationPolicy
  options:
  - never
  - interval
  - indicatorType
  - suddenDeath
  type: 17
  required: false
  section: Collect
  advanced: true
- defaultvalue: '20160'
  name: feedExpirationInterval
  display: ""
  type: 1
  required: false
  section: Collect
  advanced: true
- defaultvalue: '240'
  display: Feed Fetch Interval
  name: feedFetchInterval
  type: 19
  required: false
  section: Collect
  advanced: true
- additionalinfo: When selected, the exclusion list is ignored for indicators from this feed. This means that if an indicator from this feed is on the exclusion list, the indicator might still be added to the system.
  display: Bypass exclusion list
  name: feedBypassExclusionList
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: If selected, the indicator type will be auto detected for each indicator.
  defaultvalue: 'true'
  display: Auto detect indicator type
  name: auto_detect_type
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: Type of the indicator in the feed, If auto-detect is checked then the value set as Indicator Type will be ignored.
  display: Indicator Type
  name: indicator_type
  type: 0
  required: false
  section: Collect
  advanced: true
- display: URL
  name: url
  required: true
  type: 0
  section: Connect
- display: Username
  name: credentials
  type: 9
  required: false
  section: Connect
- additionalinfo: Time (in seconds) before HTTP requests timeout.
  defaultvalue: '20'
  display: Request Timeout
  name: polling_timeout
  required: true
  type: 0
  section: Collect
  advanced: true
- additionalinfo: Python regular expression for lines to ignore.
  display: Ignore Regex
  name: ignore_regex
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: The names to apply to the fields in the CSV feed. The name for the field containing the indicator should be "value". If left empty, the fetch will use existing headers only ('value' header is expected).
  display: Field Names
  name: fieldnames
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: A one-character string used to separate fields. For example, a comma ",".
  defaultvalue: ','
  display: Delimiter
  name: delimiter
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Controls how instances of quotechar in a field should themselves be quoted. When True, the character is doubled.
  display: Double quote
  name: doublequote
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: A one-character string used by the writer to escape the delimiter.
  display: Escape character
  name: escapechar
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: A one-character string used to quote fields containing special characters.
  defaultvalue: '"'
  display: Quote Character
  name: quotechar
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: When "True", whitespace immediately following the delimiter is ignored.
  display: Skip Initial Space
  name: skipinitialspace
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: Text encoding for the CSV file. Defaults to latin-1.
  display: Text Encoding
  name: encoding
  type: 0
  required: true
  defaultvalue: latin-1
  section: Collect
  advanced: true
- display: Tags
  name: feedTags
  type: 0
  required: false
  additionalinfo: Supports CSV values.
  section: Collect
  advanced: true
- display: Trust any certificate (not secure)
  name: insecure
  type: 8
  required: false
  section: Connect
  advanced: true
- display: Use system proxy settings
  name: proxy
  required: false
  type: 8
  section: Connect
  advanced: true
- display: Enrichment Excluded
  name: enrichmentExcluded
  type: 8
  required: false
  defaultvalue: 'false'
  additionalinfo: Select this option to exclude the fetched indicators from the enrichment process.
  hidden:
  - xsoar_on_prem
  section: Collect
description: Fetch indicators from a CSV feed.
display: CSV Feed
name: CSVFeed
script:
  commands:
  - arguments:
    - defaultValue: '50'
      description: The maximum number of results to return to the output.
      name: limit
    - description: The indicator type. If the configuration parameter 'Auto detect indicator type' is marked true for the integration instance, then this value will be ignored.
      name: indicator_type
    description: Gets indicators from the feed.
    name: csv-get-indicators
    outputs:
    - contextPath: CSV.Indicator.value
      description: The indicator value.
      type: String
    - contextPath: CSV.Indicator.type
      description: The indicator type.
      type: String
    - contextPath: CSV.Indicator.rawJSON
      description: The indicator rawJSON value.
      type: Unknown
  dockerimage: demisto/py3-tools:1.0.0.114656
  feed: true
  runonce: false
  script: ''
  subtype: python3
  type: python
tests:
- CSV_Feed_Test
fromversion: 5.5.0
sectionorder:
- Connect
- Collect