Plain Text Feed

Fetches indicators from a plain text feed.

Data Enrichment & Threat Intelligence · Plain Text Feed · Feed

Details

ID	`Plain Text Feed`
Provider	Open Source
Category	Data Enrichment & Threat Intelligence
From Version	5.5.0
Docker Image	`demisto/py3-tools:1.0.0.6127057`
Supported Modules	Agentix XSIAM

README

Plain Text Feed Integration

Fetches indicators from a plain text feed. The integration allows a many user configurations to support different types of plain text feeds.

Configuration

Server URL - URL of the feed.
Indicator Type - The type of indicators in the feed. If the Custom option is selected, the Custom Indicator Type parameter must be provided.
Custom Indicator Type - The indicator type to be used in case of Custom option chosen in the Indicator Type field.
Username + Password - Credentials to access feeds that require basic authentication.
These fields also support the use of API key headers. To use API key headers, specify the header name and value in the following format:
_header:<header_name> in the Username field and the header value in the Password field.
Ignore Regex - Python regular expression for lines that should be ignored.
Indicator extraction pattern - A JSON string of an extraction pattern for the indicator value in the text that consists of a regular expression and a transform template for each regex group. For example:

{
  "regex": "^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})",
  "transform": "\\1-\\2"
}

Fields extraction pattern - A JSON string of an extraction pattern for the additional fields in the text that consists of a regular expression and a transform template for each regex group. For example:

{
  "number_of_attacks":
    {
        "regex": "^.*\\t.*\\t[0-9]+\\t([0-9]+)",
        "transform": "\\1"
    },
  "name":
     {  
        "regex": ",^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)",
        "transform": "\\1"
     }
}

For more information about regular expression extraction, see this python documentation.

Headers - CSV list of headers to send in the HTTP request in the format of “header_name:header_value”. For example:

Content-Type:text/plain,Accept:application/json

Step by step configuration

As an example, we’ll be looking at the Recommended Block List feed by DShield. This feed will ingest indicators of type CIDR. These are the feed instance configuration parameters for our example.

Indicator Type - CIDR.

Server URL: https://www.dshield.org/block.txt

Credentials - This feed does not require authentication.

From a quick look at the feed in the web browser, we are going to configure the rest of the parameters:

Ignore Regex - We are going to need to ignore all the text inside the part enclosed within the # character (included)
so we’ll configure ^# as the regular expression to use to ignore this text.

Indicator extraction pattern - We would like to extract the IP range and turn it into a CIDR. For that, we will configure a regular expression to extract both IP addresses in the range into 2 groups,
and transform the 2 groups to an IP range. We will then convert the IP range into a CIDR in the integration code.

This would be our extraction pattern object as a JSON string which we will fill in the field in the instance configuration:

{
  "regex": "^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})",
  "transform": "\\1-\\2"
}

**Fields extraction pattern** - We want to extract the name and the number of attacks field for each IP range in the feed.
For each field we will configure a regular expression to extract it and then a template to grab the regex group as is.
This would be the JSON string we will use in the integration configuration:

{
  "number_of_attacks":
    {
        "regex": "^.*\\t.*\\t[0-9]+\\t([0-9]+)",
        "transform": "\\1"
    },
  "name":
     {  
        "regex": "^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)",
        "transform": "\\1"
     }
}

Then our indicator will have these 2 additional fields. We can map them to other indicator fields in the system later.

Headers - No need for additional headers.

Now we have successfully configured an instance for the DShield Black List feed, once we enable Fetches indicators the instance will start pulling indicators.

By clicking Mapping in the integration instance, we can map the fields we previously configured to actual indicator fields.
We can use Set up a new classification rule to use actual data from the feed.

Configuration parameters

feed — Fetch indicators
url — Server URL (required)
credentials — Username
auto_detect_type — Auto detect indicator type
indicator_type — Indicator Type
cidr_32_to_ip — Set /32 CIDRs as IP Indicators
custom_indicator_type — Custom Indicator Type
feedReputation — Indicator Reputation
feedReliability — Source Reliability (required)
tlp_color — Traffic Light Protocol Color
feedExpirationPolicy —
feedExpirationInterval —
feedFetchInterval — Feed Fetch Interval
feedBypassExclusionList — Bypass exclusion list
polling_timeout — Request Timeout (required)
ignore_regex — Ignore Regex
feedTags — Tags
insecure — Trust any certificate (not secure)
proxy — Use system proxy settings
feed_name — Feed name
indicator — Indicator extraction pattern
fields — Fields extraction pattern
headers — Headers
enrichmentExcluded — Enrichment Excluded

Commands (1)

plaintext-get-indicators
Gets indicators from the feed.

Source

FeedPlainText.py FeedPlainText.yml FeedPlainText_description.md

Download History Raw

category: Data Enrichment & Threat Intelligence
provider: Open Source
commonfields:
  id: Plain Text Feed
  version: -1
configuration:
- defaultvalue: 'true'
  display: Fetch indicators
  name: feed
  type: 8
  required: false
  section: Collect
- display: Server URL
  name: url
  required: true
  type: 0
  section: Connect
- additionalinfo: It is possible to provide an API key header instead. See the description for more details.
  display: Username
  name: credentials
  type: 9
  required: false
  section: Connect
- additionalinfo: If selected, the indicator type will be auto detected for each indicator.
  defaultvalue: 'true'
  display: Auto detect indicator type
  name: auto_detect_type
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: Type of the indicator in the feed. If auto-detect is checked then the value set as Indicator Type will be ignored. If the "Custom" option is selected, the Custom Indicator Type parameter must be provided.
  display: Indicator Type
  name: indicator_type
  options:
  - IP
  - CIDR
  - IPv6
  - IPv6CIDR
  - Domain
  - Email
  - URL
  - File
  - Account
  - CVE
  - Host
  - ssdeep
  - Registry Key
  - Custom
  type: 15
  required: false
  section: Collect
  advanced: true
- additionalinfo: When enabled, each /32 CIDR will also be returned as an IP indicator. Only relevant if the integration fetches /32 CIDR indicators.
  display: Set /32 CIDRs as IP Indicators
  name: cidr_32_to_ip
  required: false
  type: 8
  section: Collect
  advanced: true
- additionalinfo: The indicator type to be used in case of 'Custom' option chosen in the Indicator Type field.
  display: Custom Indicator Type
  name: custom_indicator_type
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Indicators from this integration instance will be marked with this reputation
  defaultvalue: feedInstanceReputationNotSet
  display: Indicator Reputation
  name: feedReputation
  options:
  - None
  - Good
  - Suspicious
  - Bad
  type: 18
  required: false
  section: Collect
- additionalinfo: Reliability of the source providing the intelligence data
  defaultvalue: F - Reliability cannot be judged
  display: Source Reliability
  name: feedReliability
  options:
  - A - Completely reliable
  - B - Usually reliable
  - C - Fairly reliable
  - D - Not usually reliable
  - E - Unreliable
  - F - Reliability cannot be judged
  required: true
  type: 15
  section: Collect
- additionalinfo: The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed
  display: Traffic Light Protocol Color
  name: tlp_color
  options:
  - RED
  - AMBER
  - GREEN
  - WHITE
  type: 15
  required: false
  section: Collect
- defaultvalue: indicatorType
  display: ''
  name: feedExpirationPolicy
  options:
  - never
  - interval
  - indicatorType
  - suddenDeath
  type: 17
  required: false
  section: Collect
  advanced: true
- defaultvalue: '20160'
  display: ''
  name: feedExpirationInterval
  type: 1
  required: false
  section: Collect
  advanced: true
- defaultvalue: '240'
  display: Feed Fetch Interval
  name: feedFetchInterval
  type: 19
  required: false
  section: Collect
  advanced: true
- additionalinfo: When selected, the exclusion list is ignored for indicators from this feed. This means that if an indicator from this feed is on the exclusion list, the indicator might still be added to the system.
  display: Bypass exclusion list
  name: feedBypassExclusionList
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: Time (in seconds) before HTTP requests timeout
  defaultvalue: '20'
  display: Request Timeout
  name: polling_timeout
  required: true
  type: 0
  section: Collect
  advanced: true
- additionalinfo: Python regular expression for lines to ignore
  display: Ignore Regex
  name: ignore_regex
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Supports CSV values.
  display: Tags
  name: feedTags
  type: 0
  required: false
  section: Collect
  advanced: true
- display: Trust any certificate (not secure)
  name: insecure
  type: 8
  required: false
  section: Connect
  advanced: true
- display: Use system proxy settings
  name: proxy
  type: 8
  required: false
  section: Connect
  advanced: true
- display: Feed name
  name: feed_name
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: JSON string of patterns to extract from the text response. See the description for more details.
  display: Indicator extraction pattern
  name: indicator
  type: 12
  required: false
  section: Collect
  advanced: true
- additionalinfo: JSON string of patterns to extract from the text response. See the description for more details.
  display: Fields extraction pattern
  name: fields
  type: 12
  required: false
  section: Collect
  advanced: true
- additionalinfo: CSV list of headers to send in the HTTP request in the format of "header_name:header_value".
  display: Headers
  name: headers
  type: 0
  required: false
  section: Collect
  advanced: true
- display: Enrichment Excluded
  name: enrichmentExcluded
  type: 8
  required: false
  defaultvalue: 'false'
  additionalinfo: Select this option to exclude the fetched indicators from the enrichment process.
  hidden:
  - xsoar_on_prem
  section: Collect
description: Fetches indicators from a plain text feed.
display: Plain Text Feed
name: Plain Text Feed
script:
  commands:
  - arguments:
    - defaultValue: '50'
      description: The maximum number of results to return to the output. The default value is 50.
      name: limit
    - description: The indicator type. If the configuration parameter 'Auto detect indicator type' is marked true for the integration instance, then this value will be ignored.
      name: indicator_type
    description: Gets indicators from the feed.
    name: plaintext-get-indicators
  dockerimage: demisto/py3-tools:1.0.0.6127057
  feed: true
  runonce: false
  script: '-'
  subtype: python3
  type: python
tests:
- PlainText Feed - Test
fromversion: 5.5.0
sectionorder:
- Connect
- Collect