ElasticsearchFeed

Fetches indicators stored in an Elasticsearch database.

Data Enrichment & Threat Intelligence · Elasticsearch Feed · Feed

Details

ID	`ElasticsearchFeed`
Provider	ElasticNV
Category	Data Enrichment & Threat Intelligence
From Version	5.5.0
Docker Image	`demisto/elasticsearch:1.0.0.5954979`
Supported Modules	Agentix XSIAM

README

Overview

Fetch indicators stored in an Elasticsearch database.

The Cortex XSOAR Feed contains system indicators saved in an Elasticsearch index.
The Cortex XSOAR MT Shared Feed contains indicators shared by a tenant account in a multi-tenant environment.
The Generic Feed contains a feed in a format specified by the user.

Supports version 6 and later.
This integration was integrated and tested with versions 6.6.2, 7.3, 8.4.1 of Elasticsearch.

Configure Elasticsearch Feed in Cortex

Parameter	Description	Required
Server URL		True
Name (see ?->Authentication)	Provide Username + Passoword instead of API key + API ID	False
Password		False
Client type	For Elasticsearch version 7 and below, select ‘Elasticsearch’. For Elasticsearch server version 8, select ‘Elasticsearch_v8’. In some hosted ElasticSearch environments, the standard ElasticSearch client is not supported. If you encounter any related client issues, please consider using the OpenSearch client type.	False
Trust any certificate (not secure)		False
Use system proxy settings		False
Feed Type	The Cortex XSOAR Feed contains system indicators saved in an Elasticsearch index. The Cortex XSOAR MT Shared Feed contains indicators shared by a tenant account in a multi-tenant environment. Generic Feed contains a feed in a format specified by the user	False
Fetch indicators		False
First Fetch Time	Determine how far to look back for fetched indicators (<number> <time unit>, e.g., 12 hours, 7 days).	False
Fetch Limit	The maximal number of indicators that could be fetched in a fetch cycle.	False
Indicator Reputation	Indicators from this integration instance will be marked with this reputation	False
Source Reliability	Reliability of the source providing the intelligence data	True
Traffic Light Protocol Color	The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed	False
Feed Fetch Interval		False
Tags	Supports CSV values.	False
Bypass exclusion list	When selected, the exclusion list is ignored for indicators from this feed. This means that if an indicator from this feed is on the exclusion list, the indicator might still be added to the system.	False
Indicator Value Field	Source field that contains the indicator value in the index. Relevant for generic feed type only.	False
Indicator Type Field	Source field that contains the indicator type in the index. Relevant for generic feed type only.	False
Indicator Type	Default indicator type used in case no “Indicator Type Field” was provided. Relevant for generic feed type only.	False
Index from Which To Fetch Indicators	A comma-separated list of indexes. If empty, searches all indexes.	False
Time Field Type		False
Index Time Field	Used for sorting and limiting data. If empty, results are not sorted. Relevant for generic feed type only.	False
Query	Elasticsearch query to execute when fetching indicators from Elasticsearch	False

Commands

You can execute these commands from the CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

es-get-indicators

Gets indicators available in the configured Elasticsearch database.

Base Command

es-get-indicators

Input

Argument Name	Description	Required
limit	The maximum number of indicators to fetch. Default is 50.	Required

Context Output

There is no context output for this command.

Known Limitations

Indicator Type Case Sensitivity: The indicator type field in Elasticsearch must exactly match the indicator type in Cortex XSOAR, including case sensitivity. For example, if the indicator type in XSOAR is File (with uppercase ‘F’), then the indicator type in Elasticsearch must also be File, not file (lowercase). If there is a case mismatch, indicators will not be created in XSOAR.

To resolve this issue:

Check the existing indicator types in XSOAR by navigating to: <XSOAR Domain>/configuration/indicator/types
Update your Elasticsearch documents so that the indicator type field matches the exact case of the type in XSOAR
For the Generic Feed type, ensure the field specified in the “Indicator Type Field” parameter returns values that match XSOAR’s indicator types exactly

Configuration parameters

url — Server URL (required)
credentials — Name (see ?->Authentication)
client_type — Client type
insecure — Trust any certificate (not secure)
proxy — Use system proxy settings
feed_type — Feed Type
feed — Fetch indicators
fetch_time — First Fetch Time
fetch_limit — Fetch Limit
feedReputation — Indicator Reputation
feedReliability — Source Reliability (required)
tlp_color — Traffic Light Protocol Color
feedExpirationPolicy —
feedExpirationInterval —
feedFetchInterval — Feed Fetch Interval
feedTags — Tags
feedBypassExclusionList — Bypass exclusion list
src_val — Indicator Value Field
src_type — Indicator Type Field
default_type — Indicator Type
fetch_index — Index from Which To Fetch Indicators
time_method — Time Field Type
time_field — Index Time Field
es_query — Query
feedIncremental — Incremental Feed
enrichmentExcluded — Enrichment Excluded

Commands (1)

es-get-indicators
Gets indicators available in the configured Elasticsearch database.

Source

FeedElasticsearch.py FeedElasticsearch_test.py FeedElasticsearch.yml FeedElasticsearch_description.md

Download History Raw

category: Data Enrichment & Threat Intelligence
provider: ElasticNV
commonfields:
  id: ElasticsearchFeed
  version: -1
configuration:
- display: Server URL
  name: url
  required: true
  type: 0
  section: Connect
- additionalinfo: Provide Username + Passoword instead of API key + API ID
  display: Name (see ?->Authentication)
  name: credentials
  type: 9
  required: false
  section: Connect
- additionalinfo: In some hosted ElasticSearch environments, the standard ElasticSearch client is not supported. If you encounter any related client issues, please consider using the OpenSearch client type.
  defaultvalue: OpenSearch
  display: Client type
  name: client_type
  options:
  - ElasticSearch
  - OpenSearch
  - Elasticsearch_v8
  type: 15
  required: false
  section: Connect
- defaultvalue: 'false'
  display: Trust any certificate (not secure)
  name: insecure
  type: 8
  required: false
  section: Connect
  advanced: true
- display: Use system proxy settings
  name: proxy
  type: 8
  required: false
  section: Connect
  advanced: true
- additionalinfo: The Cortex XSOAR Feed contains system indicators saved in an Elasticsearch index. The Cortex XSOAR MT Shared Feed contains indicators shared by a tenant account in a multi-tenant environment. Generic Feed contains a feed in a format specified by the user
  defaultvalue: Cortex XSOAR MT Shared Feed
  display: Feed Type
  name: feed_type
  options:
  - Cortex XSOAR Feed
  - Cortex XSOAR MT Shared Feed
  - Generic Feed (requires additional configurations)
  type: 15
  required: false
  section: Collect
  advanced: true
- display: Fetch indicators
  name: feed
  defaultvalue: 'true'
  type: 8
  required: false
  section: Collect
- additionalinfo: Determine how far to look back for fetched indicators (<number> <time unit>, e.g., 12 hours, 7 days).
  defaultvalue: 3 days
  display: First Fetch Time
  name: fetch_time
  type: 0
  required: false
  section: Collect
- defaultvalue: '10000'
  additionalinfo: The maximal number of indicators that could be fetched in a fetch cycle.
  display: Fetch Limit
  name: fetch_limit
  type: 0
  required: false
  section: Collect
- additionalinfo: Indicators from this integration instance will be marked with this reputation
  defaultvalue: feedInstanceReputationNotSet
  display: Indicator Reputation
  name: feedReputation
  options:
  - None
  - Good
  - Suspicious
  - Bad
  required: false
  type: 18
  section: Collect
- additionalinfo: Reliability of the source providing the intelligence data
  display: Source Reliability
  name: feedReliability
  options:
  - A - Completely reliable
  - B - Usually reliable
  - C - Fairly reliable
  - D - Not usually reliable
  - E - Unreliable
  - F - Reliability cannot be judged
  type: 15
  required: true
  defaultvalue: F - Reliability cannot be judged
  section: Collect
- display: 'Traffic Light Protocol Color'
  name: tlp_color
  options:
  - RED
  - AMBER
  - GREEN
  - WHITE
  type: 15
  required: false
  additionalinfo: The Traffic Light Protocol (TLP) designation to apply to indicators fetched from the feed
  section: Collect
- defaultvalue: 'indicatorType'
  display: ''
  name: feedExpirationPolicy
  type: 17
  required: false
  options:
  - never
  - interval
  - indicatorType
  section: Collect
  advanced: true
- defaultvalue: '20160'
  display: ''
  name: feedExpirationInterval
  type: 1
  required: false
  section: Collect
  advanced: true
- display: Feed Fetch Interval
  name: feedFetchInterval
  type: 19
  required: false
  defaultvalue: '1'
  section: Collect
  advanced: true
- additionalinfo: Supports CSV values.
  display: Tags
  name: feedTags
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: When selected, the exclusion list is ignored for indicators from this feed. This means that if an indicator from this feed is on the exclusion list, the indicator might still be added to the system.
  display: Bypass exclusion list
  name: feedBypassExclusionList
  type: 8
  required: false
  section: Collect
  advanced: true
- additionalinfo: Source field that contains the indicator value in the index.  Relevant for generic feed type only.
  display: Indicator Value Field
  name: src_val
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Source field that contains the indicator type in the index.  Relevant for generic feed type only.
  display: Indicator Type Field
  name: src_type
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Default indicator type used in case no "Indicator Type Field" was provided.  Relevant for generic feed type only.
  display: Indicator Type
  name: default_type
  type: 0
  required: false
  section: Collect
  advanced: true
- display: Index from Which To Fetch Indicators
  name: fetch_index
  type: 0
  required: false
  additionalinfo: A comma-separated list of indexes. If empty, searches all indexes.
  section: Collect
  advanced: true
- display: Time Field Type
  name: time_method
  type: 15
  required: false
  defaultvalue: Simple-Date
  options:
  - Simple-Date
  - Timestamp-Seconds
  - Timestamp-Milliseconds
  section: Collect
  advanced: true
- additionalinfo: Used for sorting and limiting data. If empty, results are not sorted. Relevant for generic feed type only.
  display: Index Time Field
  name: time_field
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Elasticsearch query to execute when fetching indicators from Elasticsearch
  defaultvalue: '*'
  display: Query
  name: es_query
  type: 0
  required: false
  section: Collect
  advanced: true
- additionalinfo: Incremental feeds pull only new or modified indicators that have been sent from the integration. As the determination if the indicator is new or modified happens on the 3rd-party vendor's side, and only indicators that are new or modified are sent to Cortex XSOAR, all indicators coming from these feeds are labeled new or modified.
  defaultvalue: 'true'
  display: Incremental Feed
  hidden: true
  name: feedIncremental
  type: 8
  required: false
  section: Collect
  advanced: true
- display: Enrichment Excluded
  name: enrichmentExcluded
  type: 8
  required: false
  defaultvalue: 'false'
  additionalinfo: Select this option to exclude the fetched indicators from the enrichment process.
  hidden:
  - xsoar_on_prem
  section: Collect
description: Fetches indicators stored in an Elasticsearch database.
display: Elasticsearch Feed
name: ElasticsearchFeed
script:
  commands:
  - arguments:
    - defaultValue: '50'
      description: The maximum number of indicators to fetch.
      name: limit
      required: true
    description: Gets indicators available in the configured Elasticsearch database.
    name: es-get-indicators
  dockerimage: demisto/elasticsearch:1.0.0.5954979
  feed: true
  runonce: false
  script: '-'
  subtype: python3
  type: python
fromversion: 5.5.0
tests:
- Elasticsearch_Fetch_Custom_Indicators_Test
- Elasticsearch_Fetch_Demisto_Indicators_Test
sectionorder:
- Connect
- Collect