Kafka V2 Deprecated

Deprecated. Use the Kafka v3 integration instead. The Open source distributed streaming platform.

Messaging and Conferencing · Kafka

Details

IDKafka V2
ProviderOpen Source
CategoryMessaging and Conferencing
From Version5.0.0
Docker Imagedemisto/pykafka:1.0.0.19034
Supported ModulesAgentix XSIAM

README

Use the Kafka integration to manage messages and partitions.

This integration was integrated and tested with version 2.6 of Kafka.

Configure Kafka v2 on Cortex XSOAR

  1. Navigate to Settings > Integrations > Servers & Services.
  2. Search for Kafka v2.
  3. Click Add instance to create and configure a new integration instance.
    • Name: a meaningful name for the integration instance.
    • Use proxy
    • CSV list of Kafka brokers to connect to, e.g., ip:port,ip2:port2
    • Do not validate server certificate (insecure)
    • CA certificate of Kafka server (.cer)
    • Client certificate (.cer)
    • Client certificate key (.key)
    • Additional password (if required)
    • Topic to fetch incidents from
    • Offset to fetch incidents from
    • Max number of messages to fetch
    • Incident type
    • Enable debug (will post Kafka connection logs to the War Room)
  4. Click Test to validate the URLs, token, and connection.

Commands

You can execute these commands from the Cortex XSOAR CLI, as part of an automation, or in a playbook. After you successfully execute a command, a DBot message appears in the War Room with the command details.

  1. Print all partitions for a topic: kafka-print-topics
  2. Publish a message to Kafka: kafka-publish-msg
  3. Consume a single Kafka message: kafka-consume-msg
  4. Print all partitions for a topic: kafka-fetch-partitions

1. Print all partitions for a topic

Prints all partitions of a topic.

Base Command

kafka-print-topics

Input

There is no input for this command.

Context Output
Path Type Description
Kafka.Topic.Name string Topic name.
Kafka.Topic.Partitions.ID Number Topic partition ID.
Kafka.Topic.Partitions.EarliestOffset Number Topic partition earliest offset.
Kafka.Topic.Partitions.LatestOffset Number Topic partition latest offset.

 

Command Example

!kafka-print-topics

Context Example

Human Readable Output

2. Publish a message to Kafka

Publishes a message to Kafka. 

Base Command

kafka-publish-msg

Input
Argument Name Description Required
topic A topic to filter by. Required
value Message value (string) Required
partitioning_key Message partition key (number) Optional

 

Context Output

There is no context output for this command.

Command Example

!kafka-publish-msg topic=test value="test message"

 

Human Readable Output

3. Consume a single Kafka message

Consumes a single Kafka message.

Base Command

kafka-consume-msg

 

Input
Argument Name Description Required
topic A topic to filter by Required
offset Message offset to filter by ("Earliest", "Latest", or any other offset number) Optional
partition Partition (number) Optional

 

Context Output
Path Type Description
Kafka.Topic.Name string Topic name
Kafka.Topic.Message.Value string Message value
Kafka.Topic.Message.Offset number Offset of the value in the topic

 

Command Example

!kafka-consume-msg topic=test offset=latest

Context Example

Human Readable Output

4. Print all partitions for a topic

Prints all partitions for a topic.

Base Command

kafka-fetch-partitions

Input
Argument Name Description Required
topic A topic to filter by Required

 

Context Output
Path Type Description
Kafka.Topic.Name string Topic name
Kafka.Topic.Partition number Number of partitions for the topic

 

Command Example

!kafka-fetch-partitions topic=test

Context Example

Human Readable Output

Configuration parameters

  • brokers — CSV list of Kafka brokers to connect to, e.g. 172.16.20.207:9092,172.16.20.234:9093 (required)
  • use_ssl — Use TLS for connection
  • ca_cert — CA certificate of Kafka server (.cer)
  • client_cert — Client certificate (.cer)
  • client_cert_key — Client certificate key (.key)
  • additional_password — Client certificate key password (if required)
  • topic — Topic to fetch incidents from (Required for fetch incidents)
  • partition — CSV list of partitions to fetch messages from
  • offset — Offset to fetch messages from (Exclusive)
  • max_messages — Max number of messages to fetch
  • isFetch — Fetch incidents
  • incidentType — Incident type
  • max_bytes_per_message — Max number of bytes per message

Commands (4)

  • kafka-consume-msg

    Consumes a single Kafka message.

  • kafka-fetch-partitions

    Fetch partitions for a topic.

  • kafka-print-topics

    Prints all partitions of a topic.

  • kafka-publish-msg

    Publishes a message to Kafka.

Source

Kafka_V2.py Kafka_V2_test.py Kafka_V2.yml
Download History Raw 
from Kafka_V2 import create_certificate
import os


def test_create_certificate():
    ca_cert = 'dummy_cert'
    client_cert = 'dummy_client'
    key = 'dummy_key'
    password = 'dummy_pass'
    res = create_certificate(ca_cert, client_cert, key, password)
    assert res.password == password
    with open(res.certfile, 'rb') as f:
        assert f.read() == client_cert
    os.remove(res.certfile)
    with open(res.cafile, 'rb') as f:
        assert f.read() == ca_cert
    os.remove(res.cafile)
    with open(res.keyfile, 'rb') as f:
        assert f.read() == key
    os.remove(res.keyfile)