# Kafka

Apica Ascent can run streaming analytics from one or more Kafka topics. Both push and pull modes are supported. The data in Kafka can be pushed to Apica Ascent if the Kafka servers are running inside the customer network. If the Kafka endpoint is accessible to the Apica Ascent network Apica Ascent can pull the data directly from Kafka.

## PUSH Method: Pushing data using logstash

![Push data to Kafka](https://2948796384-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LmzGprckLqwd5v6bs6m%2Fuploads%2FzwVWw9JxhWX338eh0ah3%2Fkafka_push.png?alt=media\&token=62dd5283-d44f-4166-91cb-0cf0f78034dd)

Logstash instances deployed in the customer network can read data from Kafka and push it to Apica Ascent. Use the below configuration to read from Kafka.

### Input Logstash configuration

```
input {
    kafka {
            bootstrap_servers => "localhost:9092"
            topics => "test_topic"
    }
}
```

### Output Logstash Configuration

```
output {
   http {
        url => "https://ascent-dns-or-ip/v1/json_batch"
        headers => { "Authorization" => "Bearer <Auth token>" }
        http_method => "post"
        format => "json_batch"
        content_type => "json_batch"
        pool_max => 300
        pool_max_per_route => 100
        socket_timeout => 60
       }
}
```

You can additionally control the data organization by specifying additional fields

```
filter {
  mutate {
    add_field => { "cluster_id" => "demo-http-test" }
    add_field => { "namespace" => "namespace_name" }
    add_field => { "app_name" => "application_name" }
  }
}
```

## PULL Method : Pulling Data from Kafka topics

Apica Ascent can pull data using Kafka Input Plugins. This method requires the Kafka endpoint to be reachable from Apica Ascent network.

Kafka endpoint, Topic name, namespace, and application name are needed to configure the Kafka input plugin. Namespace and Application define how the data is partitioned in Apica Ascent, see [here for more information](https://docs.apica.io/observe/log-management-overview/logs-terminology).

![Kafka Input App Extension](https://2948796384-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LmzGprckLqwd5v6bs6m%2Fuploads%2FxURnxvrQiofnepv4ygId%2Fkafka_pull.png?alt=media\&token=a3b430ce-bc05-4249-81ef-e9cdec170697)

{% hint style="info" %}
Multiple Pull extensions can be created and running simultaneously
{% endhint %}

Creating a Kafka pull application can be done from the "*Explore*" -> "*App Extensions*"

<figure><img src="https://2948796384-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LmzGprckLqwd5v6bs6m%2Fuploads%2Ff4jlqZH4hFbJ2iYJmbd2%2FScreen%20Shot%202023-01-02%20at%205.18.11%20PM.png?alt=media&#x26;token=8326d839-dbee-47d8-a5ec-2112ada2193f" alt=""><figcaption><p>Kafka/Confluent App Extension</p></figcaption></figure>

The plugin configuration page provides the various options to pick a Kafka topic, consumer group etc.

<figure><img src="https://2948796384-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LmzGprckLqwd5v6bs6m%2Fuploads%2Fx1poy9tfs7KB7BF2bVGE%2FScreen%20Shot%202023-01-02%20at%205.18.40%20PM.png?alt=media&#x26;token=61415cd4-c8e2-45aa-b6cc-fff6049b2925" alt=""><figcaption></figcaption></figure>