KAFKA EXPORTER HOW TO

This dashboard shows the consumption lag of each topic, including offset lag in quantity, estimated time in seconds, and message throughput per minute and second. This dashboard provides a deep dive into each topic’s health and shows the throughput in bytes and number of messages as well as the offsets. This is a comprehensive dashboard showing the overall healthiness of your Kafka cluster, including how many brokers are alive in the cluster metrics for your partitions JVM, throughput, requests, and response queues size Zookeeper connections and producer and consumer metrics.

KAFKA EXPORTER HOW TO

The details on how to configure the JMX Exporters and the Grafana Agent can be found here.īelow is a brief explanation of each dashboard. Only the lag consumption dashboard is fed by an external exporter, which is embedded in the Grafana Agent for ease of use. Most of the dashboards rely on collecting data through a JMX exporter running alongside each instance of your Kafka components, as an agent. Our Grafana Cloud integration, which is heavily based on this blog post from Confluent, provides dashboards for your Kafka Broker clusters, Zookeeper clusters, Kafka Connect clusters, Schema Registry clusters, and ksqldb clusters, along with a specific dashboard for topics information and consumption lag. Kafka is the most used event streaming platform worldwide, and its ecosystem includes a wide variety of pieces for data governance, querying, batch processing, and connectors. With the list of brokers, you can configure the collector accordingly.We are happy to announce that the Kafka integration is available for Grafana Cloud, our composable observability platform bringing together metrics, logs, and traces with Grafana.Īpache Kafka is an open source distributed event streaming platform that provides high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Select the authentication method accordingly to the options supported by the receivers/exporters.Īfter the cluster is provisioned, you can get the list of brokers by selecting the newly created cluster in the console and thenĬlicking in the View client information button.

You can create a MSK cluster following this guide. The Kafka receiver and exporters can be integrated with Amazon Managed Streaming for Apache Kafka (MSK). Integrating with Amazon Managed Streaming for Apache Kafka (MSK) The ADOT collector is tested with Kafka Versions 2.8.1 and 3.2.0.

The kafka exporter uses a synchronous producer that blocks and does not batch messages, therefore it should be used with batch and queued retry processors for higher throughput and resiliency.

It is a good practice to set the protocol_version property to match the version of Kafka used in your cluster.

They need to be pre-created or you need to set theĬluster configuration to auto create topics.

These components don't create Kafka topics on your behalf.Please find bellow the documentation for each of these components:

Upstream Kafka Exporter/Receiver documentation Amazon Managed Service for Prometheus (AMP)ĪWS Open Distro for OpenTelemetry Kafka receiver and exporter.

Kafka exporter