Telegraf + Apache Kafka: metrics streaming pipeline

Overview

Telegraf's Kafka output plugin enables streaming collected metrics to Apache Kafka for distributed processing, buffering, and distribution to multiple consumers.

Integration Architecture

Sensors/PLCs → Telegraf → Kafka Topic → Multiple Consumers (InfluxDB, Analytics, Alerts)

Configuration

[[outputs.kafka]]
  brokers = ["kafka:9092"]
  topic = "factory-metrics"
  data_format = "json"

Use Cases

• Decoupling data collection from storage
• Multi-destination data distribution
• Buffering during database maintenance
• Stream processing with Kafka Streams or ksqlDB
• Building unified data pipelines across plants

Tradeoffs & Considerations

• Adds infrastructure complexity (Kafka cluster)
• Provides durability and replay capabilities
• Enables multiple independent consumers
• Higher latency than direct database writes