Apache Pulsar

Apache Pulsar is a next-generation messaging platform that unifies traditional message queuing and event streaming into a single system. Built from the ground up for cloud-native deployments, it separates compute from storage to enable independent scaling and cost optimization.

Key capabilities

Unified messaging

Pulsar supports both publish-subscribe and message queue patterns in one platform. Topics can function as streams (multiple consumers) or queues (single consumer with failover).

Tiered storage

Messages are initially stored on BookKeeper for low-latency access, then automatically offloaded to cloud storage (S3, GCS, Azure Blob) for cost-effective long-term retention. This enables indefinite message retention without broker storage limits.

Multi-tenancy

Built-in tenant isolation with authentication, authorization, and quota management. Ideal for SaaS platforms and large organizations with multiple teams.

Geo-replication

Native support for synchronous and asynchronous replication across multiple datacenters and regions with configurable consistency guarantees.

Architecture

Brokers

Stateless compute layer handling message routing and serving. Can be scaled independently of storage.

BookKeeper (bookies)

Distributed write-ahead log providing low-latency durable storage for recent messages.

ZooKeeper

Coordination service for cluster metadata and configuration (being replaced by Pulsar's own metadata layer).

Tiered storage

Long-term message retention in cloud object storage with automatic offloading.

Core features

• Multiple subscription modes: Exclusive, Failover, Shared, and Key_Shared for flexible consumer patterns
• Message replay: Rewind to any point in time for reprocessing or debugging
• Schema registry: Built-in schema management with Avro, JSON, and Protobuf support
• Functions: Lightweight compute framework for serverless stream processing
• IO connectors: Pre-built connectors for popular systems (Kafka, Cassandra, Elasticsearch, etc.)

Protocol support

• Pulsar Protocol: Native binary protocol for maximum performance
• Kafka Protocol: Kafka API compatibility for easy migration
• MQTT: Via Pulsar MQTT proxy for IoT device connectivity
• AMQP: Via protocol handler
• REST API: HTTP interface for management and simple pub/sub

Deployment options

• Bare metal/VMs: Traditional deployment with binary packages
• Docker: Official container images available
• Kubernetes: Pulsar Operator for automated cluster management
• Cloud: StreamNative Cloud (managed service) and major cloud marketplaces

Manufacturing use cases

• Unified namespace: Single platform for both real-time control messages and analytics streaming
• IoT data ingestion: High-throughput ingestion from factory sensors with MQTT compatibility
• Predictive maintenance: Stream processing for equipment health monitoring
• Audit logging: Long-term retention of manufacturing events for compliance
• Supply chain events: Reliable messaging for distributed supply chain systems

Limitations

• Higher operational complexity than Kafka due to more components (brokers + BookKeeper + ZooKeeper)
• Smaller ecosystem and fewer pre-built connectors compared to Kafka Connect
• Community is growing but still much smaller than Kafka's established base
• Pulsar Functions (serverless processing) is less powerful than Kafka Streams for complex stateful operations
• MQTT proxy support is basic compared to dedicated MQTT brokers like Mosquitto