Apache Kafka is a distributed, durable, real-time event streaming platform. It goes beyond a message queue by providing scalability, persistence, and stream processing capabilities.In this guide, we’ll quickly spin up Kafka with Docker, explore it with CLI tools, and integrate it into a Spring Boot application.


1. What is Kafka?
Apache Kafka is a distributed, durable, real-time event streaming platform.
It was originally developed at LinkedIn and is now part of the Apache Software Foundation.
Kafka is designed for high-throughput, low-latency data pipelines, streaming analytics, and event-driven applications.


What is an Event?
An event is simply a record of something that happened in the system.
Each event usually includes:

Key → identifier (e.g., user ID, order ID).


Value → the payload (e.g., “order created with total = $50”).


Timestamp → when the event occurred.

Example event:
⛶{
"key": "order-123",
"value": { "customer": "Alice", "total": 50 },
"timestamp": "2025-09-19T10:15:00Z"
}


What is an Event Streaming Platform?
An event streaming platform is a system designed to handle continuous flows of data — or events — in real time.
Instead of working in batches (processing data after the fact), it allows applications to react as events happen.


2. What Kafka Can Do
Kafka is more than a message queue—it's a real-time event backbone for modern systems.


Messaging Like a Message Queue
Kafka decouples producers and consumers, enabling asynchronous communication between services.
Example:
A banking system publishes transaction events to Kafka. Fraud detection, ledger updates, and notification services consume these events independently.


Event Streaming
Kafka streams data in real time, allowing systems to react instantly.
Example:
An insurance platform streams claim events to trigger automated validation, underwriting checks, and customer updates in real time.


Data Integration
Kafka Connect bridges Kafka with databases, cloud storage, and analytics platforms.
Example:
A semiconductor company streams sensor data from manufacturing equipment into a data lake for predictive maintenance and yield optimization.


Log Aggregation
Kafka centralizes logs from multiple services for monitoring and analysis.
Example:
An industrial automation system sends logs from PLCs and controllers to Kafka, where they’re consumed by a monitoring dashboard for anomaly detection.


Replayable History
Kafka retains events for reprocessing or backfilling.
Example:
An insurance company replays past policy events to train a model that predicts claim risk or customer churn. This avoids relying on static snapshots and gives the model a dynamic, time-aware view of behavior.


Scalable Microservices Communication
Kafka handles high-throughput messaging across distributed services.
Example:
A financial institution uses Kafka to coordinate customer onboarding, KYC checks, and account provisioning across multiple microservices.


3. Core Concepts
Let’s break down the key components that power Kafka’s event-driven architecture:


Concept
Description




Event
The basic unit in Kafka, including key, value, and timestamp.


Topic
A category for events, like a database table.


Partition
A Topic can be split into multiple partitions for parallelism and scalability.


Producer
Application that sends events to Kafka.


Consumer
Application that reads events from Kafka.


Consumer Group
A group of Consumers that share the load of processing.


Broker
Kafka server node storing data and handling requests.


Offset
A unique ID for each record within a Partition.





4. QuickStart with Docker
This configuration sets up a single-node Kafka broker using the KRaft. It’s ideal for development, testing scenarios
⛶name: kafka
services:
kafka:
image: apache/kafka:4.1.0
container_name: kafka
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: BROKER://:9092,CONTROLLER://:9093
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: BROKER://localhost:9092
KAFKA_CLUSTER_ID: "kafka-1"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_LOG_DIRS: /var/lib/kafka/data
volumes:
- kafka_data:/var/lib/kafka/data
ports:
- "9092:9092"
volumes:
kafka_data:


How to Run
Start the Kafka container using:
⛶docker compose upKafka will be available at localhost:9092 for producers and consumers, and internally at localhost:9093 for controller communication.


5. Kafka CLI
Before running Kafka commands, log into the Kafka container:
⛶docker container exec -it localhost bash


Create Topic
Create a topic named quickstart with one partition and a replication factor of 1:
⛶/opt/kafka/bin/kafka-topics.sh --create \
--bootstrap-server localhost:9092 \
--replication-factor 1 \
--partitions 1 \
--topic quickstart


List Topic
Check all existing topics:
⛶/opt/kafka/bin/kafka-topics.sh --list \
--bootstrap-server localhost:9092


Consume Message
Read messages from the order topic starting from the beginning:
⛶/opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic quickstart \
--from-beginning


Send Message
You can send messages to the quickstart topic using either direct input or a file.


Option A: Send a single message
⛶echo 'This is Event 1' | \
/opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server localhost:9092 \
--topic quickstart


Option B: Send multiple messages from a file
⛶echo 'This is Event 2' messages.txt
echo 'This is Event 3' messages.txt
cat messages.txt | \
/opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server localhost:9092 \
--topic quickstart


5. Spring Boot Integration
This configuration enables seamless integration between a Spring Boot application and an Apache Kafka broker. It defines both producer and consumer settings for message serialization, deserialization, and connection behavior.


pom.xml
⛶org.springframework.boot
spring-boot-starter-web
3.4.9



org.springframework.kafka
spring-kafka
3.3.9



org.projectlombok
lombok
1.18.30
true


applicaiton.yml
⛶spring:
kafka:
bootstrap-servers: localhost:9092
template:
default-topic: orders
consumer:
group-id: quickstart-group
auto-offset-reset: latest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
properties:
spring.json.trusted.packages: "dev.aratax.messaging.kafka.model"
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer


Topic Setup
⛶@Bean
public NewTopic defaultTopic() {
return new NewTopic("orders", 1, (short) 1);
}


Event Model
⛶public class OrderEvent {
private String id;
private Status status;
private BigDecimal totalAmount;
private Instant createdAt = Instant.now();
private String createdBy;

public enum Status {
IN_PROGRESS,
COMPLETED,
CANCELLED
}
}


Producer Example
⛶@RestController
@RequestMapping("/api")
@RequiredArgsConstructor
public class OrderEventController {

private final KafkaTemplateString, OrderEvent kafkaTemplate;

@PostMapping("/orders")
public String create(@RequestBody OrderEvent event) {
event.setId(UUID.randomUUID().toString());
event.setCreatedAt(Instant.now());
kafkaTemplate.sendDefault(event.getId(), event);
return "Order sent to Kafka";
}
}


Consumer Example
⛶@Component
public class OrderEventsListener {

@KafkaListener(topics = "orders")
public void handle(OrderEvent event) {
System.out.println("Received order: " + event);
}
}


6. Demo Project
I built a demo project using Spring Boot and Kafka to demonstrate basic producer/consumer functionality.
Check it out on GitHub: springboot-kafka-quickstart


7. Key Takeaways

Kafka is more than a message queue—it's a scalable, durable event streaming platform.
Events are central to Kafka’s architecture, enabling real-time data flow across systems.
Docker makes setup easy, allowing you to spin up Kafka locally for development and testing.
Kafka CLI tools help you explore topics, produce messages, and consume events interactively.
Spring Boot integration simplifies Kafka usage with built-in support for producers and consumers.
Real-world use cases span industries like banking, insurance, semiconductor, and automation.



8. Conclusion
Apache Kafka empowers developers to build reactive, event-driven systems with ease. Whether you're streaming financial transactions, processing insurance claims, or monitoring factory equipment, Kafka provides the backbone for scalable, real-time communication.With Docker and Spring Boot, you can get started in minutes—no complex setup required. This quickstart gives you everything you need to explore Kafka hands-on and begin building production-grade event pipelines.Ready to go deeper? Try explore its design/implementation, stream processing, or Kafka Connect integrations next.