Certificate Authentication

Introduction

Overview

 Fundamentals of messaging systems.
 Use Cases & need of distributed messaging systems.
 What is Apache Kafka & why.
 Apache Kafka vs. traditional messaging systems. (Brief)
 Overview of Kafka Features and Ecosystem (Brief)
 Apache Kafka On-premise vs. in the Cloud (Brief)
 Apache Kafka variants & in Docker/Kubernetes (Brief)
 Data formats & structures.
 Stream processing vs. batch processing
 Kafka Features & ecosystem

Understanding Kafka fundamentals

Broker, Replicas, Controller, Leader, Follower, Topics, Partitions
Messages, Offsets, Consumer Group, Log distribution, Zookeeper and its need, KRaft, compression,
compaction etc.
Kafka APIs: Producers, Consumers, Stream processing, Connector

Kafka Installation & Setup

Installing a standalone Kafka Broker (single node Kafka cluster)
Installing and Configuring Apache Kafka multi-node cluster.
Setting up Zookeeper to Manage the Kafka Cluster.
The Controller –as per new version where ZK is not used (brief introduction of KRaft)
Testing the Cluster
Setting up the Development Environment (IDE integration)
Understanding basic important configs.
Working with command line

Deep Dive into Kafka

Understanding Kafka Internals & architecture

• Leader, follower role for Kafka brokers (Load balancing)
• Zookeeper and its role (Leader-follower, Znodes)
• Understanding fault tolerance & messaging semantics.
• Understanding reliability guarantees & consistency.
• Physical storage & understanding underlying log segments/index files.
• Log durability, retention & compaction
• The Minimum In-Sync Replicas
• Important Server, Client Configs

Understanding Kafka APIs

• Kafka Java Client APIs
• Kafka Producer Java API
• Kafka Consumer Java API
• Kafka AdminClient Java API

Kafka Producers

Constructing a Kafka Producer, publishing messages, configurations for producers,
understanding serializers, interceptors, headers, partitions, consistency, retries,
compression, quotas/ throttling etc.
Related configurations for optimized performance.

Kafka Consumers

Constructing a consumer, working with consumers, consumer Groups.
Subscribing & consuming from topics.
Understanding Polling & heartbeat thread, commits & offsets, fetch behaviour, auto offset
or preferred read, rebalancing listeners, serializers & deserializers
Related configurations for optimized performance.

Working with command line
Working with IDE (basic applications for producing, consuming & stream processing)

Kafka Streams Java API (brief)
Kafka Connect Java API (brief)
Kafka Connect Versus Producer and Consumer (brief)

Administration & operations

CLI

• Topic operations.
• Balancing leadership.
• Checking consumer position & consumer groups.
• Mirroring data between clusters
• Scaling or de-scaling cluster.
• Clean-up policy (compact/delete)
• Dynamic Configuration Changes
• Partition Management

Managing Kafka Programmatically

• AdminClient Overview
• AdminClient Lifecycle: Creating, Configuring and Closing
• Configuration management
• Consumer group management
• Cluster Metadata

Monitoring Kafka

• Using tools to monitor Kafka Cluster.
• Understanding emitted metrics from Kafka & zookeepers.
• Client, performance & Lag Monitoring.
• Kafka logs.
• Known issues and optimizing Kafka & its components.
• Troubleshooting.

Multi-cluster architectures (Additionals)

• Hub-and-Spokes Architecture
• Active-Active Architecture
• Active-Standby Architecture
• Cross-Cluster Data Mirroring – MirrorMaker

Summary & Conclusion
FAQs.