Certificate Authentication

Introduction (Brief)

Apache Kafka vs. traditional message brokers (Brief)

Overview of Kafka Features and Ecosystem (Brief)

Apache Kafka On-premise vs. in the Cloud (Brief)

Apache Kafka variants & in Docker/Kubernetes (Brief)

 

Setup

Installing and Configuring Apache Kafka

Setting up Zookeeper to Manage the Kafka Cluster

Testing the Cluster

Testing IDE, APIs integration with Kafka

• Note** A lab with distributed Kafka Cluster will already be setup (Kafka with zookeeper)

 

Deep Dive into Kafka

Understanding Kafka Internals & architecture

 

• Cluster formation & Membership
• Zookeeper and its role
• Leader, follower role for Kafka brokers & zookeeper (Load balancing)
• The Controller –as per new version where ZK is not used (brief introduction)
• Kraft Mode –as per new version where ZK is not used (brief introduction)
• Topics, Partitions & Segments.
• Messages & batches
• Producers & consumers
• Consumer groups, offsets and fault tolerance.
• Brokers & replication.
• Fault tolerance & semantics.
• Important configurations.
• Physical storage & understanding underlying log/index files.
• Log retention & compaction

 

Understanding Kafka APIs

 

• Kafka Java Client APIs
• Kafka Producer Java API
• Kafka Consumer Java API
• Kafka AdminClient Java API
• Kafka Streams Java API
• Kafka Connect Java API
• Terminologies

 

Working with Kafka programmatically & understanding reliability

 

Kafka Producers

Constructing a Kafka Producer, publishing messages, configurations for producers, understanding serializers, interceptors, headers, partitions, consistency, retries, compression, quotas/ throttling etc.

Related configurations for optimized performance.

 

Kafka Consumers

Constructing a consumer, working with consumers, consumer Groups.

Subscribing & consuming from topics.

Understanding Polling & heartbeat thread, commits & offsets, fetch behaviour, auto offset or preferred read, rebalancing listeners, serializers & deserializers

Related configurations for optimized performance.

 

      Reliable Data Delivery

Reliability Guarantees & Validating System Reliability

Understanding semantics & ensuring

Data durability & retention

Important considerations

 

      Kafka Streaming API

      Kafka Connect API

Building Data Pipelines

Considerations

Kafka Connect Versus Producer and Consumer

Kafka Connect

 

Managing Kafka Programmatically

 

AdminClient Lifecycle: Creating, Configuring and Closing, Configuration management, Consumer group management, Cluster Metadata & Testing 

 

Managing, administering & Monitoring Kafka

 

Cross-Cluster Data Mirroring

Use Cases, Hub-and-Spokes Architecture, Active-Active Architecture, Active-Standby Architecture and Apache Kafka’s MirrorMaker.

 

Administering Kafka

Topic Operations, Consumer & Consumer Groups, Dynamic Configuration Changes

Partition Management

 

Monitoring Kafka

Using tools to monitor Kafka Cluster

Understanding emitted metrics from Kafka & zookeepers

Client, performance & Lag Monitoring

Kafka logs

Known issues and optimizing Kafka & its components

Troubleshooting

           

Summary & conclusion