参考链接
syllabus
| Session | Topic | Detailed Topics |
|---|---|---|
| 20 | KAFKA |
1. Kafka - concepts, how it works and how message is sent to partition 2. Consumer Group, assignment strategy 3. Message in Order |
| 21 | KAFKA2 |
1. Kafka Duplicate Message 2. Kafka Message Loss 3. Poison Failure, DLQ 4. Kafka Security (SASL, ACLs, Encrypt etc) |
Here are the differences between Controller and RestController as well as those between Kafka and RabbitMQ:
Differences between Kafka and RabbitMQ
- Message Model
- Kafka: It is a distributed streaming platform that follows a publish-subscribe model with a messaging system based on topics. Producers send messages to topics, and consumers subscribe to topics to receive messages. It is designed for high-throughput, real-time data streaming and is often used for applications like log aggregation, real-time analytics, and event-driven architectures.
- RabbitMQ: It is a message broker that supports multiple messaging models, including publish-subscribe, point-to-point (queue-based), and request-reply. It is more flexible in terms of message routing and can handle a variety of messaging use cases, such as task queues, asynchronous processing, and message-based communication between different components of an application.
- Data Durability and Persistence
- Kafka: It stores messages in a distributed and durable log structure. Messages are written to disk and can be configured to have multiple replicas for fault tolerance. It has a high level of data durability and can handle large volumes of data with efficient disk I/O operations.
- RabbitMQ: It also supports message persistence, but its approach is different. Messages can be persisted to disk, and it uses a combination of in-memory queues and disk storage to manage messages. The durability can be configured based on the specific requirements of the application.
- Performance and Throughput
- Kafka: It is optimized for high throughput and low latency in handling large volumes of data. It can handle a large number of messages per second and is well-suited for applications that require real-time processing of streaming data.
- RabbitMQ: It is more focused on reliability and flexibility rather than extreme high throughput. It can handle a significant amount of messages, but its performance characteristics are different from Kafka. It is often used in scenarios where message reliability and guaranteed delivery are more important than raw throughput.
- Use Cases
- Kafka: Commonly used in big data processing, real-time analytics, event sourcing, and applications where there is a need to handle and process a large volume of streaming data in real-time. It is also popular in microservices architectures for event-driven communication between services.
- RabbitMQ: Used in a wide range of applications for message queuing, task scheduling, asynchronous communication between different parts of an application, and in enterprise integration scenarios where reliable message delivery and flexible routing are required.
同一个消费组对于一个分区是共享同一个offset的是吧
是的，在 Kafka 里，同一个消费组对于一个分区共享同一个 offset。下面从原理、目的和影响等方面详细解释：
原理
- 分区分配机制：Kafka 为了保证消息在分区内的顺序消费和避免重复消费，采用了一个分区只能被一个消费组内的一个消费者消费的策略。这意味着在同一个消费组中，针对特定的一个分区，只会有一个消费者负责处理该分区的消息。既然只有一个消费者处理这个分区，那么该分区对应的消费偏移量（offset）在这个消费组内就是唯一的，也就是共享同一个 offset。
- offset 提交：消费者在消费完消息后，需要向 Kafka 提交 offset，以此告知 Kafka 该消费组已经消费到了分区的哪个位置。在一个消费组里，无论哪个消费者实际消费了分区的消息，最终提交的都是该分区统一的 offset。例如，消费者 A 消费了分区 P0 的部分消息后提交 offset，这个 offset 就代表了整个消费组在分区 P0 的消费进度。
目的
- 保证消息顺序性：确保同一个消费组对分区内的消息按顺序处理。如果每个消费者都有自己独立的 offset，就可能出现多个消费者乱序消费分区消息的情况，破坏了消息在分区内的顺序性。
- 避免重复消费：统一的 offset 能让消费组清楚知道哪些消息已经被消费过。当消费者出现故障重启或者进行分区重平衡后，能从正确的 offset 位置继续消费，避免重复处理已经消费过的消息。
影响
- 故障恢复：若消费组内负责某分区的消费者出现故障，Kafka 会将该分区分配给其他消费者，新的消费者会从共享的 offset 位置继续消费，保证消费的连续性。
- 分区重平衡：当消费组发生分区重平衡时，所有消费者会重新分配分区，每个分区对应的共享 offset 能帮助新接手的消费者准确找到消费起点。
What is kafka dead letter queue and how do you handle it
A Kafka dead - letter queue (DLQ) is a special topic in a Kafka cluster that is used to store messages that cannot be successfully processed for some reason. Here’s an overview of what it is and how to handle it:
- Purpose: The main purpose of a DLQ is to prevent messages from being lost when they encounter processing failures. Instead of discarding the messages, they are sent to the DLQ for further analysis and possible re - processing.
- How Messages End Up in the DLQ: Messages can end up in the DLQ due to various reasons, such as application - level errors (e.g., incorrect message format, missing required fields), network issues, or problems with the processing logic. When a consumer fails to process a message after a certain number of retries, the message is typically redirected to the DLQ.
- Monitoring and Analysis:
- Monitor DLQ Size: Regularly check the size of the DLQ to identify any unusual spikes. A growing DLQ may indicate a problem with the message processing pipeline.
- Inspect Messages: Examine the messages in the DLQ to determine the cause of the processing failures. This can involve looking at the message payload, headers, and any error messages associated with the failed processing attempts.
- Error Resolution:
- Fix Application Bugs: If the errors are due to bugs in the consumer application, fix the code and redeploy the application.
- Data Correction: If the messages in the DLQ contain incorrect data, correct the data either manually or through an automated process.
- Message Re - processing:
- Manual Re - processing: For critical or complex messages, you may choose to re - process them manually. This allows for careful inspection and ensures that the processing is done correctly.
- Automated Re - processing: Set up a mechanism to automatically re - process messages from the DLQ. This can be a separate consumer that reads from the DLQ and attempts to process the messages again, perhaps with some additional error - handling logic.
- Configuration and Tuning:
- Retry Policies: Review and adjust the retry policies for your consumers. Determine the appropriate number of retries and the delay between retries based on the nature of your application and the expected failure scenarios.
- DLQ Topic Configuration: Configure the DLQ topic with appropriate settings, such as retention policies and replication factors. Ensure that the DLQ has enough storage to handle the potentially large number of messages.
Handling a Kafka dead - letter queue effectively requires a combination of monitoring, error resolution, message re - processing, and proper configuration to ensure the reliability and integrity of the message processing system.