CreateTopics Extension
#46
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
…ve protocol inspector links and clarifying APIVersions requirements.
…xistent topics, including error response details and interactive protocol inspector links
…s for invalid topics, including error response details
…lid partitions, detailing error responses and adding interactive protocol inspector links
… Produce stage description
…uction process, including validation steps, log file format, and updated response fields
…uction by consolidating validation, storage, and response steps into a single sentence
… format and add interactive protocol inspector link for cluster metadata topic's log file
…le records in a single RecordBatch, including validation, storage, and updated response fields
…e partitions, including validation, response structure, and log file requirements
…le topics and partitions, including validation, response structure, and offset management
…ng on baseOffset and log_start_offset handling in Kafka
…cluding multiple stages for handling various production scenarios such as invalid topics, partitions, and batch production
ryan-gang
force-pushed
the
create-topics
branch
2 times, most recently
from
5c8a486 to
b3c0cd6
Compare
… invalid topics and partitions, enhancing clarity on error handling and response structure
…tions in a single request, including validation, independent offset management, and nested response structure
…f invalid topics or partitions, enhancing consistency in error response messaging
… handling and response structure, ensuring consistency and accuracy
… and consistency in topic and partition handling
Introduce the first stage description for the CreateTopics API, detailing the inclusion of the API in the APIVersions response. This stage outlines the necessary tests and notes for implementation, setting the groundwork for subsequent stages.
…mentation Introduce detailed stage descriptions for the CreateTopics API, covering various scenarios such as invalid topic names, existing topics, and authorization checks. This addition enhances the documentation for implementing the API and provides clear guidelines for error handling and validation processes.
…ion and testing Enhance the stage description for the CreateTopics API by providing a clearer structure, including a dedicated section for the API and its tests. Update the MaxVersion requirement for the CreateTopics API and add links to interactive protocol inspectors for better guidance on request and response structures.
…eTopics API Introduce a new stage description that outlines the handling of `CreateTopics` requests with invalid topic names. This includes validation rules, expected error responses, and testing procedures to ensure compliance with Kafka's naming conventions.
…teTopics API Introduce a new stage description that details the handling of `CreateTopics` requests with reserved topic names. This includes validation rules, expected error responses, and testing procedures to ensure compliance with Kafka's naming conventions, particularly focusing on the `__cluster_metadata` topic and other system-reserved names.
…ics API Introduce a new stage description that details the process for detecting existing topics in the CreateTopics API. This includes validation steps, expected error responses, and testing procedures to ensure proper handling of the `TOPIC_ALREADY_EXISTS` error code. Additionally, links to interactive protocol inspectors for request and response structures are provided to aid in understanding the implementation.
Introduce a new stage description that outlines the process for successfully creating topics with valid parameters in the CreateTopics API. This includes detailed steps for topic creation, validation of parameters, and the expected response structure. Additionally, links to interactive protocol inspectors for request and response structures are provided, along with testing procedures to ensure proper implementation.
Introduce a new stage description that outlines the process for handling multiple topics in a single CreateTopics request. This includes detailed steps for validation, creation, and response handling for each topic, as well as testing procedures to ensure correct implementation. Links to interactive protocol inspectors for request and response structures are also provided.
Introduce a new stage description that outlines the process for handling CreateTopics requests with the `validate_only` flag. This includes detailed validation steps, expected responses, and testing procedures to ensure correct implementation without side effects. Links to interactive protocol inspectors for request and response structures are also provided.
… in CreateTopics API Introduce a new stage description that outlines the process for successfully creating topics with valid parameters in the CreateTopics API. This includes detailed steps for validation, metadata persistence, directory creation, and expected responses. Additionally, links to interactive protocol inspectors for request and response structures are provided, along with testing procedures to ensure proper implementation.
Update all stage descriptions for the CreateTopics API to reference version 6 instead of version 7. This includes adjustments to the interactive protocol inspector links and the testing procedures to ensure consistency across all stages. The changes clarify the expected request and response structures for the API.
Introduce new stage descriptions for the CreateTopics API, covering various scenarios including handling invalid topic names, partition values, replication factors, and authorization checks. This update enhances the documentation by providing clear guidelines for validation, error handling, and expected responses, ensuring a thorough understanding of the CreateTopics functionality.
…cs API Update the stage description for handling multiple topics in a single CreateTopics request to improve clarity. The title has been modified to specify "single CreateTopics request," ensuring better understanding of the functionality being documented.
Introduce multiple new stages for the CreateTopics API, focusing on various scenarios such as creating topics with valid, invalid, reserved, and existing names, as well as validation-only mode and batch creation. This update enhances the documentation and provides clear guidelines for implementing the CreateTopics functionality.
Adjust the MaxVersion for the CreateTopics API in the stage description to reflect the correct version 6, ensuring consistency with previous updates and clarifying the expected API behavior.
… description Clarify the instructions for implementing basic topic name validation by specifying that error responses for invalid topic names can be hard-coded. Additionally, note that checks against existing topics and system topics will be addressed in later stages. Update the testing command to include the server properties file for better clarity.
…xisting topic names Introduce a new stage that outlines the process for validating topic existence in the CreateTopics API. This includes error handling for existing and reserved topic names, along with detailed testing procedures and links to interactive protocol inspectors for better understanding of the topic metadata structure.
Clarify the handling of CreateTopics requests in validation mode by updating the stage description. Changes include improved wording for validation steps, specifying that no topic metadata is written to the log file, and enhancing the testing command for better clarity.
…reateTopics API Introduce new stage descriptions for handling CreateTopics requests, detailing the processes for creating a single topic and multiple topics in a single request. The updates include validation steps, metadata handling, and testing procedures, along with links to interactive protocol inspectors for enhanced understanding of the request and response structures.
Enhance the documentation for the CreateTopics API by clarifying the handling of metadata. Specifically, update the descriptions to indicate that both `TOPIC_RECORD` and `PARTITION_RECORD` are not written to the `__cluster_metadata` log file during topic creation. Additionally, refine the steps for writing `PARTITION_RECORD`s to ensure clarity on their creation for each partition. Adjust the testing command details to reflect the correct parameters for topic creation requests.
…ics API Refine the documentation for handling multiple topics in a CreateTopics request. Update the steps to clarify the writing of `PARTITION_RECORD`s to the `__cluster_metadata` log file for each partition and adjust the testing command to include the server properties file. Ensure that the handling of mixed success/failure scenarios is clearly articulated.
…es in topic creation handling Revise the documentation for the CreateTopics API stages, including updates to the handling of existing topic names and validation-only mode. Consolidate stages for clarity, ensuring that error codes and success cases are accurately described. Adjust the sequence of stages to improve the flow of topic creation scenarios.
rohitpaulk
requested changes
Jul 20, 2025
all-stages.md
Outdated
There was a problem hiding this comment.
Let's remove this extra file, stick to the md & definition files.
final-stages.md
Outdated
|
|
||
| - You'll need to parse the `CreateTopics` request in this stage to get the topic names. | ||
| - The official docs for the `CreateTopics` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_CreateTopics). | ||
| - Topic name validation logic is in the Kafka source code at `clients/src/main/java/org/apache/kafka/common/internals/Topic.java`. |
| Reserved topic names include: | ||
| - `__cluster_metadata` - KRaft metadata topic | ||
|
|
||
| If the topic name is reserved, the broker returns an error code of `42` (INVALID_REQUEST), with the `error_message` field containing `Creation of internal topic __cluster_metadata is prohibited.`. |
There was a problem hiding this comment.
Let's remove the "reserved topics" thing altogether - doesn't seem worth doing
| We've also created an interactive protocol inspector for the `__cluster_metadata` topic's log file: | ||
| - 🔎 [__cluster_metadata topic's log file](https://binspec.org/kafka-cluster-metadata) | ||
|
|
||
| This would help you understand the structure of the `TOPIC_RECORD` record inside the `__cluster_metadata` topic's log file. |
There was a problem hiding this comment.
Let's try to link directly to the TOPIC_RECORD bytes here
| 2. Checks that the topic doesn't already exist (from previous stages) | ||
| 3. Validates partition count and replication factor parameters | ||
| 4. Creates a new topic UUID (for successful topics) | ||
| 5. Writes a `TOPIC_RECORD` to the `__cluster_metadata` topic's log file (for successful topics) |
There was a problem hiding this comment.
Suggested change
| 5. Writes a `TOPIC_RECORD` to the `__cluster_metadata` topic's log file (for successful topics) | |
| 5. Writes `TOPIC_RECORD`s to the `__cluster_metadata` topic's log file |
There was a problem hiding this comment.
(Shorter, and the "successful topics" case is obvious)
| 7. Creates partition directories in the log directory (for successful topics) | ||
| 8. Returns individual results for each topic in the response array | ||
|
|
||
| Mixed success/failure scenarios are handled gracefully - if one topic fails validation, it doesn't prevent other valid topics from being created. |
There was a problem hiding this comment.
Suggested change
| Mixed success/failure scenarios are handled gracefully - if one topic fails validation, it doesn't prevent other valid topics from being created. | |
| If one topic fails validation, it doesn't prevent other valid topics from being created. |
| @@ -0,0 +1,50 @@ | |||
| In this stage, you'll add support for handling `CreateTopics` requests with multiple topics. | |||
There was a problem hiding this comment.
On second though, let's remove this stage altogether - unlike producing multiple messages, creating multiple topics doesn't seem like a very common use-case. Simpler to just remove - we can add it back when we have skippable tests
|
|
||
| - slug: "ve7" | ||
| primary_extension_slug: "creating-topics" | ||
| name: "CreateTopics with Invalid Topic Name" |
There was a problem hiding this comment.
Suggested change
| name: "CreateTopics with Invalid Topic Name" | |
| name: "CreateTopics with an invalid topic" |
(Remove unnecessary capitalization, and shorten)
|
|
||
| - slug: "fl3" | ||
| primary_extension_slug: "creating-topics" | ||
| name: "CreateTopics with Multiple Topics" |
There was a problem hiding this comment.
@ryan-gang you really need to pay more attention to detail re: consistency. Why is the above stage name "CreateTopics with a single topic", and this is "CreateTopics with Multiple Topics"? What's the reason for different capitalization?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce two new markdown files outlining the structure and error cases for the CreateTopics API in Kafka. The first file provides a logical progression of stages for implementing the API, while the second file details early error cases that can be handled with request parsing and response encoding. This aligns with existing documentation practices and enhances clarity for future development.