Skip to content

Commit 318a9b4

Browse files
ryan-gangryan-gang
committed
feat: add comprehensive stage descriptions for CreateTopics API implementation
Introduce detailed stage descriptions for the CreateTopics API, covering various scenarios such as invalid topic names, existing topics, and authorization checks. This addition enhances the documentation for implementing the API and provides clear guidelines for error handling and validation processes.
1 parent 861e8f9 commit 318a9b4

File tree

3 files changed

+311
-0
lines changed

3 files changed

+311
-0
lines changed

‎stage_descriptions/createtopics.md

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
# Kafka / CreateTopics API
2+
3+
## Stage 1: Include CreateTopics in APIVersions
4+
5+
In this stage, you'll add an entry for the `CreateTopics` API to the APIVersions response.
6+
7+
### APIVersions
8+
9+
Your Kafka implementation should include the CreateTopics API (key=19) in the ApiVersions response before implementing topic creation functionality. This lets the client know that the broker supports the CreateTopics API.
10+
11+
### Tests
12+
13+
The tester will execute your program like this:
14+
15+
```bash
16+
./your_program.sh
17+
```
18+
19+
It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request.
20+
21+
The tester will validate that:
22+
23+
- The first 4 bytes of your response (the "message length") are valid.
24+
- The correlation ID in the response header matches the correlation ID in the request header.
25+
- The error code in the response body is `0` (No Error).
26+
- The response body contains at least one entry for the API key `19` (CreateTopics).
27+
- The `MaxVersion` for the CreateTopics API is at least 7.
28+
29+
### Notes
30+
31+
- You don't have to implement support for the `CreateTopics` request in this stage. We'll get to this in later stages.
32+
- You'll still need to include the entry for `APIVersions` in your response to pass the previous stage.
33+
34+
## Stage 2: CreateTopics with Invalid Topic Names
35+
36+
In this stage, you'll add support for handling CreateTopics requests with invalid topic names and return proper error responses.
37+
38+
### Handling Invalid Topic Names
39+
40+
Your Kafka implementation should validate topic names and return appropriate error codes when a client tries to create topics with invalid names. Topic names must follow these rules:
41+
- Cannot be empty string
42+
- Cannot be "." or ".."
43+
- Maximum length of 249 characters
44+
- Only ASCII alphanumerics, '.', '_', and '-' are allowed
45+
- Cannot have period/underscore collision patterns
46+
47+
### Tests
48+
49+
The tester will execute your program like this:
50+
51+
```bash
52+
./your_program.sh
53+
```
54+
55+
It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request with an invalid topic name.
56+
57+
The tester will validate that:
58+
59+
- The first 4 bytes of your response (the "message length") are valid.
60+
- The correlation ID in the response header matches the correlation ID in the request header.
61+
- The error code in the topic result is `17` (INVALID_TOPIC_EXCEPTION).
62+
- The `throttle_time_ms` field in the response is `0`.
63+
- The `name` field in the topic result corresponds to the topic name in the request.
64+
- The `error_message` field contains a descriptive error message.
65+
66+
### Notes
67+
68+
- You'll need to parse the `CreateTopics` request in this stage to get the topic names.
69+
- The official docs for the `CreateTopics` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_CreateTopics).
70+
- Topic name validation logic is in the Kafka source code at `clients/src/main/java/org/apache/kafka/common/internals/Topic.java`.
71+
72+
## Stage 3: CreateTopics with Invalid Partition/Replication Parameters
73+
74+
In this stage, you'll add support for handling CreateTopics requests with invalid partition counts and replication factors.
75+
76+
### Handling Invalid Parameters
77+
78+
Your Kafka implementation should validate numeric parameters and return appropriate errors:
79+
- Partition count must be greater than 0 (or -1 for defaults)
80+
- Replication factor must be greater than 0 (or -1 for defaults)
81+
- Both parameters cannot be 0 or negative (except -1)
82+
83+
### Tests
84+
85+
The tester will execute your program like this:
86+
87+
```bash
88+
./your_program.sh
89+
```
90+
91+
It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests with invalid parameters.
92+
93+
The tester will validate that:
94+
95+
- The first 4 bytes of your response (the "message length") are valid.
96+
- The correlation ID in the response header matches the correlation ID in the request header.
97+
- The error code in the topic result is `37` (INVALID_PARTITIONS) for invalid partition count.
98+
- The error code in the topic result is `38` (INVALID_REPLICATION_FACTOR) for invalid replication factor.
99+
- The `throttle_time_ms` field in the response is `0`.
100+
- The `name` field in the topic result corresponds to the topic name in the request.
101+
- The `error_message` field contains a descriptive error message.
102+
103+
### Notes
104+
105+
- Test cases will include: partition count = 0, partition count = -2, replication factor = 0, replication factor = -2.
106+
- Valid values are positive integers or -1 (for broker defaults).
107+
108+
## Stage 4: CreateTopics with Existing Topics
109+
110+
In this stage, you'll add support for detecting when a topic already exists and return the appropriate error response.
111+
112+
### Handling Existing Topics
113+
114+
Your Kafka implementation should check if a topic already exists before attempting to create it. You can check for existing topics by:
115+
- Looking for topic directories in the log directory (e.g., `/tmp/kraft-combined-logs/topic-name-0/`)
116+
- Reading the `__cluster_metadata` topic for TopicRecord entries
117+
118+
### Tests
119+
120+
The tester will execute your program like this:
121+
122+
```bash
123+
./your_program.sh
124+
# Create a topic first
125+
# Then try to create the same topic again
126+
```
127+
128+
It'll then connect to your server on port 9092 and send a `CreateTopics` (v7) request for an existing topic.
129+
130+
The tester will validate that:
131+
132+
- The first 4 bytes of your response (the "message length") are valid.
133+
- The correlation ID in the response header matches the correlation ID in the request header.
134+
- The error code in the topic result is `36` (TOPIC_ALREADY_EXISTS).
135+
- The `throttle_time_ms` field in the response is `0`.
136+
- The `name` field in the topic result corresponds to the topic name in the request.
137+
- The `error_message` field contains a descriptive error message.
138+
139+
### Notes
140+
141+
- This stage requires basic topic existence checking without full topic creation logic.
142+
- You can use filesystem checks or metadata parsing to detect existing topics.
143+
144+
## Stage 5: CreateTopics with Invalid Manual Assignment
145+
146+
In this stage, you'll add support for validating manual partition assignments and return appropriate error responses.
147+
148+
### Handling Invalid Manual Assignment
149+
150+
Your Kafka implementation should validate manual partition assignments when provided:
151+
- Manual assignment requires partition count and replication factor to be -1
152+
- Partition indices must be consecutive starting from 0
153+
- Replica lists cannot be empty
154+
- Broker IDs must be valid (though you can mock this validation)
155+
- No duplicate broker IDs within a partition assignment
156+
157+
### Tests
158+
159+
The tester will execute your program like this:
160+
161+
```bash
162+
./your_program.sh
163+
```
164+
165+
It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests with invalid manual assignments.
166+
167+
The tester will validate that:
168+
169+
- The first 4 bytes of your response (the "message length") are valid.
170+
- The correlation ID in the response header matches the correlation ID in the request header.
171+
- The error code in the topic result is `39` (INVALID_REPLICA_ASSIGNMENT).
172+
- The `throttle_time_ms` field in the response is `0`.
173+
- The `name` field in the topic result corresponds to the topic name in the request.
174+
- The `error_message` field contains a descriptive error message.
175+
176+
### Notes
177+
178+
- Test cases include: non-(-1) partition count with assignment, non-consecutive partitions, empty replica lists, duplicate broker IDs.
179+
- Manual assignment validation is complex but can be implemented without broker state.
180+
181+
## Stage 6: CreateTopics with Authorization Failure
182+
183+
In this stage, you'll add support for authorization checks and return appropriate error responses.
184+
185+
### Handling Authorization
186+
187+
Your Kafka implementation should check authorization before creating topics:
188+
- Require CREATE permission on CLUSTER resource or specific TOPIC resource
189+
- You can implement a simple authorization check based on configuration or mock it
190+
- Return authorization failure for unauthorized requests
191+
192+
### Tests
193+
194+
The tester will execute your program like this:
195+
196+
```bash
197+
./your_program.sh
198+
```
199+
200+
It'll then connect to your server on port 9092 and send `CreateTopics` (v7) requests without proper authorization.
201+
202+
The tester will validate that:
203+
204+
- The first 4 bytes of your response (the "message length") are valid.
205+
- The correlation ID in the response header matches the correlation ID in the request header.
206+
- The error code in the topic result is `29` (TOPIC_AUTHORIZATION_FAILED).
207+
- The `throttle_time_ms` field in the response is `0`.
208+
- The `name` field in the topic result corresponds to the topic name in the request.
209+
- The `error_message` field contains "Authorization failed."
210+
211+
### Notes
212+
213+
- This stage focuses on authorization error handling rather than implementing full ACL logic.
214+
- You can use a simple configuration-based approach to determine authorization.
215+
216+
---
217+
218+
## Setup Commands
219+
220+
```bash
221+
# Clean environment
222+
rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs
223+
224+
# Generate cluster ID and format storage
225+
KAFKA_CLUSTER_ID="$(/usr/local/kafka-latest/bin/kafka-storage.sh random-uuid)"
226+
/usr/local/kafka-latest/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /usr/local/kafka-latest/config/kraft/server.properties
227+
228+
# Start Kafka server
229+
/usr/local/kafka-latest/bin/kafka-server-start.sh /usr/local/kafka-latest/config/kraft/server.properties
230+
231+
# Test with Kafka CLI
232+
/usr/local/kafka-latest/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 3
233+
```
234+
235+
## Testing Commands
236+
237+
```bash
238+
# Build tester
239+
make build
240+
241+
# Run individual test
242+
go run main.go
243+
244+
# Verify with Kafka CLI
245+
/usr/local/kafka-latest/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
246+
```
247+
248+
## Error Code Reference
249+
250+
| Code | Name | Stages |
251+
|------|------|---------|
252+
| 0 | NO_ERROR | Success cases |
253+
| 17 | INVALID_TOPIC_EXCEPTION | 2 |
254+
| 29 | TOPIC_AUTHORIZATION_FAILED | 6 |
255+
| 36 | TOPIC_ALREADY_EXISTS | 4 |
256+
| 37 | INVALID_PARTITIONS | 3 |
257+
| 38 | INVALID_REPLICATION_FACTOR | 3 |
258+
| 39 | INVALID_REPLICA_ASSIGNMENT | 5 |

‎stage_descriptions/final-stages.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
Stage CT1: Include CreateTopics in APIVersions
2+
- Add API key 19 (CreateTopics) to APIVersions response
3+
- Foundation stage enabling API discovery
4+
5+
Stage CT2: CreateTopics with Invalid Topic Name (Hard code error)
6+
- Handle invalid characters, reserved names, empty names
7+
- Error code: INVALID_TOPIC_EXCEPTION (17)
8+
9+
Stage CT3: CreateTopics with Reserved Topic Name (Check __cluster_metadata)
10+
- Handle attempts to create system topics (__cluster_metadata)
11+
- Error code: INVALID_REQUEST (42)
12+
13+
Stage CT4: CreateTopics with Existing Topic Name (Read __cluster_metadata)
14+
- Handle duplicate topic creation attempts
15+
- Error code: TOPIC_ALREADY_EXISTS (36)
16+
17+
Stage CT5: CreateTopics with Valid Parameters (Success Case)
18+
- Successfully create single topic with valid parameters
19+
- Core success functionality
20+
21+
Stage CT6: Multiple topics in single request
22+
- Handle multiple topics in a single request
23+
24+
Stage CT7: CreateTopics with Validation Only
25+
- Handle validation only mode

‎stage_descriptions/stages.yaml

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
- Include CreateTopics in APIVersions
2+
- CreateTopics with Invalid Topic Name (empty, ., .., length > 249, invalid characters)
3+
- CreateTopics with Invalid Partition Values (negative, zero)
4+
- CreateTopics with Invalid Replication Values (negative, zero, > broker count)
5+
- CreateTopics with Reserved Topic Name (__cluster_metadata)
6+
- CreateTopics with Existing Topic (read __cluster_metadata)
7+
- CreateTopics with Authorization Check (no CREATE permission on CLUSTER or TOPIC resource)
8+
- CreateTopics with Single Topic (single topic in single request)
9+
- CreateTopics with Multiple Topics (multiple topics in single request)
10+
- CreateTopics with Manual Assignments (manual assignment of partitions and replicas)
11+
- CreateTopics with Validation Only
12+
- CreateTopics with Invalid Request (Duplicate topic names in single request)
13+
14+
15+
1. Topic Name Validation - INVALID_TOPIC_EXCEPTION (17)
16+
- Empty topic name or "." or ".."
17+
- Length > 249 characters
18+
- Invalid characters (only ASCII alphanumerics, '.', '_', '-')
19+
- Internal topic collision detection
20+
21+
22+
23+
Stage CT8: CreateTopics with Invalid Parameters
24+
- Handle negative/zero partitions, invalid replication factors
25+
- Duplicate topic names in single request
26+
- Error codes: INVALID_PARTITIONS (37), INVALID_REPLICATION_FACTOR (38)
27+
28+
- CT8: Topic-specific configurations

0 commit comments

Comments
 (0)