Skip to content

Commit bc59ef4

Browse files
emasabemasab
committed
PR confluentinc#4970: Code and tests fixes to make the full test suite pass
1 parent 34b944a commit bc59ef4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+1440
-324
lines changed

‎.semaphore/run-all-tests.yml

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
version: v1.0
2+
name: run-all-tests
3+
4+
agent:
5+
machine:
6+
type: s1-prod-ubuntu22-04-amd64-1
7+
8+
execution_time_limit:
9+
hours: 3
10+
11+
global_job_config:
12+
prologue:
13+
commands:
14+
- checkout
15+
- '[[ -z "$GIT_REF" ]] || git checkout $GIT_REF'
16+
- wget -O rapidjson-dev.deb https://launchpad.net/ubuntu/+archive/primary/+files/rapidjson-dev_1.1.0+dfsg2-3_all.deb
17+
- sudo dpkg -i rapidjson-dev.deb
18+
- sudo apt update
19+
- sudo apt remove -y needrestart
20+
- sudo apt install -y valgrind
21+
- python3 -m pip install -U pip
22+
- python3 -m pip -V
23+
- (cd tests && python3 -m pip install -r requirements.txt)
24+
- ./configure --install-deps
25+
- make -j all
26+
- make -j -C tests build
27+
28+
blocks:
29+
- name: "Run all tests (x86_64)"
30+
dependencies: []
31+
task:
32+
agent:
33+
machine:
34+
type: s1-prod-ubuntu22-04-amd64-2
35+
prologue:
36+
commands:
37+
- if [[ "$TEST_ARCHES" != *"x86_64"* ]]; then exit 0; fi
38+
jobs:
39+
- name: "PLAINTEXT cluster (x86_64)"
40+
env_vars:
41+
- name: TEST_SSL
42+
value: "False"
43+
commands:
44+
- if [[ "$TEST_TYPE" != *"plaintext"* ]]; then exit 0; fi
45+
- ./tests/run-all-tests.sh
46+
- name: "SSL cluster (x86_64)"
47+
env_vars:
48+
- name: TEST_SSL
49+
value: "True"
50+
commands:
51+
- if [[ "$TEST_TYPE" != *"ssl"* ]]; then exit 0; fi
52+
- ./tests/run-all-tests.sh
53+
- name: "Run all tests (aarch64)"
54+
dependencies: []
55+
task:
56+
agent:
57+
machine:
58+
type: s1-prod-ubuntu22-04-arm64-2
59+
prologue:
60+
commands:
61+
- if [[ "$TEST_ARCHES" != *"aarch64"* ]]; then exit 0; fi
62+
jobs:
63+
- name: "PLAINTEXT cluster (aarch64)"
64+
env_vars:
65+
- name: TEST_SSL
66+
value: "False"
67+
commands:
68+
- if [[ "$TEST_TYPE" != *"plaintext"* ]]; then exit 0; fi
69+
- ./tests/run-all-tests.sh
70+
- name: "SSL cluster (aarch64)"
71+
env_vars:
72+
- name: TEST_SSL
73+
value: "True"
74+
commands:
75+
- if [[ "$TEST_TYPE" != *"ssl"* ]]; then exit 0; fi
76+
- ./tests/run-all-tests.sh

‎.semaphore/semaphore.yml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -384,3 +384,25 @@ blocks:
384384
# Upload all packages to project artifact store
385385
- artifact push project packages --destination librdkafka-packages-${SEMAPHORE_GIT_TAG_NAME}-${SEMAPHORE_WORKFLOW_ID}
386386
- echo Thank you
387+
388+
promotions:
389+
- name: Run all tests on master commits
390+
pipeline_file: run-all-tests.yml
391+
parameters:
392+
env_vars:
393+
- required: true
394+
name: TEST_KAFKA_GIT_REF
395+
default_value: 3.8.0
396+
- required: true
397+
name: TEST_TYPE
398+
default_value: plaintext,ssl
399+
- required: true
400+
name: TEST_ARCHES
401+
default_value: x86_64,aarch64
402+
- required: true
403+
name: TEST_PARALLEL
404+
default_value: "1"
405+
auto_promote_on:
406+
- result: passed
407+
branch:
408+
- "master"

‎CHANGELOG.md

Lines changed: 83 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,33 @@
1-
# librdkafka v2.8.1
1+
# librdkafka v2.9.0
22

3-
librdkafka v2.8.1 is a maintenance release:
3+
librdkafka v2.9.0 is a feature release:
44

55
* Identify brokers only by broker id (#4557, @mfleming)
66
* Remove unavailable brokers and their thread (#4557, @mfleming)
7+
* Fix for librdkafka yielding before timeouts had been reached (#)
8+
* Removed a 500ms latency when a consumer partition switches to a different
9+
leader (#)
10+
* The mock cluster implementation removes brokers from Metadata response
11+
when they're not available, this simulates better the actual behavior of
12+
a cluster that is using KRaft (#).
13+
* Doesn't remove topics from cache on temporary Metadata errors but only
14+
on metadata cache expiry (#).
15+
* Doesn't mark the topic as unknown if it had been marked as existent earlier
16+
and `topic.metadata.propagation.max.ms` hasn't passed still (#).
17+
* Doesn't update partition leaders if the topic in metadata
18+
response has errors (#).
19+
* Only topic authorization errors in a metadata response are considered
20+
permanent and are returned to the user (#).
21+
* The function `rd_kafka_offsets_for_times` refreshes leader information
22+
if the error requires it, allowing it to succeed on
23+
subsequent manual retries (#).
24+
* Deprecated `api.version.request`, `api.version.fallback.ms` and
25+
`broker.version.fallback` configuration properties (#).
26+
* When consumer is closed before destroying the client, the operations queue
27+
isn't purged anymore as it contains operations
28+
unrelated to the consumer group (#).
29+
* When making multiple changes to the consumer subscription in a short time,
30+
no unknown topic error is returned for topics that are in the new subscription but weren't in previous one (#).
731

832

933
## Fixes
@@ -22,6 +46,63 @@ librdkafka v2.8.1 is a maintenance release:
2246
temporarily or permanently so we always remove it and it'll be added back when
2347
it becomes available again.
2448
Happens since 1.x (#4557, @mfleming).
49+
* Issues: #
50+
librdkafka code using `cnd_timedwait` was yielding before a timeout occurred
51+
without the condition being fulfilled because of spurious wake-ups.
52+
Solved by verifying with a monotonic clock that the expected point in time
53+
was reached and calling the function again if needed.
54+
Happens since 1.x (#).
55+
* Issues: #
56+
Doesn't remove topics from cache on temporary Metadata errors but only
57+
on metadata cache expiry. It allows the client to continue working
58+
in case of temporary problems to the Kafka metadata plane.
59+
Happens since 1.x (#).
60+
* Issues: #
61+
Doesn't mark the topic as unknown if it had been marked as existent earlier
62+
and `topic.metadata.propagation.max.ms` hasn't passed still. It achieves
63+
this property expected effect even if a different broker had
64+
previously reported the topic as existent.
65+
Happens since 1.x (#).
66+
* Issues: #
67+
Doesn't update partition leaders if the topic in metadata
68+
response has errors. It's in line with what Java client does and allows
69+
to avoid segmentation faults for unknown partitions.
70+
Happens since 1.x (#).
71+
* Issues: #
72+
Only topic authorization errors in a metadata response are considered
73+
permanent and are returned to the user. It's in line with what Java client
74+
does and avoids returning to the user an error that wasn't meant to be
75+
permanent.
76+
Happens since 1.x (#).
77+
78+
### Consumer fixes
79+
80+
* Issues: #
81+
When switching to a different leader a consumer could wait 500ms
82+
(`fetch.error.backoff.ms`) before starting to fetch again. The fetch backoff wasn't reset when joining the new broker.
83+
Solved by resetting it, given it's not needed to backoff
84+
the first fetch on a different node. This way faster leader switches are
85+
possible.
86+
Happens since 1.x (#).
87+
* Issues: #
88+
The function `rd_kafka_offsets_for_times` refreshes leader information
89+
if the error requires it, allowing it to succeed on
90+
subsequent manual retries. Similar to the fix done in 2.3.0 in
91+
`rd_kafka_query_watermark_offsets`. Additionally, the partition
92+
current leader epoch is taken from metadata cache instead of
93+
from passed partitions.
94+
Happens since 1.x (#).
95+
* Issues: #
96+
When consumer is closed before destroying the client, the operations queue
97+
isn't purged anymore as it contains operations
98+
unrelated to the consumer group.
99+
Happens since 1.x (#).
100+
* Issues: #
101+
When making multiple changes to the consumer subscription in a short time,
102+
no unknown topic error is returned for topics that are in the new subscription
103+
but weren't in previous one. This was due to the metadata request relative
104+
to previous subscription.
105+
Happens since 1.x (#).
25106

26107

27108

‎CONFIGURATION.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,10 +54,10 @@ resolve_cb | * | |
5454
opaque | * | | | low | Application opaque (set with rd_kafka_conf_set_opaque()) <br>*Type: see dedicated API*
5555
default_topic_conf | * | | | low | Default topic configuration for automatically subscribed topics <br>*Type: see dedicated API*
5656
internal.termination.signal | * | 0 .. 128 | 0 | low | Signal that librdkafka will use to quickly terminate on rd_kafka_destroy(). If this signal is not set then there will be a delay before rd_kafka_wait_destroyed() returns true as internal threads are timing out their system calls. If this signal is set however the delay will be minimal. The application should mask this signal as an internal signal handler is installed. <br>*Type: integer*
57-
api.version.request | * | true, false | true | high | Request broker's supported API versions to adjust functionality to available protocol features. If set to false, or the ApiVersionRequest fails, the fallback version `broker.version.fallback` will be used. **NOTE**: Depends on broker version >=0.10.0. If the request is not supported by (an older) broker the `broker.version.fallback` fallback is used. <br>*Type: boolean*
57+
api.version.request | * | true, false | true | high | **DEPRECATED** **Post-deprecation actions: remove this configuration property, brokers < 0.10.0 won't be supported anymore in librdkafka 3.x.** Request broker's supported API versions to adjust functionality to available protocol features. If set to false, or the ApiVersionRequest fails, the fallback version `broker.version.fallback` will be used. **NOTE**: Depends on broker version >=0.10.0. If the request is not supported by (an older) broker the `broker.version.fallback` fallback is used. <br>*Type: boolean*
5858
api.version.request.timeout.ms | * | 1 .. 300000 | 10000 | low | Timeout for broker API version requests. <br>*Type: integer*
59-
api.version.fallback.ms | * | 0 .. 604800000 | 0 | medium | Dictates how long the `broker.version.fallback` fallback is used in the case the ApiVersionRequest fails. **NOTE**: The ApiVersionRequest is only issued when a new connection to the broker is made (such as after an upgrade). <br>*Type: integer*
60-
broker.version.fallback | * | | 0.10.0 | medium | Older broker versions (before 0.10.0) provide no way for a client to query for supported protocol features (ApiVersionRequest, see `api.version.request`) making it impossible for the client to know what features it may use. As a workaround a user may set this property to the expected broker version and the client will automatically adjust its feature set accordingly if the ApiVersionRequest fails (or is disabled). The fallback broker version will be used for `api.version.fallback.ms`. Valid values are: 0.9.0, 0.8.2, 0.8.1, 0.8.0. Any other value >= 0.10, such as 0.10.2.1, enables ApiVersionRequests. <br>*Type: string*
59+
api.version.fallback.ms | * | 0 .. 604800000 | 0 | medium | **DEPRECATED** **Post-deprecation actions: remove this configuration property, brokers < 0.10.0 won't be supported anymore in librdkafka 3.x.** Dictates how long the `broker.version.fallback` fallback is used in the case the ApiVersionRequest fails. **NOTE**: The ApiVersionRequest is only issued when a new connection to the broker is made (such as after an upgrade). <br>*Type: integer*
60+
broker.version.fallback | * | | 0.10.0 | medium | **DEPRECATED** **Post-deprecation actions: remove this configuration property, brokers < 0.10.0 won't be supported anymore in librdkafka 3.x.** Older broker versions (before 0.10.0) provide no way for a client to query for supported protocol features (ApiVersionRequest, see `api.version.request`) making it impossible for the client to know what features it may use. As a workaround a user may set this property to the expected broker version and the client will automatically adjust its feature set accordingly if the ApiVersionRequest fails (or is disabled). The fallback broker version will be used for `api.version.fallback.ms`. Valid values are: 0.9.0, 0.8.2, 0.8.1, 0.8.0. Any other value >= 0.10, such as 0.10.2.1, enables ApiVersionRequests. <br>*Type: string*
6161
allow.auto.create.topics | * | true, false | false | low | Allow automatic topic creation on the broker when subscribing to or assigning non-existent topics. The broker must also be configured with `auto.create.topics.enable=true` for this configuration to take effect. Note: the default value (true) for the producer is different from the default value (false) for the consumer. Further, the consumer default value is different from the Java consumer (true), and this property is not supported by the Java producer. Requires broker version >= 0.11.0.0, for older broker versions only the broker configuration applies. <br>*Type: boolean*
6262
security.protocol | * | plaintext, ssl, sasl_plaintext, sasl_ssl | plaintext | high | Protocol used to communicate with brokers. <br>*Type: enum value*
6363
ssl.cipher.suites | * | | | low | A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. See manual page for `ciphers(1)` and `SSL_CTX_set_cipher_list(3). <br>*Type: string*

‎service.yml

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,180 @@ semaphore:
1616
- /semaphore.*/
1717
- /dev_.*/
1818
- /feature\/.*/
19+
tasks:
20+
- name: verify-linux-packages
21+
branch: master
22+
scheduled: false
23+
pipeline_file: .semaphore/verify-linux-packages.yml
24+
parameters:
25+
- name: LIBRDKAFKA_VERSION
26+
required: true
27+
description: librdkafka version to verify
28+
- name: run-all-tests
29+
branch: master
30+
scheduled: false
31+
pipeline_file: .semaphore/run-all-tests.yml
32+
parameters:
33+
- name: GIT_REF
34+
required: false
35+
description: |-
36+
librdkafka git ref to test.
37+
38+
- name: TEST_KAFKA_GIT_REF
39+
required: true
40+
default_value: ''
41+
description: |-
42+
Kafka version to test against. If it's a version, the corresponding
43+
package is downloaded. [<username>/]<git_ref>[@<commit_hash>] is also supported
44+
to clone from GitHub and build from source.
45+
46+
- name: TEST_CP_VERSION
47+
required: false
48+
description: |-
49+
CP version to test against
50+
51+
- name: TEST_KAFKA_VERSION
52+
required: false
53+
description: |-
54+
Kafka version to assume in this branch. If empty (default)
55+
uses TEST_KAFKA_GIT_REF if it's a version, otherwise 9.9.9.
56+
57+
- name: TEST_KRAFT
58+
required: true
59+
default_value: 'True'
60+
description: |-
61+
Run a KRaft cluster.
62+
options:
63+
- 'True'
64+
- 'False'
65+
66+
- name: TEST_TYPE
67+
required: true
68+
default_value: 'plaintext,ssl'
69+
description: |-
70+
Run PLAINTEXT or SSL tests or both. Defaults to both.
71+
options:
72+
- 'plaintext,ssl'
73+
- 'plaintext'
74+
- 'ssl'
75+
76+
- name: TEST_SSL_INTERMEDIATE_CA
77+
required: false
78+
description: |-
79+
Run SSL tests with an intermediate CA certificate.
80+
options:
81+
- 'True'
82+
- 'False'
83+
84+
- name: TEST_SASL
85+
required: false
86+
description: |-
87+
SASL mechanism to use.
88+
options:
89+
- ''
90+
- 'PLAIN'
91+
- 'SCRAM-SHA-256'
92+
- 'SCRAM-SHA-512'
93+
- 'GSSAPI'
94+
- 'OAUTHBEARER'
95+
96+
- name: TEST_QUICK
97+
required: true
98+
default_value: 'False'
99+
description: |-
100+
Run only quick tests.
101+
options:
102+
- 'True'
103+
- 'False'
104+
105+
- name: TEST_LOCAL
106+
required: true
107+
default_value: 'All'
108+
description: |-
109+
Run only local, only non-local or all tests.
110+
options:
111+
- 'All'
112+
- 'Local'
113+
- 'Non-local'
114+
115+
- name: TEST_PARALLEL
116+
required: false
117+
description: |-
118+
Number of parallel tests to run. Empty means default value.
119+
120+
- name: TEST_MODE
121+
required: false
122+
description: |-
123+
Test mode to use. Empty means default value.
124+
125+
- name: TESTS
126+
required: false
127+
description: |-
128+
Run specified tests only. Empty means run all tests.
129+
Can be a comma-separated list of test numbers or ranges,
130+
even overlapping. Example:
131+
1,3,9-11,10-15,17-,-10
132+
133+
- name: TESTS_BATCH_SIZE
134+
required: false
135+
description: |-
136+
Run tests in batches of this size.
137+
Empty means no batches, run all tests at once.
138+
139+
- name: TEST_ITERATIONS
140+
required: false
141+
description: |-
142+
Run specified tests this many times.
143+
Empty means run tests once.
144+
145+
- name: TEST_ASSERT
146+
required: true
147+
default_value: 'False'
148+
description: |-
149+
Assert on failures
150+
options:
151+
- 'True'
152+
- 'False'
153+
154+
- name: TEST_DEBUG
155+
required: false
156+
description: |-
157+
Enable debug output for tests.
158+
159+
- name: TEST_LEVEL
160+
required: false
161+
description: |-
162+
Test verbosity level. Default is 2. Increase to 3 or 4 for more output.
163+
164+
- name: TEST_CONF
165+
required: false
166+
description: |-
167+
Additional broker configuration as a JSON array. Example:
168+
["group.coordinator.rebalance.protocols=classic,consumer"]
169+
170+
- name: TEST_TRIVUP_PARAMETERS
171+
required: false
172+
description: |-
173+
Additional trivup parameters.
174+
175+
- name: TEST_RUNNER_PARAMETERS
176+
required: false
177+
description: |-
178+
Addition test runner parameters.
179+
180+
- name: TEST_ENV_VARIABLES
181+
required: false
182+
description: |-
183+
Additional environment variables to set for tests.
184+
Example: A=1,B=2,C=3
185+
186+
- name: TEST_ARCHES
187+
required: true
188+
default_value: 'x86_64,aarch64'
189+
description: |-
190+
Test on these architectures: x86_64, aarch64 or both
191+
options:
192+
- 'x86_64,aarch64'
193+
- 'x86_64'
194+
- 'aarch64'
195+

0 commit comments

Comments
 (0)