Skip to content

Commit f860ddd

Browse files
LaPetiteSourisLaPetiteSouris
committed
Merge remote-tracking branch 'upstream/master' into stream-configuration
2 parents db16bae + c21c8e2 commit f860ddd

File tree

11 files changed

+348
-258
lines changed

11 files changed

+348
-258
lines changed

‎Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
FROM golang:1.13-alpine as build-base
22
RUN apk update && apk upgrade && \
3-
apk add --no-cache bash git openssh make bzr
3+
apk add --no-cache bash git openssh make
44
ADD . /go/src/github.com/liftbridge-io/liftbridge
55
WORKDIR /go/src/github.com/liftbridge-io/liftbridge
66
ENV GO111MODULE on

‎docker/dev-standalone/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
FROM golang:1.13-alpine as build-base
22
RUN apk update && apk upgrade && \
3-
apk add --no-cache bash git openssh make bzr
3+
apk add --no-cache bash git openssh make
44

55
WORKDIR /workspace
66

‎documentation/configuration.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ USAGE:
2424
liftbridge [global options] command [command options] [arguments...]
2525

2626
VERSION:
27-
v1.0.0
27+
v1.1.0
2828

2929
COMMANDS:
3030
help, h Shows a list of commands or help for one command
@@ -73,6 +73,7 @@ An example configuration file is shown below.
7373
```yaml
7474
---
7575
listen: localhost:9293
76+
host: localhost
7677
data.dir: /tmp/liftbridge/server-2
7778
activity.stream.enabled: true
7879

@@ -100,6 +101,21 @@ clustering:
100101
replica.max.lag.time: 20s
101102
```
102103
104+
## Overriding configuration settings with environment variables
105+
106+
For configuration set in the configuration file the value can be overridden
107+
with environment variables prefixed with `LIFTBRIDGE_`. The key must exist in
108+
the config file to be overridden.
109+
110+
For example using the config file from above one could override the host and
111+
logging level with:
112+
113+
```sh
114+
env LIFTBRIDGE_HOST=liftbridge.example.com \
115+
LIFTBRIDGE_LOGGING_LEVEL=error \
116+
liftbridge --config config.yaml
117+
```
118+
103119
## Configuration Settings
104120
105121
Below is the list of Liftbridge configuration settings, including the name of

‎documentation/quick_start.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ only be set on one server when bootstrapping a cluster.**
4343

4444
```shell
4545
$ liftbridge --raft-bootstrap-seed
46-
INFO[2020-04-27 17:20:21] Liftbridge Version: v1.0.0
46+
INFO[2020-04-27 17:20:21] Liftbridge Version: v1.1.0
4747
INFO[2020-04-27 17:20:21] Server ID: agTv5PkzvgKygm688EMd4C
4848
INFO[2020-04-27 17:20:21] Namespace: liftbridge-default
4949
INFO[2020-04-27 17:20:21] Retention Policy: [Age: 1 week, Compact: false]
@@ -56,7 +56,7 @@ We set the `--data-dir` and `--port` flags to avoid clobbering the first server.
5656

5757
```shell
5858
$ liftbridge --data-dir /tmp/liftbridge/server-2 --port=9293
59-
INFO[2020-04-27 17:21:10] Liftbridge Version: v1.0.0
59+
INFO[2020-04-27 17:21:10] Liftbridge Version: v1.1.0
6060
INFO[2020-04-27 17:21:10] Server ID: OtKIIhwxBztcSkgZD0xXQP
6161
INFO[2020-04-27 17:21:10] Namespace: liftbridge-default
6262
INFO[2020-04-27 17:21:10] Retention Policy: [Age: 1 week, Compact: false]

‎k8s/Dockerfile.k8s

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
FROM golang:1.13-alpine as build-base
22
RUN apk update && apk upgrade && \
3-
apk add --no-cache bash git openssh make bzr wget
3+
apk add --no-cache bash git openssh make wget
44
WORKDIR /workspace
55
ENV GO111MODULE on
66

‎server/version.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
package server
22

33
// Version of the Liftbridge server.
4-
const Version = "v1.0.0"
4+
const Version = "v1.1.0"

‎website/i18n/en.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,12 @@
178178
},
179179
"version-v1.0.0/version-v1.0.0-quick-start": {
180180
"title": "Quick Start"
181+
},
182+
"version-v1.1.0/version-v1.1.0-configuration": {
183+
"title": "Configuration"
184+
},
185+
"version-v1.1.0/version-v1.1.0-quick-start": {
186+
"title": "Quick Start"
181187
}
182188
},
183189
"links": {

‎website/versioned_docs/version-v1.1.0/configuration.md

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
---
2+
id: version-v1.1.0-configuration
3+
title: Configuration
4+
original_id: configuration
5+
---
6+
7+
Liftbridge provides limited configuration through command-line flags and full
8+
configuration using a configuration file. Flags will always take precedent
9+
over settings in the configuration file.
10+
11+
The configuration file is passed in using the `--config` flag:
12+
13+
```shell
14+
$ liftbridge --config liftbridge.yaml
15+
```
16+
17+
To get a full list of the CLI flags, use the `--help` flag:
18+
19+
```shell
20+
$ liftbridge --help
21+
NAME:
22+
liftbridge - Lightweight, fault-tolerant message streams
23+
24+
USAGE:
25+
liftbridge [global options] command [command options] [arguments...]
26+
27+
VERSION:
28+
v1.1.0
29+
30+
COMMANDS:
31+
help, h Shows a list of commands or help for one command
32+
33+
GLOBAL OPTIONS:
34+
--config FILE, -c FILE load configuration from FILE
35+
--server-id value, --id value ID of the server in the cluster if there is no stored ID (default: random ID)
36+
--namespace value, --ns value cluster namespace (default: "liftbridge-default")
37+
--nats-servers ADDR[,ADDR], -n ADDR[,ADDR] connect to NATS cluster at ADDR[,ADDR] (default: "nats://127.0.0.1:4222")
38+
--data-dir DIR, -d DIR store data in DIR (default: "/tmp/liftbridge/<namespace>")
39+
--port value, -p value port to bind to (default: 9292)
40+
--tls-cert value server certificate file
41+
--tls-key value private key for server certificate
42+
--level value, -l value logging level [debug|info|warn|error] (default: "info")
43+
--raft-bootstrap-seed bootstrap the Raft cluster by electing self as leader if there is no existing state
44+
--raft-bootstrap-peers value bootstrap the Raft cluster with the provided list of peer IDs if there is no existing state
45+
--help, -h show help
46+
--version, -v print the version
47+
```
48+
49+
## Configuration File Format
50+
51+
The configuration file uses a YAML format. The configuration settings are
52+
described below. Settings follow a hierarchical pattern delimited by periods.
53+
The full, flattened name or nested names can be used. For example:
54+
55+
```yaml
56+
logging.level: debug
57+
logging.recovery: true
58+
logging.raft: true
59+
```
60+
61+
The above configuration is equivalent to:
62+
63+
```yaml
64+
logging:
65+
level: debug
66+
recovery: true
67+
raft: true
68+
```
69+
70+
## Example Configuration File
71+
72+
An example configuration file is shown below.
73+
74+
```yaml
75+
---
76+
listen: localhost:9293
77+
host: localhost
78+
data.dir: /tmp/liftbridge/server-2
79+
activity.stream.enabled: true
80+
81+
# Configure logging.
82+
logging:
83+
level: debug
84+
raft: true
85+
86+
# Define NATS cluster to connect to.
87+
nats.servers:
88+
- nats://localhost:4300
89+
- nats://localhost:4301
90+
91+
# Specify message stream settings.
92+
streams:
93+
retention.max:
94+
age: 24h
95+
messages: 100
96+
compact.enabled: true
97+
98+
# Specify cluster settings.
99+
clustering:
100+
server.id: server-2
101+
raft.bootstrap.seed: true
102+
replica.max.lag.time: 20s
103+
```
104+
105+
## Overriding configuration settings with environment variables
106+
107+
For configuration set in the configuration file the value can be overridden
108+
with environment variables prefixed with `LIFTBRIDGE_`. The key must exist in
109+
the config file to be overridden.
110+
111+
For example using the config file from above one could override the host and
112+
logging level with:
113+
114+
```sh
115+
env LIFTBRIDGE_HOST=liftbridge.example.com \
116+
LIFTBRIDGE_LOGGING_LEVEL=error \
117+
liftbridge --config config.yaml
118+
```
119+
120+
## Configuration Settings
121+
122+
Below is the list of Liftbridge configuration settings, including the name of
123+
the setting in the configuration file and the CLI flag if it exists.
124+
125+
| Name | Flag | Description | Type | Default | Valid Values |
126+
|:----|:----|:----|:----|:----|:----|
127+
| listen | | The server listen host/port. This is the host and port the server will bind to. If this is not specified but `host` and `port` are specified, these values will be used. If neither `listen` nor `host`/`port` are specified, the default listen address will be used. | string | 0:0:0:0:9292 | |
128+
| host | | The server host that is advertised to clients, i.e. the address clients will attempt to connect to based on metadata API responses. If not set, `listen` will be returned to clients. This value may differ from `listen` in situations where the external address differs from the internal address, e.g. when running in a container. If `listen` is not specified, the server will also bind to this host. | string | localhost | |
129+
| port | port | The server port that is advertised to clients. See `host` for more information on how this behaves. | int | 9292 | |
130+
| tls.key | tls-key | The private key file for server certificate. This must be set in combination with `tls.cert` to enable TLS. | string | |
131+
| tls.cert | tls-cert | The server certificate file. This must be set in combination with `tls.key` to enable TLS. | string | |
132+
| tls.client.auth.enabled | tls-client-auth | Enforce client-side authentication via certificate. | bool | false |
133+
| tls.client.auth.ca | tls-client-auth-ca | The CA certificate file to use when authenticating clients. | string | |
134+
| logging.level | level | The logging level. | string | info | [debug, info, warn, error] |
135+
| logging.recovery | | Log messages resulting from the replay of the Raft log on server recovery. | bool | false | |
136+
| logging.raft | | Enables logging in the Raft subsystem. | bool | false | |
137+
| data.dir | data-dir | The directory to store data in. | string | /tmp/liftbridge/namespace | |
138+
| batch.max.messages | | The maximum number of messages to batch when writing to disk. | int | 1024 |
139+
| batch.max.time | | The maximum time to wait to batch more messages when writing to disk. | duration | 0 | |
140+
| metadata.cache.max.age | | The maximum age of cached broker metadata. | duration | 2m | |
141+
| nats | | NATS configuration. | map | | [See below](#nats-configuration-settings) |
142+
| streams | | Write-ahead log configuration for message streams. | map | | [See below](#streams-configuration-settings) |
143+
| clustering | | Broker cluster configuration. | map | | [See below](#clustering-configuration-settings) |
144+
| activity | | Meta activity event stream configuration. | map | | [See below](#activity-configuration-settings) |
145+
146+
### NATS Configuration Settings
147+
148+
Below is the list of the configuration settings for the `nats` part of
149+
the configuration file.
150+
151+
| Name | Flag | Description | Type | Default | Valid Values |
152+
|:----|:----|:----|:----|:----|:----|
153+
| servers | nats-servers | List of NATS hosts to connect to. | list | nats://localhost:4222 | |
154+
| user | | Username to use to connect to NATS servers. | string | | |
155+
| password | | Password to use to connect to NATS servers. | string | | |
156+
157+
### Streams Configuration Settings
158+
159+
Below is the list of the configuration settings for the `streams` part of the
160+
configuration file.
161+
162+
| Name | Flag | Description | Type | Default | Valid Values |
163+
|:----|:----|:----|:----|:----|:----|
164+
| retention.max.bytes | | The maximum size a stream's log can grow to, in bytes, before we will discard old log segments to free up space. A value of 0 indicates no limit. | int64 | 0 | |
165+
| retention.max.messages | | The maximum size a stream's log can grow to, in number of messages, before we will discard old log segments to free up space. A value of 0 indicates no limit. | int64 | 0 | |
166+
| retention.max.age | | The TTL for stream log segment files, after which they are deleted. A value of 0 indicates no TTL. | duration | 168h | |
167+
| cleaner.interval | | The frequency to check if a new stream log segment file should be rolled and whether any segments are eligible for deletion based on the retention policy or compaction if enabled. | duration | 5m | |
168+
| segment.max.bytes | | The maximum size of a single stream log segment file in bytes. Retention is always done a file at a time, so a larger segment size means fewer files but less granular control over retention. | int64 | 268435456 | |
169+
| segment.max.age | | The maximum time before a new stream log segment is rolled out. A value of 0 means new segments will only be rolled when `segment.max.bytes` is reached. Retention is always done a file at a time, so a larger value means fewer files but less granular control over retention. | duration | value of `retention.max.age` | |
170+
| compact.enabled | | Enables stream log compaction. Compaction works by retaining only the latest message for each key and discarding older messages. The frequency in which compaction runs is controlled by `cleaner.interval`. | bool | false | |
171+
| compact.max.goroutines | | The maximum number of concurrent goroutines to use for compaction on a stream log (only applicable if `compact.enabled` is `true`). | int | 10 | |
172+
173+
### Clustering Configuration Settings
174+
175+
Below is the list of the configuration settings for the `clustering` part of
176+
the configuration file.
177+
178+
| Name | Flag | Description | Type | Default | Valid Values |
179+
|:----|:----|:----|:----|:----|:----|
180+
| server.id | server-id | ID of the server in the cluster. | string | random id | string with no spaces or periods |
181+
| namespace | namespace | Cluster namespace. | string | liftbridge-default | string with no spaces or periods |
182+
| raft.snapshot.retain | | The number Raft log snapshots to retain on disk. | int | 2 | |
183+
| raft.snapshot.threshold | | Controls how many outstanding logs there must be before taking a snapshot. This prevents excessive snapshots when a small set of logs can be replayed. | int | 8192 | |
184+
| raft.cache.size | | The number of Raft logs to hold in memory for quick lookup. | int | 512 | |
185+
| raft.bootstrap.seed | raft-bootstrap-seed | Bootstrap the Raft cluster by electing self as leader if there is no existing state. If this is enabled, `raft.bootstrap.peers` should generally not be used, either on this node or peer nodes, since cluster topology is not being explicitly defined. Instead, peers should be started without bootstrap flags which will cause them to automatically discover the bootstrapped leader and join the cluster. | bool | false | |
186+
| raft.bootstrap.peers | raft-bootstrap-peers | Bootstrap the Raft cluster with the provided list of peer IDs if there is no existing state. This should generally not be used in combination with `raft.bootstrap.seed` since it is explicitly defining cluster topology and the configured topology will elect a leader. Note that once the cluster is established, new nodes can join without setting bootstrap flags since they will automatically discover the elected leader and join the cluster. | list | | |
187+
| replica.max.lag.time | | If a follower hasn't sent any replication requests or hasn't caught up to the leader's log end offset for at least this time, the leader will remove the follower from ISR. | duration | 15s | |
188+
| replica.max.leader.timeout | | If a leader hasn't sent any replication responses for at least this time, the follower will report the leader to the controller. If a majority of the replicas report the leader, a new leader is selected by the controller. | duration | 15s | |
189+
| replica.max.idle.wait | | The maximum amount of time a follower will wait before making a replication request once the follower is caught up with the leader. This value should always be less than `replica.max.lag.time` to avoid frequent shrinking of ISR for low-throughput streams. | duration | 10s | |
190+
| replica.fetch.timeout | | Timeout duration for follower replication requests. | duration | 3s | |
191+
| min.insync.replicas | | Specifies the minimum number of replicas that must acknowledge a stream write before it can be committed. If the ISR drops below this size, messages cannot be committed. | int | 1 | [1,...] |
192+
193+
### Activity Configuration Settings
194+
195+
Below is the list of the configuration settings for the `activity` part of
196+
the configuration file.
197+
198+
| Name | Flag | Description | Type | Default | Valid Values |
199+
|:----|:----|:----|:----|:----|:----|
200+
| stream.enabled | | Enables the activity stream. This will create an internal stream called `__activity` which events will be published to. | bool | false | |
201+
| stream.publish.timeout | | The timeout for publishes to the activity stream. This is the time to wait for an ack from the activity stream, which means it's related to `stream.publish.ack.policy`. If the ack policy is `none`, this has no effect. | duration | 5s | |
202+
| stream.publish.ack.policy | | The ack policy to use for publishes to the activity stream. The value `none` means publishes will not wait for an ack, `leader` means publishes will wait for the ack sent when the leader has committed the event, and `all` means publishes will wait for the ack sent when all replicas have committed the event. | string | all | [none, leader, all] |
203+

0 commit comments

Comments
 (0)