[enhancement]: Support tikv as a redis replacement

[williamdes]

Which feature or improvement would you like to request?

See #23 as a thread reference

https://github.com/tikv/tikv

https://github.com/tikv/client-rust

Please let me know if this is a viable option to be implemented, so I can do the server setup and let you know if it works like it promises.

To be noted that in fact it runs a RocksDB on each node.
See https://tikv.org/docs/7.1/reference/architecture/storage/

Also, why can tikv not also be a data store?

Is your feature request related to a problem?

I do not want to deploy Redis, and will probably deploy something that I feel is better designed for multi site/datacenter

Code of Conduct

• I agree to follow this project's Code of Conduct

[alvinpeters]

Looks similar to FoundationDB. I will work on it.

Working branch: https://github.com/alvinpeters/stalwart-mail-server/tree/add-tikv

[alvinpeters]

@williamdes LOL I got it working somehow (albeit in a shithouse condition). Wanna help me dogfood/test this?

You can clone and compile the branch now:

git clone --branch add-tikv https://github.com/alvinpeters/stalwart-mail-server.git
cd stalwart-mail-server
cargo build --bin stalwart-mail  --features tikv

Since web-admin doesn't show it yet, you'd have to manually add it to the config.toml:

[store."tikv"]
type = "tikv"
pd-endpoints = ["127.0.0.1:2379"]

Then, you have to manually set the stores used since you can't just select it in the web admin, like:

[storage]
data = "tikv"
fts = "tikv"
blob = "tikv"
lookup = "tikv"
directory = "internal"

[directory."internal"]
type = "internal"
store = "tikv"

Currently setting up tests. I reckon I can get this in a ready shape within next week.

[williamdes]

Thank you so much, I will help you testing
That said I keep the data in Garage garagehq.deuxfleurs.fr/
And directory in LDAP

[mdecimus]

@alvinpeters have you tried running the store tests?

Add the tikv store to tests/src/store/mod.rs and then run:

$ cd tests
$ STORE=tikv cargo test store_tests -- --nocapture

If that works try with the IMAP and JMAP tests:

$ STORE=tikv cargo test imap_tests -- --nocapture
$ STORE=tikv cargo test jmap_tests -- --nocapture

[alvinpeters]

Thanks! I didn't know how to properly run the test, so was only testing by just running the entire thing and was working kinda smoothly lol.

Either I messed up with the blobbing or I set the limit too high. I'll fix it when I wake up.

running 1 test
Testing store tikv...
Creating blobs...
Creating account data...
Creating queue, config and lookup data...
Creating directory data...
Calculating store hash...
thread 'store::store_tests' panicked at tests/src/store/import_export.rs:330:14:
called `Result::unwrap()` on an `Err` value: Event { inner: Store(TikvError), keys: [(Reason, String("gRPC api error: status: OutOfRange, message: \"Error, message length too large: found 21149150 bytes, the limit is: 4194304 bytes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"grpc-accept-encoding\": \"identity, deflate, gzip\"} }")), (CausedBy, Static("crates/store/src/dispatch/store.rs:117"))] }

Should I submit a PR now so other people can easily contribute?

[williamdes]

I definitely think a PR would be great to discuss the implementation

[alvinpeters]

Good thing I saw this before I sleep. Submitted #668 and stalwartlabs/webadmin#15 as draft PRs so other people can see/help.

[williamdes]

Thank you for the implementation, it will help me if I try to implement etcd
Here is a pure Rust implementation compatible with etcd (https://etcd.io): https://github.com/xline-kv/Xline

etcd is used in the PD of TiKV

I am still searching for the best lightweight solution to setup.
For now most of them seem to use RocksDB: TiKV, Xline

For now TiKV launches quite a lot of processes and requires a PD process.
TiKV is not in Debian, but etcd-server is.

Will keep you all updated

Update 1, see: #742 for etcd

PS (personal opinion): I think that having to use software like tiup to run a setup is bullshit, it is nice to have but hides config details. Xline and etcd are much simpler, just run them after install. No downloading of components..