Web-Crawler in Rust

[tclaus]

Here is my crawler made in Rust.
Supports seed database, adds external links to database, simple parsing and adding to elastic search.

https://github.com/tclaus/web_crawler

[let4be]

Kindly please add mine
It's a broad web crawler written in Rust and Redis, with heavy focus on performance, scalability and politeness (almost touching 10 gbit/s on 96 core c5.metal)
https://github.com/let4be/crusty