Release v0.21.2 · huggingface/tokenizers

What's Changed

This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!

Update the release builds following 0.21.1. by @Narsil in #1746
replace lazy_static with stabilized std::sync::LazyLock in 1.80 by @sftse in #1739
Fix no-onig no-wasm builds by @414owen in #1772
Fix typos in strings and comments by @co63oc in #1770
Fix type notation of merges in BPE Python binding by @Coqueue in #1766
Bump http-proxy-middleware from 2.0.6 to 2.0.9 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1762
Fix data path in test_continuing_prefix_trainer_mismatch by @GaetanLepage in #1747
clippy by @ArthurZucker in #1781
Update pyo3 and rust-numpy depends for no-gil/free-threading compat by @Qubitium in #1774
Use ApiBuilder::from_env() in from_pretrained function by @BenLocal in #1737
Upgrade onig, to get it compiling with GCC 15 by @414owen in #1771
Itertools upgrade by @sftse in #1756
Bump webpack-dev-server from 4.10.0 to 5.2.1 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1792
Bump brace-expansion from 1.1.11 to 1.1.12 in /bindings/node by @dependabot in #1796
Fix features blending into a paragraph by @bionicles in #1798
Adding throughput to benches to have a more consistent measure across by @Narsil in #1800
Upgrading dependencies. by @Narsil in #1801
[docs] Whitespace by @stevhliu in #1785
Hotfixing the stub. by @Narsil in #1802
Bpe clones by @sftse in #1707
Fixed Length Pre-Tokenizer by @jonvet in #1713
Consolidated optimization ahash dary compact str by @Narsil in #1799
🚨 breaking: Fix training with special tokens by @ArthurZucker in #1617

Full Changelog: v0.21.1...v0.21.2rc0