Discussion: Add Voxtral backend option for WhisperX

[TKaluza]

💬 Discussion: Add Voxtral backend option for WhisperX

Why Voxtral?

• Two models: Voxtral Small (24 B) for production and Mini (3 B) for edge/local deployment, both Apache 2.0 open-source (mistral.ai)
• Outperforms Whisper large‑v3 on transcription (lower WER), especially in multilingual benchmarks (FLEURS, Common Voice) (mistral.ai)
• Supports long-form audio (up to 30 min transcribe, 40 min for reasoning) with 32K‑token context
• Built-in summarization, Q&A, and voice‑triggered function‑calling—no separate LLM needed

Proposal Scope:

• Add optional Voxtral backend flags (e.g. --voxtral-mini, --voxtral-small)
• Support both local and API use (incorporating Mini‑Transcribe)
• Enable semantic features: Q&A, summarization, voice-based function calls
• Benchmark against WhisperX pipeline on key metrics (WER, latency, multilingual support)

Questions

1. How to manage chunking vs. Voxtral’s long-context?
2. How to map output formats and integrate semantic features?
3. Should function-calling and QA be optional extras?

---

I am not sure if this is suited for whisper-x.

[remic33]

Whisper has close to no evolution since the large-v3 and we it seems Voxtral crack some things that could not have been built with whisper (large context first)
I think the implementation could be something interesting to evolve the lib behind whisper, we will have to see if some optimized version of voxtral are created too.