Add MLX backend implementation for WhisperX #1185
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add MLX backend adapter (mlx_whisper_v2.py) with auto-download from HuggingFace - Implement unified pipeline interface with load_mlx_pipeline() - Add process separation to handle PyTorch/MLX conflicts - Implement batch processing for 30s chunks - Add model conversion tools for Whisper, Wav2Vec2, and VAD models - Create comprehensive documentation and examples - Add unit tests and performance validation - Update CLI to support --backend mlx flag Performance results: - Tiny model: 122x real-time - Large-v3 model: 15x real-time (7.4x faster than CPU) - Meets and exceeds all roadmap targets Co-Authored-By: Claude <[email protected]>
- MLX backend now accepts --align_model flag for word timestamps (instead of --word_level) - Enable word timestamps automatically when --align_model is specified with MLX - Skip redundant alignment step for MLX (word timestamps included natively) - Fix PyTorch 2.6 compatibility for pyannote VAD - Update benchmarking script with accuracy analysis - MLX shows 4.44x average speedup with high transcription accuracy 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]>
Implements smart batch processing for MLX WhisperX backend that achieves 16% performance improvement with optimal batch size of 8. Key improvements: - Add BatchedMLXWhisperBackend class for optimized segment processing - Group segments by similar length to minimize padding overhead - Load model once and reuse for all segments - Fix KeyError in performance reporting - Add comprehensive batch vs sequential performance testing Performance results (120s test audio, 4 segments): - Sequential: 7.31s (16.4x RTF) - Batch size 8: 6.31s (19.0x RTF) - 16% faster - No accuracy loss - identical transcription results The optimization works by minimizing framework overhead and improving memory access patterns, even though MLX doesn't support true parallel batch processing at the encoder level. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This commit completes the batch processing implementation by: - Passing batch_size and use_batch_processing to ProcessSeparatedPipeline - Adding quantization parameter support for batch processing - Fixing batch_size parameter error in mlx_whisper_v2 backend These changes enable the 16% performance improvement with batch size 8 that was implemented but not fully connected to the CLI. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This commit renames the project from WhisperX to WhisperX-MLX to better reflect its focus on MLX backend support for Apple Silicon. Changes: - Update project name in pyproject.toml to whisperx-mlx - Update repository URL to github.com/sooth/whisperx-mlx - Add MLX dependencies (mlx, mlx-whisper) to requirements - Create new README focused on MLX features and performance - Add CHANGELOG_MLX documenting enhancements over original WhisperX The project maintains full compatibility with original WhisperX while adding optimized MLX backend support for Apple Silicon Macs. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add MlxWhisperBackend implementation for optimized inference on Apple Silicon - Add base backend class for common functionality - Fix import in __init__.py to use correct mlx_whisper module - Include PyAnnote VAD patch for PyTorch 2.6+ compatibility This enables MLX backend usage in fresh clones of the repository. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
The faster_whisper.py file was present locally but not tracked in git, causing ImportError in fresh clones. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
BREAKING CHANGE: This fork now exclusively uses the MLX backend for Apple Silicon. - Replace asr.py with MLX-only implementation - Remove FasterWhisperBackend and all faster-whisper dependencies - Update backends/__init__.py to only export MlxWhisperBackend - Maintain API compatibility through MLXWhisperPipeline wrapper - Add Apple Silicon hardware check in load_model - Simplify codebase by removing conditional backend logic This change makes WhisperX an MLX-exclusive fork optimized for Apple Silicon, providing significant performance improvements over the original implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…essing - Fix VAD audio segmentation in pipeline - Restore batch processing capability (17.5x real-time for large-v3) - Fix backend selection logic (standard vs batch) - Ensure both Silero and PyAnnote VAD work correctly - Fix mlx_batch_optimized to process audio segments correctly - Add proper model name mapping for MLX models Tested on 30-minute audio: - Large-v3 processes in 103.53 seconds (1.73 minutes) - Achieves 17.5x real-time factor - Successfully segments and transcribes with VAD 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add --backend argument to CLI with choices: auto, mlx, standard, batch - Pass backend parameter through to load_model - Add int4 as a valid compute_type choice - Backend defaults to "auto" which selects appropriate MLX backend 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Remove device from kwargs when initializing PyAnnote to avoid TypeError - Device was being passed both as positional argument and in kwargs - This fixes the "multiple values for argument 'device'" error 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Handle TranscriptionResult objects properly in writer - Convert TranscriptionResult to dict when needed - Ensure language is set even when alignment is skipped - Fix case where no_align=True prevented output files This fixes the issue where transcription completes but no output files are generated. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add GETTING_STARTED.md with installation and usage instructions - Add README_MLX.md as a concise fork-specific README - Add MLX_FORK_STATUS.md documenting current implementation status - Add test_mlx_complete.py for comprehensive testing - Add test_fresh_clone.sh to verify fresh clone functionality
- Fork mlx-whisper with performance optimizations - Update pyproject.toml to use Git dependency - Add missing MLX and librosa dependencies - Remove legacy ctranslate2 and faster-whisper dependencies - Update package name to whisperx-mlx
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Performance results: