📖Project Page ｜🤗 Hugging Face｜ 🤖 ModelScope

• 🚀 First Unified Continuous Speech Tokenizer: the first continuous audio tokenizer to effectively integrate semantic and acoustic features, suitable for both understanding and generation tasks.
• 🎧 High-Quality Reconstruction: Achieve high-quality audio generation by modeling continuous features with a VAE, minimizing information loss and preserving intricate acoustic textures.
• 🌐 Convolution-Free Efficiency: Built on a pure causal transformer architecture, completely eliminating convolutional layers for superior efficiency and a simpler design.

pip install -r requirements.txt

import torch
import torchaudio

from audio_tokenizer.modeling_audio_vae import AudioVAE

model = AudioVAE.from_pretrained('inclusionAI/MingTok-Audio')
model = model.cuda()
model.eval()

waveform, sr = torchaudio.load('data/1089-134686-0000.flac', backend='soundfile')
sample = {'waveform': waveform.cuda(), 'waveform_length': torch.tensor([waveform.size(-1)]).cuda()}

with torch.no_grad():
    with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
        latent, frame_num = model.encode_latent(**sample)
        output_waveform = model.decode(latent)

torchaudio.save('./1089-134686-0000_reconstruct.wav', output_waveform.cpu()[0], sample_rate=16000)

1. We borrowed a lot of code from X-Codec-2.0 for tokenizer training.
2. We thank the OpenAI team for developing the Whisper model and making its weights publicly available.

This code repository is licensed under the MIT License, and the Legal Disclaimer is located in the LEGAL.md file under the project's root directory.

If you find our work helpful, feel free to give us a cite.