Supervised speech tokenizer

[LqNoob]

Is your feature request related to a problem? Please describe.
The paper includes methods about "Supervised speech tokenizer", but the repository does not contain the training and reconstruction test code for this part.

Describe the solution you'd like
Did you perhaps forget to open-source this part of the code? Will you open-source this implementation in the future?

[aluminumbox]

we have no plan to opensource speech tokenizer, we only release its onnx file

[LqNoob]

we have no plan to opensource speech tokenizer, we only release its onnx file

This message is quite regrettable, but thank you for your quick reply!

However, I have another question I would like to consult with you about. When you were conducting experiments with the speech tokenizer, how did you determine that "using ASR as backbone and having the tokenizer provide only semantic information (without acoustic information)" was feasible?

Thank you！

[aluminumbox]

we have no plan to opensource speech tokenizer, we only release its onnx file

This message is quite regrettable, but thank you for your quick reply!

However, I have another question I would like to consult with you about. When you were conducting experiments with the speech tokenizer, how did you determine that "using ASR as backbone and having the tokenizer provide only semantic information (without acoustic information)" was feasible?

Thank you！

we did some experiement and verified that such token can recover original mel info. @ZhihaoDU can tell you more details

[LqNoob]

we have no plan to opensource speech tokenizer, we only release its onnx file

This message is quite regrettable, but thank you for your quick reply!

However, I have another question I would like to consult with you about. When you were conducting experiments with the speech tokenizer, how did you determine that "using ASR as backbone and having the tokenizer provide only semantic information (without acoustic information)" was feasible?

Thank you！

@ZhihaoDU Could you provide some detailed and specific information? I would be very grateful.

[ZhihaoDU]

we have no plan to opensource speech tokenizer, we only release its onnx file

This message is quite regrettable, but thank you for your quick reply!
However, I have another question I would like to consult with you about. When you were conducting experiments with the speech tokenizer, how did you determine that "using ASR as backbone and having the tokenizer provide only semantic information (without acoustic information)" was feasible?
Thank you！

@ZhihaoDU Could you provide some detailed and specific information? I would be very grateful.

Thanks for your attention. I believe, we have not claimed there is only semantic information (without any acoustic information) in the S3 tokens. In fact, decoupling semantic and acoustic information is still an open problem to my best knowledge. What we can do is modeling semantic information in the token as more as possible. That is also the motivation, we employ the ASR task to learn the tokenizer in a supervised manner. We think such training process can maintain more semantic information into the tokens than self-supervised or unsupervised methods, such as Encoder, SoundStream, HuBert, wave2bert and so on.

[LqNoob]

@ZhihaoDU
Thank you for your explanation of the motivation behind the speech tokenizer in the paper; it has been very helpful to me.
I will close this issues.