F5-TTS

F5-TTS is an open source fully non-autoregressive text-to-speech conversion system (Text-to-Speech, TTS) based on stream matching developed by Shanghai Jiao Tong University. It stands out for its high efficiency, naturalness and multi-language support, which is close to commercial level.

GitHub address: https: //github.com/SWivid/F5-TTS

Here are some key features and technical highlights of F5-TTS:

1. Fully non-autoregressive architecture: F5-TTS adopts a fully non-autoregressive model, which is capable of processing the entire speech synthesis task in parallel, significantly improving processing speed and efficiency, and achieving a real-time factor (RTF) inference speed of 0.15, which is far better than the current diffusion-based TTS model.
2. Stream Matching Technology: Advanced stream matching technology is used in F5-TTS, which is an optimal transmission path-based approach for improving the learning process of the generative model. This technique allows the model to more accurately simulate the distribution of the target data, thus improving the naturalness and accuracy of the synthesised speech.
3. Application of Diffusion Transformer (DiT): F5-TTS makes use of the Diffusion Transformer, a Transformer structure that incorporates a diffusion model specifically designed to deal with alignment and data generation issues in generative tasks.DiT can maintain high quality in text-to-speech synthesis while providing faster response time and lower resource consumption.
4. Sway Sampling Strategy: At inference time, F5-TTS introduces the Sway Sampling strategy, a novel sampling method for more efficient sample selection in the flow step of the model. This approach can further enhance the naturalness and intelligibility of speech generation while maintaining a high degree of consistency with the original text.
5. Simplified training process: Compared to traditional models, F5-TTS does not require complex components such as duration predictors, text encoders or phoneme alignment modules. This simplification greatly reduces the complexity of the model, making the training process faster while reducing the demand on computational resources.
6. High performance and multilingual support: F5-TTS was trained on a public 100K hour multilingual dataset, demonstrating its high natural processing power for multiple languages. The model supports seamless code-switching capabilities and speed control, allowing it to excel in diverse application scenarios.