Model-definition framework for state-of-the-art text, vision, audio and multimodal machine learning models, for inference and training.
167.9K34.9KPythonApache-2.0Updated yesterday
espnet/espnet
An end-to-end speech processing toolkit built on PyTorch for speech recognition, text-to-speech, translation, enhancement and more.
- Stars
- 9,993
- Forks
- 2,445
- Open issues
- 92
- Last commit
- 9 hours ago
What is it?
ESPnet is a PyTorch-based toolkit for end-to-end speech processing. It covers speech recognition (ASR), text-to-speech (TTS), speech translation, speech enhancement and separation, speaker diarization, spoken language understanding, singing voice synthesis and speech language models. Each task is organized as a recipe with a corpus-agnostic pipeline, from data preparation through training to evaluation.
The project provides pretrained models on Hugging Face, including OWSM models for multilingual recognition and translation. It can be installed with pip and used from the command line or Python, for example to transcribe or translate an audio file, synthesize speech from text, or enhance a noisy recording. An MCP server exposes the same tasks to AI agents.
Training supports distributed setups such as DDP, multi-node runs with Slurm or MPI, and DeepSpeed. Researchers and engineers building speech systems can use the shared recipe structure to reproduce published results or adapt models to their own data.
Key features
- 01Speech recognition, translation, synthesis, enhancement and phonemization via CLI and Python.
- 02Pretrained models on Hugging Face, including OWSM multilingual models.
- 03Reproducible recipes for 200+ corpora with a shared pipeline.
- 04Supports Transformer, Conformer, Branchformer and Transducer architectures.
- 05Distributed training with DDP, multi-node Slurm/MPI and DeepSpeed.
- 06MCP server exposing speech tasks to AI agents.
Use cases
- Transcribing or translating recorded audio in many languages.
- Training a custom speech recognition or TTS model on your own corpus.
- Reproducing published speech benchmarks with standard recipes.
- Adding speech enhancement or speaker verification to an application.
Quick start
- 1
Install PyTorch first from the official PyTorch site
- 2
Install ESPnet to run pretrained models
pip install espnet - 3
Transcribe an audio file from the terminal
espnet transcribe audio.wav
Things to know
- Training needs the extra install espnet[train] and recipes need a fuller setup from the installation guide.
- ESPnet1 is no longer supported, so use ESPnet2 or ESPnet3 instead.
Topics
FAQ
- What is espnet?
- An end-to-end speech processing toolkit built on PyTorch for speech recognition, text-to-speech, translation, enhancement and more. ESPnet is a PyTorch-based toolkit for end-to-end speech processing. It covers speech recognition (ASR), text-to-speech (TTS), speech translation, speech enhancement and separation, speaker diarization, spoken language understanding, singing voice synthesis and speech language models. Each task is organized as a recipe with a corpus-agnostic pipeline, from data preparation through training to evaluation.
- Is espnet free to use? What license does it have?
- espnet is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is espnet still maintained?
- Actively maintained. The last commit to espnet was on Oct 21, 2026.
- What can I use espnet for?
- Transcribing or translating recorded audio in many languages. Training a custom speech recognition or TTS model on your own corpus. Reproducing published speech benchmarks with standard recipes. Adding speech enhancement or speaker verification to an application.
- How do I install espnet?
- Install PyTorch first from the official PyTorch site Install ESPnet to run pretrained models: pip install espnet Transcribe an audio file from the terminal: espnet transcribe audio.wav
- Where can I get espnet?
- The source code and installation instructions are on GitHub: https://github.com/espnet/espnet
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
Open-source voice AI family from Microsoft, with long-form speech recognition (ASR), multi-speaker text-to-speech (TTS), and real-time streaming TTS models.
54.9K6.2KPythonMITUpdated 13 days ago
Open-source speech recognition toolkit for offline and streaming ASR, with VAD, punctuation, speaker labels and OpenAI-compatible serving.
20.8K2.1KPythonMITUpdated 3 days ago
NVIDIA's open-source toolkit for building, fine-tuning and deploying speech AI models, covering speech recognition (ASR), text-to-speech (TTS) and speech LLMs.
18.6K3.7KPythonApache-2.0Updated yesterday
Open-source text-to-speech system using flow matching with a Diffusion Transformer, supporting zero-shot voice cloning from a short reference audio.
AI & Machine Learning#text-to-speech#speech-synthesis#flow-matching#diffusion-transformer#voice-cloning
15.4K2.3KPythonMITUpdated 7 days ago
Swift SDK for running speech-to-text, text-to-speech and speaker diarization fully on-device on Apple silicon.
6.4K622SwiftMITUpdated 11 days ago