Repolode
Submit
Picovoice

Picovoice/speech-to-text-benchmark

A minimalist, extensible framework for benchmarking speech-to-text engines on WER, CER, punctuation error, latency, CPU cost and model size.

View on GitHubWebsite
Stars
699
Forks
73
Open issues
0
Last commit
yesterday

What is it?

This project is a benchmarking framework for comparing speech-to-text engines on the same audio datasets. It runs batch and streaming engines, including cloud services such as Amazon Transcribe, Azure Speech-to-Text and Google Speech-to-Text, and local models such as Whisper, Whisper.cpp, Vosk, Moonshine, Nemotron and Picovoice Cheetah and Leopard, against common corpora like LibriSpeech, Common Voice, TED-LIUM and Fleurs.

Results are reported on several axes: word error rate, character error rate for Korean and Japanese, punctuation error rate for periods and question marks, word emission latency for streaming engines, CPU core-hours per hundred hours of audio, and aggregate model size. Results are broken out by language, covering English, French, German, Italian, Spanish, Portuguese, Korean and Japanese.

The codebase is a Python command-line tool. Each engine is selected with a flag, and separate scripts handle accuracy and latency measurement. Datasets must be downloaded locally, and cloud engines require the relevant account credentials.

Key features

  • 01Compares batch and streaming engines on the same datasets
  • 02Measures WER, CER and punctuation error rate
  • 03Reports word emission latency for streaming engines
  • 04Measures CPU core-hours and model size for local engines
  • 05Covers English, French, German, Italian, Spanish, Portuguese, Korean and Japanese

Use cases

  • Choosing a speech recognition engine for a product by comparing accuracy and latency
  • Evaluating offline models against cloud APIs on your own language
  • Comparing computational cost and model size for on-device deployment
  • Adding a new STT engine to a consistent evaluation pipeline

Quick start

  1. 1

    Install FFmpeg

  2. 2

    Install the Python requirements

    pip3 install -r requirements.txt
  3. 3

    Download the benchmark datasets

  4. 4

    Run benchmark.py with your chosen engine, dataset and language, as described in the README

Things to know

  • The benchmark was developed and tested only on Ubuntu 22.04 with Python 3.12.
  • Many engines need cloud accounts or API keys, such as Amazon, Azure, Google, IBM Watson and Picovoice.
  • You must download large speech datasets yourself before running the benchmark.

In collections

Privacy & local-first toolsAI video & audio tools

Topics

#speech-recognition#speech-to-text#benchmark#asr#offline#evaluation#aws-transcribe#cheetah#deep-learning#deep-neural-networks

FAQ

What is speech-to-text-benchmark?
A minimalist, extensible framework for benchmarking speech-to-text engines on WER, CER, punctuation error, latency, CPU cost and model size. This project is a benchmarking framework for comparing speech-to-text engines on the same audio datasets. It runs batch and streaming engines, including cloud services such as Amazon Transcribe, Azure Speech-to-Text and Google Speech-to-Text, and local models such as Whisper, Whisper.cpp, Vosk, Moonshine, Nemotron and Picovoice Cheetah and Leopard, against common corpora like LibriSpeech, Common Voice, TED-LIUM and Fleurs.
Is speech-to-text-benchmark free to use? What license does it have?
speech-to-text-benchmark is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is speech-to-text-benchmark still maintained?
Actively maintained. The last commit to speech-to-text-benchmark was on Nov 6, 2026.
What can I use speech-to-text-benchmark for?
Choosing a speech recognition engine for a product by comparing accuracy and latency Evaluating offline models against cloud APIs on your own language Comparing computational cost and model size for on-device deployment Adding a new STT engine to a consistent evaluation pipeline
How do I install speech-to-text-benchmark?
Install FFmpeg Install the Python requirements: pip3 install -r requirements.txt Download the benchmark datasets Run benchmark.py with your chosen engine, dataset and language, as described in the README
Where can I get speech-to-text-benchmark?
The source code and installation instructions are on GitHub: https://github.com/Picovoice/speech-to-text-benchmark

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects