Model-definition framework for state-of-the-art text, vision, audio and multimodal machine learning models, for inference and training.
167.9K34.9KPythonApache-2.0Updated yesterday
shibing624/pycorrector
Python toolkit for Chinese text error correction, covering spelling, phonetic, shape-similar and grammar errors with models from KenLM to Qwen-based LLMs.
- Stars
- 6,534
- Forks
- 1,165
- Open issues
- 25
- Last commit
- 3 months ago
What is it?
pycorrector is a Python toolkit for correcting errors in Chinese text. It targets phonetically similar, visually similar, and grammatical mistakes, and also handles extra or missing characters and word-order errors in its newer LLM-based models. Users can call a ready-made corrector with a few lines of code, or run the bundled command-line tool for batch files.
The project wraps several approaches behind a common interface: a KenLM n-gram language model combined with confusion sets, MacBERT-based and T5-based seq2seq correctors, an ERNIE-CSC model on PaddlePaddle, and GPT-style correctors built on ChatGLM3, Qwen2.5 and Qwen3. Each method has its own example directory with training and prediction scripts, and the project reports sentence-level F1 on SIGHAN-2015, EC-LAW and MCSC for comparison.
The toolkit also includes an English spelling corrector, traditional-to-simplified Chinese conversion, custom confusion sets for whitelisting or adding known terms, and a Gradio demo. Models are downloaded from Hugging Face on first use.
Key features
- 01Kenlm n-gram model with confusion sets for Chinese spelling correction.
- 02MacBERT4CSC, T5, ERNIE-CSC and ChatGLM/Qwen-based correctors behind one interface.
- 03Detection of error positions with character- or word-level output.
- 04Custom confusion sets and language models can be loaded.
- 05Batch correction from the command line and an English spelling corrector.
Use cases
- Cleaning up typos in Chinese user input such as search queries or chat messages.
- Proofreading speech recognition or OCR output that contains phonetic or shape-based mistakes.
- Comparing different Chinese correction models on a shared test set.
- Fine-tuning a correction model on a domain-specific dataset of errors.
Quick start
- 1
Install the package from PyPI
pip install -U pycorrector - 2
Run the kenlm correction example in Python
Things to know
- The default kenlm language model file is about 2.8G, which can be heavy on low-memory machines.
- Stronger LLM-based models such as Qwen require a GPU for practical speed.
Topics
FAQ
- What is pycorrector?
- Python toolkit for Chinese text error correction, covering spelling, phonetic, shape-similar and grammar errors with models from KenLM to Qwen-based LLMs. pycorrector is a Python toolkit for correcting errors in Chinese text. It targets phonetically similar, visually similar, and grammatical mistakes, and also handles extra or missing characters and word-order errors in its newer LLM-based models. Users can call a ready-made corrector with a few lines of code, or run the bundled command-line tool for batch files.
- Is pycorrector free to use? What license does it have?
- pycorrector is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is pycorrector still maintained?
- Updated within 6 months. The last commit to pycorrector was on Jul 25, 2026.
- What can I use pycorrector for?
- Cleaning up typos in Chinese user input such as search queries or chat messages. Proofreading speech recognition or OCR output that contains phonetic or shape-based mistakes. Comparing different Chinese correction models on a shared test set. Fine-tuning a correction model on a domain-specific dataset of errors.
- How do I install pycorrector?
- Install the package from PyPI: pip install -U pycorrector Run the kenlm correction example in Python
- Where can I get pycorrector?
- The source code and installation instructions are on GitHub: https://github.com/shibing624/pycorrector
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
Python library that runs large language models layer by layer so 70B-class models can infer on a single 4GB GPU, without quantization.
39.5K4KJupyter NotebookApache-2.0Updated 2 days ago
Open-source large language models fine-tuned for financial tasks such as sentiment analysis, forecasting and financial NLP benchmarks.
21.4K3KJupyter NotebookMITUpdated 28 days ago
A fast BPE tokeniser for OpenAI's models, used to convert text into tokens and back losslessly.
19.5K1.7KPythonMITUpdated 2 months ago
A retrieval toolkit from BAAI providing BGE embedding and reranker models plus training, evaluation and tutorials for search and RAG.
12.3K1.1KPythonMITUpdated 2 months ago
A Python library from Microsoft that compresses LLM prompts and KV cache by up to 20x by dropping non-essential tokens, with minimal loss in performance.
6.8K444PythonMITUpdated last month