Repolode
Submit
segment-any-text

segment-any-text/wtpsplit

A Python toolkit that splits text into sentences or other semantic units, with pretrained SaT and WtP models covering 85 languages.

View on GitHub
Stars
1,378
Forks
84
Open issues
2
Last commit
2 days ago

What is it?

wtpsplit segments text into sentences, and can also split paragraphs using newline predictions. It provides the SaT (Segment any Text) models, which cover 85 languages, along with the earlier WtP models for reproducibility. Models are loaded by name from Hugging Face, and a simple split call returns a list of sentences for one text or an iterator over many texts.

Beyond plain sentence splitting, the library can enforce minimum and maximum segment lengths using a Viterbi or greedy search over model boundary probabilities, optionally guided by length priors such as Gaussian or log-normal distributions. Pretrained LoRA adapters tune segmentation to particular languages, domains, or writing styles, and users can train their own adapters from a small set of correctly split sentences.

Inference can run on PyTorch with GPU, TPU, or TorchInductor compilation, or through ONNX Runtime with CUDA or CPU providers. The project is written in Python and released under the MIT license.

Key features

  • 01Sentence segmentation for 85 languages using pretrained SaT models.
  • 02Length-constrained splitting with min_length and max_length options.
  • 03Optional paragraph segmentation based on predicted newline probabilities.
  • 04LoRA adapters for specific languages, domains, and writing styles.
  • 05Inference via PyTorch on GPU or TPU, TorchInductor compilation, or ONNX Runtime.

Use cases

  • Splitting raw documents into sentences before embedding or indexing them.
  • Preprocessing transcripts or subtitles that lack reliable punctuation.
  • Segmenting multilingual corpora for NLP training data preparation.
  • Producing chunks within a size limit for downstream language model pipelines.

Quick start

  1. 1

    Install the package with pip

    pip install wtpsplit
  2. 2

    Import SaT and load a pretrained model

    from wtpsplit import SaT
  3. 3

    Split text into sentences

    sat.split("This is a test This is another test.")

Things to know

  • Running large models efficiently is much faster with a GPU, which the README optionally recommends.
  • Some advanced features such as ONNX and AITune require separate installs and extra setup.

In collections

Natural language processing tools & libraries

Topics

#nlp#sentence-segmentation#text-processing#pytorch#onnx#multilingual#deep-learning#machine-learning#natural-language-processing#pretrained-models

FAQ

What is wtpsplit?
A Python toolkit that splits text into sentences or other semantic units, with pretrained SaT and WtP models covering 85 languages. wtpsplit segments text into sentences, and can also split paragraphs using newline predictions. It provides the SaT (Segment any Text) models, which cover 85 languages, along with the earlier WtP models for reproducibility. Models are loaded by name from Hugging Face, and a simple split call returns a list of sentences for one text or an iterator over many texts.
Is wtpsplit free to use? What license does it have?
wtpsplit is open source under the MIT license. Check the license terms before using it in commercial products.
Is wtpsplit still maintained?
Actively maintained. The last commit to wtpsplit was on Nov 5, 2026.
What can I use wtpsplit for?
Splitting raw documents into sentences before embedding or indexing them. Preprocessing transcripts or subtitles that lack reliable punctuation. Segmenting multilingual corpora for NLP training data preparation. Producing chunks within a size limit for downstream language model pipelines.
How do I install wtpsplit?
Install the package with pip: pip install wtpsplit Import SaT and load a pretrained model: from wtpsplit import SaT Split text into sentences: sat.split("This is a test This is another test.")
Where can I get wtpsplit?
The source code and installation instructions are on GitHub: https://github.com/segment-any-text/wtpsplit

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects