Repolode
Submit
go-ego

go-ego/gse

A Go text segmentation and NLP library for English, Chinese, Japanese and other languages, implementing the jieba approach in Go.

View on GitHub
Stars
2,847
Forks
233
Open issues
15
Last commit
3 days ago

What is it?

gse is a Go library for splitting text into words across multiple languages, including English, Chinese, Japanese and others. It is a Go implementation of the jieba segmentation approach, extended with additional NLP features such as part-of-speech tagging.

The segmenter offers several modes: standard, search engine, full and precise segmentation, plus an HMM mode that uses the Viterbi algorithm to recognize words missing from the dictionary. Users can load custom and embedded dictionaries, along with stop-word lists, and analyze or trim the resulting segments. Traditional Chinese is supported as well.

The library is commonly used to prepare text for full-text search, and it has integrations with riot, zincsearch, Elasticsearch and bleve.

Key features

  • 01Multiple segmentation modes: standard, search engine, full and precise
  • 02HMM mode using the Viterbi algorithm for out-of-dictionary words
  • 03Custom and embedded dictionaries, with stop-word support
  • 04Part-of-speech tagging and segment analysis
  • 05Supports English, Chinese (including Traditional), Japanese and other languages
  • 06Includes a JSON RPC service example

Use cases

  • Tokenizing Chinese or Japanese text for full-text search indexes
  • Building a Go service that segments user input for keyword extraction
  • Adding word segmentation to a search backend such as Elasticsearch or bleve
  • Applying custom dictionaries to domain-specific text processing

Quick start

  1. 1

    Install the gse package with Go modules

    go get -u github.com/go-ego/gse
  2. 2

    Import the package in your Go code

    import "github.com/go-ego/gse"
  3. 3

    Create a segmenter, load the dictionary and call Cut on your text

Things to know

  • Chinese, Japanese and other dictionaries and embedded data are needed for non-English segmentation, which adds setup effort.

In collections

Natural language processing tools & libraries

Topics

#nlp#text-segmentation#chinese-segmentation#go#search#jieba#word-segmentation#multilingual#chinese#english

FAQ

What is gse?
A Go text segmentation and NLP library for English, Chinese, Japanese and other languages, implementing the jieba approach in Go. gse is a Go library for splitting text into words across multiple languages, including English, Chinese, Japanese and others. It is a Go implementation of the jieba segmentation approach, extended with additional NLP features such as part-of-speech tagging.
Is gse free to use? What license does it have?
gse is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is gse still maintained?
Actively maintained. The last commit to gse was on Nov 4, 2026.
What can I use gse for?
Tokenizing Chinese or Japanese text for full-text search indexes Building a Go service that segments user input for keyword extraction Adding word segmentation to a search backend such as Elasticsearch or bleve Applying custom dictionaries to domain-specific text processing
How do I install gse?
Install the gse package with Go modules: go get -u github.com/go-ego/gse Import the package in your Go code: import "github.com/go-ego/gse" Create a segmenter, load the dictionary and call Cut on your text
Where can I get gse?
The source code and installation instructions are on GitHub: https://github.com/go-ego/gse

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects