chinese-word-segmentation

Jcseg is a light weight NLP framework developed with Java. Provide CJK and English segmentation based on MMSEG algorithm, With also keywords extraction, key sentence extraction, summary extraction implemented based on TEXTRANK algorithm. Jcseg had a build-in http server and search modules for lucene,solr,elasticsearch,opensearch

java nlp natural-language-processing chinese-nlp chinese-text-segmentation nlp-keywords-extraction pos-tagging solr-plugin chinese-word-segmentation jcseg mmseg lucene-analyzer elasticsearch-analyzer keywords-extraction lucene-tokenizer jcseg-analyzer opensearch-analyzer opensearch-tokenizer elasticsearch-tokenizer

Updated Sep 18, 2023
Java

mammothb / symspellpy

Star

Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm

python spellcheck fuzzy-search fuzzy-matching edit-distance levenshtein levenshtein-distance spelling spell-check chinese-text-segmentation word-segmentation approximate-string-matching spelling-correction damerau-levenshtein text-segmentation chinese-word-segmentation symspell

Updated Nov 17, 2024
Python

messense / jieba-rs

Sponsor

Star

The Jieba Chinese Word Segmentation Implemented in Rust

nlp wasm jieba chinese-word-segmentation jieba-chinese

Updated Jun 1, 2024
Rust

lionsoul2014 / friso

Star

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

c tokenizer full-text-search chinese-word-segmentation chinese-tokenizer php-tokenizer korean-tokenizer japanese-tokenizer cjk-tokenizer

Updated Oct 29, 2023
C

monpa-team / monpa

Star

MONPA 罔拍是一個提供正體中文斷詞、詞性標註以及命名實體辨識的多任務模型

nlp named-entity-recognition pos ner word-segmentation albert bert pos-tagging chinese-word-segmentation

Updated Jul 18, 2022
Python

Kyubyong / g2pC

Star

g2pC: A Context-aware Grapheme-to-Phoneme Conversion module for Chinese

crf pinyin chinese-nlp crfsuite chinese-word-segmentation g2p

Updated Jul 10, 2019
Python

hemingkx / WordSeg

Star

A PyTorch implementation of a BiLSTM \ BERT \ Roberta ( BiLSTM CRF) model for Chinese Word Segmentation (中文分词) .

pytorch bert chinese-word-segmentation bilstm-crf roberta bert-crf

Updated Jul 28, 2022
Python

supercoderhawk / DeepLearning_NLP

Star

基于深度学习的自然语言处理库

natural-language-processing deep-learning tensorflow named-entity-recognition relation-extraction chinese-word-segmentation chinese-tokenizer

Updated Nov 3, 2018
Python

howl-anderson / MicroTokenizer

Sponsor

Star

一个轻量且功能全面的中文分词器，帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..

tokenizer chinese-nlp nlp-machine-learning chinese-word-segmentation chinese-tokenizer educational-project dag-network