中文分词笔记2026

目录结巴分词使用例子数字人断句Genius结巴分词速度特别快基本不占用时间pip install jieba -i https://pypi.tuna.tsinghua.edu.cn/simple使用例子import jieba import time text 我来到了北京清华大学 # 精确模式分词并统计时间 start_time time.time() seg_list jieba.cut(text, cut_allFalse) end_time time.time() # 将生成器转换为列表以便查看结果 result list(seg_list) elapsed_time (end_time - start_time) * 1000 # 转换为毫秒 print(f分词结果: {/ .join(result)}) print(f执行时间: {elapsed_time:.4f} 毫秒)数字人断句def merge_tokens(tokens, min_len4, tail_min_len3): 合并 tokens以括号闭合作为优先断句点 result [] current bracket_stack [] # 用栈记录括号位置 for token in tokens: current token # 记录括号 for ch in token: if ch in ([{: bracket_stack.append(ch) elif ch in )]}: if bracket_stack: bracket_stack.pop() # 判断截取条件 should_cut False # 优先括号栈为空且刚遇到右括号即括号已完全闭合 if not bracket_stack and any(ch in current for ch in )]}): should_cut True # 其次达到最小长度 elif len(current) min_len: should_cut True if should_cut: result.append(current) current # 处理尾巴 if current: if result and len(current) tail_min_len: result[-1] current else: result.append(current) return resultGeniusGenius是一个开源的python中文分词组件采用 CRF(Conditional Random Field)条件随机场算法。https://github.com/duanhongyi/genius序列标签https://github.com/guillaumegenthial/sequence_taggingBidirectional LSTM-CRF for Sequence Labeling. Easy-to-use and state-of-the-art performance.https://github.com/Hironsan/anagoLSTM-CRF models for sequence labeling in text.https://github.com/abhyudaynj/LSTM-CRF-modelshttps://github.com/LiyuanLucasLiu/LM-LSTM-CRF