用Python的NLTK库玩转WordNet：从词义查询到语义相似度计算实战

用Python的NLTK库玩转WordNet从词义查询到语义相似度计算实战在自然语言处理领域WordNet堪称是一座语义关系的宝库。不同于传统词典的字母排序它将英语词汇组织成一张庞大的语义网络每个节点代表一个概念同义词集节点间的连线则刻画了丰富的语义关系。对于Python开发者而言通过NLTK库调用WordNet接口能够轻松实现词义查询、语义关系挖掘乃至词语相似度计算等高级功能。本文将带你从零开始通过代码实例掌握这些实用技巧。1. 环境准备与基础查询1.1 安装与初始化首先确保已安装NLTK库及WordNet数据包pip install nltk python -m nltk.downloader wordnet初始化WordNet接口只需简单导入from nltk.corpus import wordnet as wn1.2 同义词集查询获取单词car的所有同义词集car_synsets wn.synsets(car) print(fFound {len(car_synsets)} synsets for car:) for syn in car_synsets: print(f- {syn.name()}: {syn.definition()})典型输出会显示多个角度的词义解释例如car.n.01: 机动车通常四轮car.n.02: 火车车厢car.n.03: 电梯轿厢1.3 词性标注与精确查询通过指定词性可缩小查询范围# 仅查询名词词性的bank bank_noun wn.synsets(bank, poswn.NOUN) # 仅查询动词词性 bank_verb wn.synsets(bank, poswn.VERB)WordNet支持四种词性常量wn.NOUN: 名词wn.VERB: 动词wn.ADJ: 形容词wn.ADV: 副词2. 语义关系网络探索2.1 上下位关系追踪上位词更抽象和下位词更具体构成WordNet的层级骨架dog wn.synset(dog.n.01) # 获取所有上位词 hypernyms dog.hypernyms() print(fHypernyms of dog: {[h.name() for h in hypernyms]}) # 获取所有下位词 hyponyms dog.hyponyms() print(fHyponyms of dog (first 5): {[h.name() for h in hyponyms[:5]]})输出示例Hypernyms of dog: [canine.n.02, domestic_animal.n.01] Hyponyms of dog: [puppy.n.01, great_dane.n.01, ...]2.2 整体-部分关系挖掘meronym部分关系和holonym整体关系揭示物体组成tree wn.synset(tree.n.01) # 获取组成部分 parts tree.part_meronyms() print(fParts of a tree: {[p.name() for p in parts]})可能输出树根、树枝等组成部分。2.3 动词关系图谱动词间的继承关系体现动作细化communicate wn.synset(communicate.v.01) # 查看更具体的通信方式 specific_actions communicate.hyponyms() print(fSpecific communication methods: {[a.name() for a in specific_actions]})3. 语义相似度计算实战3.1 路径相似度Path Similarity基于节点间最短路径计算相似度car wn.synset(car.n.01) automobile wn.synset(automobile.n.01) bicycle wn.synset(bicycle.n.01) print(fcar-automobile: {car.path_similarity(automobile):.3f}) print(fcar-bicycle: {car.path_similarity(bicycle):.3f})输出示例car-automobile: 1.0 car-bicycle: 0.3333.2 Wu-Palmer相似度考虑概念在层次结构中的深度print(fWUP car-automobile: {car.wup_similarity(automobile):.3f}) print(fWUP car-bicycle: {car.wup_similarity(bicycle):.3f})3.3 相似度算法对比算法原理适用场景计算复杂度Path Similarity最短路径倒数通用比较O(n)Wu-Palmer深度加权路径层级结构O(n)Leacock-Chodorow对数缩放路径深层网络O(n)Resnik信息内容需要语料库O(n²)4. 实际应用案例4.1 词义消歧系统结合上下文选择最合适的词义from nltk import word_tokenize def disambiguate(word, context, posNone): context_words set(word_tokenize(context.lower())) best_syn None max_overlap -1 for syn in wn.synsets(word, pospos): # 获取同义词集的定义和例句词汇 signature set(word_tokenize(syn.definition().lower())) for example in syn.examples(): signature.update(word_tokenize(example.lower())) # 计算上下文重叠度 overlap len(context_words signature) if overlap max_overlap: max_overlap overlap best_syn syn return best_syn # 测试不同上下文 context1 I parked my car in the garage print(disambiguate(car, context1).definition()) context2 The trains dining car was crowded print(disambiguate(car, context2).definition())4.2 智能问答系统增强利用语义关系扩展查询def expand_query(term): expansions set() for syn in wn.synsets(term): # 添加同义词 expansions.update(syn.lemma_names()) # 添加上位词 for hyper in syn.hypernyms(): expansions.update(hyper.lemma_names()) # 添加下位词前3个 for hypo in syn.hyponyms()[:3]: expansions.update(hypo.lemma_names()) return expansions print(expand_query(vehicle))4.3 文本相似度计算组合WordNet与词向量方法from sklearn.metrics.pairwise import cosine_similarity import numpy as np def hybrid_similarity(text1, text2, w2v_model, alpha0.5): # 词向量相似度 vec1 average_vectors(text1, w2v_model) vec2 average_vectors(text2, w2v_model) w2v_sim cosine_similarity([vec1], [vec2])[0][0] # WordNet语义相似度 words1 set(word_tokenize(text1.lower())) words2 set(word_tokenize(text2.lower())) wn_sim 0 count 0 for w1 in words1: for w2 in words2: synsets1 wn.synsets(w1) synsets2 wn.synsets(w2) if synsets1 and synsets2: max_sim max( s1.path_similarity(s2) for s1 in synsets1 for s2 in synsets2 if s1.path_similarity(s2) is not None ) if max_sim: wn_sim max_sim count 1 wn_sim wn_sim / count if count 0 else 0 return alpha * w2v_sim (1-alpha) * wn_sim