时间戳

首页 > 解决方案 > 从列表中删除密切匹配/相似的短语

python - 从列表中删除密切匹配/相似的短语

问题描述

我正在努力删除列表中的类似短语，但我遇到了一个小障碍。

我有句子和短语，短语都与句子有关。一个句子的所有短语都在一个列表中。

让短语列表为：p=[['This is great','is great','place for drinks','for drinks'],['Tonight is a good','good night','is a good','for movies']]

我希望我的输出是[['This is great','place for drinks'],['Tonight is a good','for movies']]

基本上，我想获得列表中所有最长的唯一短语。

我查看了fuzzywuzzy 库，但我无法找到一个好的解决方案。

这是我的代码：

def remove_dup(arr, threshold=80):
    ret_arr =[]
    for item in arr:
        if item[1]<threshold:
            ret_arr.append(item[0])
    return ret_arr

def find_important(sents=sents, phrase=phrase):

    import os, random
    from fuzzywuzzy import process, fuzz

    all_processed = [] #final array to be returned
    for i in range(len(sents)):

        new_arr = [] #reshaped phrases for a single sentence
        for item in phrase[i]:
            new_arr.append(item)

        new_arr.sort(reverse=True, key=lambda x : len(x)) #sort with highest length

        important = [] #array to store terms
        important = process.extractBests(new_arr[0], new_arr) #to get levenshtein distance matches
        to_proc = remove_dup(important) #remove_dup removes all relatively matching terms.
        to_proc.append(important[0][0]) #the term with highest match is obviously the important term.


        all_processed.append(to_proc) #add non duplicates to all_processed[]

    return all_processed

有人可以指出我缺少什么，或者有什么更好的方法吗？提前致谢！

标签： pythonliststring-comparison

解决方案

我会使用每个短语和所有其他短语之间的差异。如果一个短语与所有其他短语相比至少有一个不同的词，那么它是唯一的，应该保留。

我还使它对精确匹配和添加空格变得健壮

sentences = [['This is great','is great','place for drinks','for drinks'],
['Tonight is a good','good night','is a good','for movies'],
['Axe far his favorite brand for deodorant body spray',' Axe far his favorite brand for deodorant spray','Axe is']]

new_sentences = []
s = " "
for phrases in sentences :
    new_phrases = []
    phrases = [phrase.split() for phrase in phrases]
    for i in range(len(phrases)) :
        phrase = phrases[i]
        if all([len(set(phrase).difference(phrases[j])) > 0 or i == j for j in range(len(phrases))]) :
            new_phrases.append(phrase)
    new_phrases = [s.join(phrase) for phrase in new_phrases]
    new_sentences.append(new_phrases)
print(new_sentences)

输出：

[['这很棒'，'喝酒的地方']，

['今晚好'，'晚安'，'电影']，

['Axe 是他最喜欢的除臭身体喷雾品牌'，'Axe is']]

推荐阅读