时间戳

首页 > 解决方案 > 如何优雅地从字符串中删除长度为 n 的椭圆（NLP with spacy）？

python - 如何优雅地从字符串中删除长度为 n 的椭圆（NLP with spacy）？

问题描述

我目前正在对这个垃圾短信数据集进行数据清理。这些短信中有很多省略号，例如：

mystr = 'Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...'

如您所见，有 2 个句点 ( ..) 或 3 个句点 ( ...)的椭圆

我最初的解决方案是编写一个函数spacy_tokenizer来标记我的字符串，删除停用词和标点符号：

import spacy
nlp = spacy.load('en_core_web_sm')
from nltk.corpus import stopwords

stopWords = set(stopwords.words('english'))
print(stopWords)

import string
punctuations = string.punctuation

def spacy_tokenizer(sentence):
    # Create token object
    mytokens = nlp(sentence)
    # Case normalization and Lemmatization
    mytokens = [ word.lemma_.lower() if word.lemma_ != "-PRON-" else word.lower_ for word in mytokens ]
    # Remove stop words and punctuations
    mytokens = [ word.strip(".") for word in mytokens if word not in stopWords and word not in punctuations ]
    # return preprocessed list of tokens
    return mytokens

然而，这个函数并没有去掉省略号

IN: print(spacy_tokenizer(mystr))
OUT: ['go', 'jurong', 'point', 'crazy', '', 'available', 'bugis', 'n', 'great', 'world', 'la', 'e', 'buffet', '', 'cine', 'get', 'amore', 'wat', '']

如您所见，有些标记len(token) = 0显示为''

我的解决方法是添加另一个列表理解spacy_tokenizer，看起来像这样：[ word for word in mytokens if len(word) > 0]

def spacy_tokenizer(sentence):
    # Create token object
    mytokens = nlp(sentence)
    # Case normalization and Lemmatization
    mytokens = [ word.lemma_.lower() if word.lemma_ != "-PRON-" else word.lower_ for word in mytokens ]
    # Remove stop words and punctuations
    mytokens = [ word.strip(".") for word in mytokens if word not in stopWords and word not in punctuations ]
    # remove empty strings
    mytokens = [ word for word in mytokens if len(word) > 0]
    return mytokens

IN: print(spacy_tokenizer(mystr))
OUT: ['go', 'jurong', 'point', 'crazy', 'available', 'bugis', 'n', 'great', 'world', 'la', 'e', 'buffet', 'cine', 'get', 'amore', 'wat']

所以新函数给出了预期的结果，但这不是我认为的最优雅的解决方案。有没有人有任何替代的想法？

标签： pythonnlpspacy

解决方案

这将删除 2 或 3 个周期的椭圆：

import re

regex = r"[.]{2,3}"
test_str = "Go until jurong point, crazy.. Available only. in bugis n great world la e buffet... Cine there got amore wat..."
subst = ""

result = re.sub(regex, subst, test_str)

if result:
    print (result)

如果你愿意，你也可以在这里玩。

推荐阅读