时间戳

首页 > 解决方案 > Python数据框从列表中删除句子编号

python - Python数据框从列表中删除句子编号

问题描述

我在数据框中有一列（相当长的）文本，对于每个文本，我想删除的句子索引列表。当我将文本拆分为句子时，Spacy 会生成句子索引。请考虑以下示例：

import pandas as pd
import spacy
nlp = spacy.load('en_core_web_sm')

data = {'text': ['I am A. I am 30 years old. I live in NY.','I am B. I am 25 years old. I live in SD.','I am C. I am 30 years old. I live in TX.'], 'todel': [[1, 2], [1], [1, 2]]}

df = pd.DataFrame(data)

def get_sentences(text):
    text_clean = nlp(text)
    sentences = text_clean.sents
    sents_list = []
    for sentence in sentences:
        sents_list.append(str(sentence))
    return sents_list

df['text'] = df['text'].apply(get_sentences)

print(df)

这给出了以下内容：

                                           text   todel
0  [I am A., I am 30 years old., I live in NY.]  [1, 2]
1   [I am B. I am 25 years old., I live in SD.]     [1]
2   [I am C. I am 30 years old., I live in TX.]  [1, 2]

todel知道我有一个非常大的数据集，每行要删除 50 多个句子，您将如何有效地删除存储的句子？

我的预期输出是：

                                  text   todel
0                      [I live in NY.]  [1, 2]
1  [I am 25 years old., I live in SD.]     [1]
2                      [I live in TX.]  [1, 2]

标签： pythonlistdataframeapplyspacy

解决方案

试试这个：

import pandas as pd

data = {'text': ['I am A. I am 30 years old. I live in NY.','I am B. I am 25 years old. I live in SD.','I am C. I am 30 years old. I live in TX.'], 'todel': [[1, 2], [1], [1, 2]]}

df = pd.DataFrame(data)

def fun(sen, lst):
    return  ('.'.join(s for idx, s in enumerate(sen.split('.')) if idx+1 not in lst))

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

输出：

                                text   todel
0                      I live in NY.  [1, 2]
1   I am 25 years old. I live in SD.     [1]
2                      I live in TX.  [1, 2]

编辑基于编辑的问题：

如果df['text']你不需要拆分的句子列表，你可以试试这个：

data = {'text': [['I am A.', 'I am 30 years old.', 'I live in NY.'], 
                 ['I am B.', 'I am 25 years old.', 'I live in SD.'],
                 ['I am C.','I am 30 years old.',' I live in TX.']], 'todel': [[1, 2], [1], [1, 2]]}
df = pd.DataFrame(data)
#                                           text     todel
# 0   [I am A., I am 30 years old., I live in NY.]  [1, 2]
# 1   [I am B., I am 25 years old., I live in SD.]     [1]
# 2  [I am C., I am 30 years old.,  I live in TX.]  [1, 2]

def fun(sen, lst):
    return  [s for idx , s in enumerate(sen) if not idx+1 in lst]

df['text'] = df.apply(lambda row : fun(row['text'],row['todel']), axis=1)

print(df)

输出：

                                  text   todel
0                      [I live in NY.]  [1, 2]
1  [I am 25 years old., I live in SD.]     [1]
2                     [ I live in TX.]  [1, 2]

推荐阅读