时间戳

首页 > 解决方案 > Pandas：如何删除行中的重复项并进行多个主题匹配

python - Pandas：如何删除行中的重复项并进行多个主题匹配

问题描述

我有以下数据框dfstart，其中第一列包含包含各种不同主题的不同评论。标签列包含与主题相关联的关键字。

在此处输入图像描述

使用第二个数据框matchlist

在此处输入图像描述

我想创建最终的数据框dffinal，对于每条评论，您都可以看到该评论中出现的标签和主题。我还希望标签每行只出现一次。

在此处输入图像描述

我尝试通过消除重复标签for loop

for label in matchlist['label']:
    if dfstart[label[n]] == dfstart[label[n-1]]:
        dfstart['label'] == np.nan

但是，这似乎不起作用。此外，我已经设法合并dfstart以matchlist在数据框中显示第一个主题。我使用的代码是

df2 = pd.merge(df, matchlist, on='label1')

当然，我可以继续重命名标签列matchlist并不断重复该过程，但这将花费很长时间并且效率不高，因为我的真实数据框比这个玩具示例大得多。所以我想知道是否有更优雅的方式来做到这一点。

以下是三个玩具数据框：

d = {'comment':["comment1","comment2","comment3"], 'label': ["boxing, election, rain", "boxing, boxing", "election, rain, election"]}

dfstart = pd.DataFrame(data=d)

dfstart[['label1','label2', 'label3']] = dfstart.label.str.split(",",expand=True,)

d3 = {'label':["boxing","election","rain"], 'topic': ["sport","politics","weather"]}

matchlist = pd.DataFrame(data=d3)

d2 = {'comment':["comment1","comment2","comment3"],'label': ["boxing, election, rain", "boxing, boxing", "election, rain, election"], 'label1':["boxing", "boxing", "election"], 'label2':["election", np.nan, "rain"], 'label3':["rain", np.nan, np.nan], 'topic1':["sports", "sports", "politics"], 'topic2':["politics", np.nan, "weather"], 'topic3':["weather", np.nan, np.nan]}

dffinal = pd.DataFrame(data=d2)

谢谢你的帮助！

标签： pythonpandasdataframenlp

解决方案

使用str.extractall代替，str.split这样您就可以一次获得所有匹配项，然后将结果展平并map到您的matchlist，最后concat全部放在一起：

d = {'comment':["comment1","comment2","comment3"],
     'label': ["boxing, election, rain", "boxing, boxing", "election, rain, election"]}

df = pd.DataFrame(d)

matchlist = pd.DataFrame({'label':["boxing","election","rain"], 'topic':["sport","politics","weather"]})

s = matchlist.set_index("label")["topic"]

found = (df["label"].str.extractall("|".join(f"(?P<label{num}>{i})" for num, i in enumerate(s.index, 1)))
                    .groupby(level=0).first())

print (pd.concat([df, found,
                  found.apply(lambda d: d.map(s))
                  .rename(columns={f"label{i+1}":f"topic{i+1}" for i in range(1, 4)})], axis=1) )

    comment                     label  label1    label2 label3 label1    topic2   topic3
0  comment1    boxing, election, rain  boxing  election   rain  sport  politics  weather
1  comment2            boxing, boxing  boxing       NaN    NaN  sport       NaN      NaN
2  comment3  election, rain, election     NaN  election   rain    NaN  politics  weather

推荐阅读