时间戳

首页 > 解决方案 > 如何从 spaCy 标记重建原始文本，即使在复杂的空白和标点符号的情况下

nlp - 如何从 spaCy 标记重建原始文本，即使在复杂的空白和标点符号的情况下

问题描述

' '.join(token_list) 在连续有多个空格和标点符号的情况下不会重建原始文本。

例如：

from spacy.tokenizer import Tokenizer
from spacy.lang.en import English

nlp = English()
# Create a blank Tokenizer with just the English vocab
tokenizerSpaCy = Tokenizer(nlp.vocab)

context_text = 'this    is a     test \n \n \t\t test for    \n testing  -  ./l \t'

contextSpaCyToksSpaCyObj = tokenizerSpaCy(context_text)
spaCy_toks = [i.text for i in contextSpaCyToksSpaCyObj]

reconstruct = ' '.join(spaCy_toks)
reconstruct == context_text

>False

是否有从 spaCy 令牌重建原始文本的既定方法？

已建立的答案应与此边缘案例文本一起使用（您可以通过单击“改进此问题”按钮直接获取来源）

" UNCLASSIFIED U.S. Department of State Case No. F-2014-20439 Doc No. C05795279 Date: 01/07/2016\n\n\n RELEASE IN PART\n B5, B6\n\n\n\n\nFrom: H <hrod17@clintonemail.com>\nSent: Monday, July 23, 2012 7:26 AM\nTo: 'millscd @state.gov'\nCc: 'DanielJJ@state.gov.; 'hanleymr@state.gov'\nSubject Re: S speech this morning\n\n\n\n Waiting to hear if Monica can come by and pick up at 8 to take to Josh. If I don't hear from her, can you send B5\nsomeone else?\n\n Original Message ----\nFrom: Mills, Cheryl D [MillsCD@state.gov]\nSent: Monday, July 23, 2012 07:23 AM\nTo: H\nCc: Daniel, Joshua J <Daniel1.1@state.gov>\nSubject: FW: S speech this morning\n\nSee below\n\n B5\n\ncdm\n\n Original Message\nFrom: Shah, Rajiv (AID/A) B6\nSent: Monday, July 23, 2012 7:19 AM\nTo: Mills, Cheryl D\nCc: Daniel, Joshua.'\nSubject: S speech this morning\n\nHi cheryl,\n\nI look fwd to attending the speech this morning.\n\nI had one last minute request - I understand that in the final version there is no reference to the child survival call to\naction, but their is a reference to family planning efforts. Could you and josh try to make sure there is some specific\nreference to the call to action?\n\nAlso, in terms of acknowledgements it would be good to note torn friedan's leadership as everyone is sensitive to our ghi\ntransition and we want to continue to send the usaid-pepfar-cdc working together public message. I don't know if he is\nthere, but wanted to flag.\n\nLook forward to it.\n\nRaj\n\n\n\n\n UNCLASSIFIED U.S. Department of State Case No. F-2014-20439 Doc No. C05795279 Date: 01/07/2016\n\x0c"

标签： nlpspacy

解决方案

您可以通过更改代码中的两行来轻松完成此操作：

spaCy_toks = [i.text + i.whitespace_ for i in contextSpaCyToksSpaCyObj]
reconstruct = ''.join(spaCy_toks)

基本上，spaCy 中的每个标记都知道它后面是否有空格。所以你打电话token.whitespace_而不是默认加入他们的空间。

推荐阅读