时间戳

首页 > 解决方案 > 从文本文件中的单词中去除特定的标点符号

python - 从文本文件中的单词中去除特定的标点符号

问题描述

我需要编写一个函数get_words_from_file(filename)，它返回一个小写单词列表。您的函数应该只处理开始和结束标记线之间的线。单词的顺序应与它们在文件中出现的顺序相同。这是一个示例文本文件：baboosh.txt：

*** START OF TEST CASE ***
......list of sentences here.....
*** END OF TEST CASE ***
This is after the end and should be ignored too.

这是我想出的：

import re
from string import punctuation

def stripped_lines(lines):
    for line in lines:
        stripped_line = line.rstrip('\n')
        yield stripped_line


def lines_from_file(fname):
    with open(fname, 'rt') as flines:
        for line in stripped_lines(flines):
            yield line


def is_marker_line(line, start='***', end='***'):
    '''
    Marker lines start and end with the given strings, which may not
    overlap. (A line containing just '***' is not a valid marker line.)
    '''
    min_len = len(start) + len(end)
    if len(line) < min_len:
        return False
    return line.startswith(start) and line.endswith(end)

def advance_past_next_marker(lines):
    '''
    '''
    for line in lines:
        if is_marker_line(line):
            break


def lines_before_next_marker(lines):

    valid_lines = []
    for line in lines:
        if is_marker_line(line):
            break
        line.replace('"', '')
        valid_lines.append(line)


    for content_line in valid_lines:
        yield content_line


def lines_between_markers(lines):
    '''
    Yields the lines between the first two marker lines.
    '''
    it = iter(lines)
    advance_past_next_marker(it)
    for line in lines_before_next_marker(it):
        yield line


def words(lines):
    text = '\n'.join(lines).lower().split()
    return text


def get_words_from_file(fname):
    return words(lines_between_markers(lines_from_file(fname)))

#This is the test code that must be executed
filename = "baboosh.txt"
words = get_words_from_file(filename)
print(filename, "loaded ok.")
print("{} valid words found.".format(len(words)))
print("Valid word list:")
for word in words:
    print(word)

我的输出

我得到了正确的单词列表。但是在打印时，我会得到标点符号，例如冒号、分号和句号。我不知道如何摆脱这些。

我怎样才能做到这一点？

标签： pythonpython-3.x

解决方案

使用re.split而不是str.split. 如果您像这样设置已编译的正则表达式：

splitter = re.compile('[ ;:".]')

然后你可以用以下方式分割你的行：

word_list = splitter.split(line)

这将返回没有标点符号的单词。

推荐阅读