python - 将文本文件解析为表格数据以进行处理
问题描述
手头的问题是使用python以表格形式解析特定数据。一小部分数据如下所示
Statistics indicator:0x222235
number of records = 3
records[0]
value one = 2
value two = 5
blocks = 2
block[0] {
some irrelevant data....
value three = 4 bytes
}
block[1]{
some irrelevant data...
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 4 bytes
}
records[2]
value one = 7
value two = 6
blocks = 2
block[0] {
some irrelevant data....
value three = 3 bytes
}
block[1]{
some irrelevant data...
value three = 4 bytes
}
Statistics indicator:0x135256
number of records = 2
records[0]
value one = 4
value two = 8
blocks = 1
block[0] {
some irrelevant data....
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 3 bytes
}
如图所示,数据具有特定的模式。它在每个特定数据块的开头都有统计指示器。它有记录数字段来指示数据块的记录数。在每条记录中,“值一”和值二是不同的。然而,有几个“值三”取决于每条记录具有的块数,由“块”字段指示。这里的问题是将以下数据以表格形式排列,将与特定记录对应的所有值 3 相加。
决赛桌应如下所示:
价值一价值二价值三
2 5 10
3 5 4
7 6 7
4 8 6
3 5 3
我正在考虑的方法是首先找到“统计指标”如果我找到“统计指标”,我将查找与每条记录相对应的记录和块的数量,以遍历块并将对应于相似值一和值二的值三求和.
这是我试图提取值一值二和值三的代码。我还没有深入研究求和值三。
import re
import pandas as pd
val_dict = { 'value_one':re.compile(r'value one = (?P<value_one>.*)\n'),
'value_two':re.compile(r'value two = (?P<value_two>.*)\n'),
'value_three':re.compile(r'value three = (?P<value_three>.*)\n')}
def _parse_line(line):
for key, val in val_dict.items():
match = val.search(line)
if match:
return key, match
# if there are no matches
return None, None
def parse_file(filepath):
data = []
with open(filepath, 'r') as file_object:
row = {} # prepare an empty row
for line in file_object:
key, match = _parse_line(line)
# search for keys in the line
if key == 'value_one':
value_one = match.group('value_one')
value_one = int(value_one)
if 'value one' in row: # we always have a full row
data.append(row) # append it to the data liest
row = {} # and reset it
row['value one'] = value_one # we have a match: store the value in row
if key == 'value_two':
value_two = match.group('value_two')
value_two = int(value_two)
if 'value two' in row:
data.append(row)
row = {}
row['value two'] = value_two
if key == 'value_three':
value_three = match.group('value_three')
value_three = int(value_three)
if 'value three' in row:
data.append(row)
row = {}
row['value three'] = value_three
if row != {}: # do not forget the last row
data.append(row)
data = pd.DataFrame(data)
return data
if __name__ == '__main__':
filepath = 'test3.txt'
data = parse_file(filepath)
解决方案
在这里,也许我们不想使用正则表达式。但是,如果我们可能想要这样做,我们可以将属性名称设置为左边界,并收集我们想要的数字,可能使用类似于以下的表达式:
value\s+(one|two|three)\s+=\s+([0-9]+)
然后,我们的问题的其余部分可以编写脚本。如果有必要,我们还可以为表达式添加更多边界。
演示
测试
# coding=utf8
# the above tag defines encoding for this document and is for Python 2.x compatibility
import re
regex = r"value\s+(one|two|three)\s+=\s+([0-9]+)"
test_str = ("Statistics indicator:0x222235\n\n"
"number of records = 3\n\n"
"records[0]\n\n"
"value one = 2\n\n"
"value two = 5\n\n"
"blocks = 2\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 4 bytes\n\n"
"}\n\n"
"block[1]{\n\n"
"some irrelevant data...\n\n"
"value three = 6 bytes\n\n"
"}\n\n"
"records[1]\n\n"
"value one = 3\n\n"
"value two = 5\n\n"
"blocks = 1\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 4 bytes\n\n"
"}\n\n"
"records[2]\n\n"
"value one = 7\n\n"
"value two = 6\n\n"
"blocks = 2\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 3 bytes\n\n"
"}\n\n"
"block[1]{\n\n"
"some irrelevant data...\n\n"
"value three = 4 bytes\n\n"
"}\n\n"
"Statistics indicator:0x135256\n\n"
"number of records = 2\n\n"
"records[0]\n\n"
"value one = 4\n\n"
"value two = 8\n\n"
"blocks = 1\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 6 bytes\n\n"
"}\n\n"
"records[1]\n\n"
"value one = 3\n\n"
"value two = 5\n\n"
"blocks = 1\n\n"
"block[0] {\n\n"
"some irrelevant data....\n\n"
"value three = 3 bytes\n\n"
"}")
matches = re.finditer(regex, test_str, re.MULTILINE)
for matchNum, match in enumerate(matches, start=1):
print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))
for groupNum in range(0, len(match.groups())):
groupNum = groupNum + 1
print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))
# Note: for Python 2.7 compatibility, use ur"" to prefix the regex and u"" to prefix the test string and substitution.
演示
const regex = /value\s+(one|two|three)\s+=\s+([0-9]+)/gm;
const str = `Statistics indicator:0x222235
number of records = 3
records[0]
value one = 2
value two = 5
blocks = 2
block[0] {
some irrelevant data....
value three = 4 bytes
}
block[1]{
some irrelevant data...
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 4 bytes
}
records[2]
value one = 7
value two = 6
blocks = 2
block[0] {
some irrelevant data....
value three = 3 bytes
}
block[1]{
some irrelevant data...
value three = 4 bytes
}
Statistics indicator:0x135256
number of records = 2
records[0]
value one = 4
value two = 8
blocks = 1
block[0] {
some irrelevant data....
value three = 6 bytes
}
records[1]
value one = 3
value two = 5
blocks = 1
block[0] {
some irrelevant data....
value three = 3 bytes
}`;
let m;
while ((m = regex.exec(str)) !== null) {
// This is necessary to avoid infinite loops with zero-width matches
if (m.index === regex.lastIndex) {
regex.lastIndex++;
}
// The result can be accessed through the `m`-variable.
m.forEach((match, groupIndex) => {
console.log(`Found match, group ${groupIndex}: ${match}`);
});
}
正则表达式
如果不需要此表达式,可以在regex101.com中对其进行修改或更改。
正则表达式电路
jex.im可视化正则表达式:
推荐阅读
- google-apps-script - 结合这两个脚本
- r - 在 R Leaflet 中改变标签颜色
- c# - dotnet publish -c Release 仍然会产生调试符号。为什么?
- javascript - 使用 datepicker 将输入设置为 AngularJS 1.5.8 中的任何字符串
- powerquery - 尝试拉入 Json 文件时出现 Token Comma Expected 错误
- ios - 无法在 iOS 13 上将视频上传到 Firebase 存储
- python - 得到结果“无”
- azure - Azure 路线图:AspCore 3.0
- fullpage.js - 整页 JS 动画锚:虚假故障
- java - 引起:org.hibernate.hql.internal.ast.QuerySyntaxException:意外令牌:订单靠近第 1 行,第 17 列,原因绝对不清楚