python - 将多个数组值匹配到csv文件中的行慢
问题描述
我有一个 numpy 数组,由大约 1200 个数组组成,每个数组包含 10 个值。np.shape = 1200, 10。每个元素的值介于 0 到 570 万之间。
接下来我有一个包含 3800 行的 .csv 文件。每行包含 2 个值。第一个值指示范围,第二个值是标识符。.csv 文件的第一行和最后 5 行:
509,47222
1425,47220
2404,47219
4033,47218
6897,47202
...,...
...,...
...,...
5793850,211
5794901,186
5795820,181
5796176,43
5796467,33
第一列上升,直到达到 570 万。对于 numpy 数组中的每个值,我想检查 .csv 文件的第一列。例如,我有值3333
,这意味着属于的标识符3333
是47218
。每行表示从前一行的第一列到该行的第一列,例如:2404 - 4033
标识符是47218
。
现在我想获取 numpy 数组中每个值的标识符,然后我想保护标识符以及在 numpy 数组中找到该标识符的频率。这意味着我需要在 12000 行的 csv 文件上循环 3800 次,然后 ++ 一个整数。这个过程大约需要 30 秒,这太长了。
这是我目前使用的代码:
numpy_file = np.fromfile(filename, dtype=np.int32)
#some code to format numpy_file correctly
with open('/identifer_file.csv') as read_file:
csv_reader = csv.reader(read_file, delimiter=',')
csv_reader = list(csv_reader)
identifier_dict = {}
for numpy_array in numpy_file:
for numpy_value in numpy_array:
#there are 12000 numpy_value in numpy_file
for row in csv_reader:
last_identifier = 0
if numpy_value <= int(row[0]):
last_identifier = int(row[1])
#adding the frequency of the identifier in numpy_file to a dict
if last_identifier in identifier_dict:
identifier_dict[last_identifier] += 1
else:
identifier_dict[last_identifier] = 1
else:
continue
break
for x, y in identifier_dict.items():
if(y > 40):
print("identifier: {} amount of times found: {}".format(x, y))
我应该实施什么算法来加快这个过程?
编辑 我尝试将 numpy 数组折叠为一维数组,因此它有 12000 个值。这对速度没有实际影响。最新测试是 33 秒
解决方案
设置:
import numpy as np
import collections
np.random.seed(100)
numpy_file = np.random.randint(0, 5700000, (1200,10))
#'''range, identifier'''
read_file = io.StringIO('''509,47222
1425,47220
2404,47219
4033,47218
6897,47202
5793850,211
5794901,186
5795820,181
5796176,43
5796467,33''')
csv_reader = csv.reader(read_file, delimiter=',')
csv_reader = list(csv_reader)
# your example code put in a function and adapted for the setup above
def original(numpy_file,csv_reader):
identifier_dict = {}
for numpy_array in numpy_file:
for numpy_value in numpy_array:
#there are 12000 numpy_value in numpy_file
for row in csv_reader:
last_identifier = 0
if numpy_value <= int(row[0]):
last_identifier = int(row[1])
#adding the frequency of the identifier in numpy_file to a dict
if last_identifier in identifier_dict:
identifier_dict[last_identifier] += 1
else:
identifier_dict[last_identifier] = 1
else:
continue
break
# for x, y in identifier_dict.items():
# if(y > 40):
# print("identifier: {} amount of times found: {}".format(x, y))
return identifier_dict
三个解决方案,每个解决方案矢量化一些操作。第一个函数消耗的内存最少,最后一个函数消耗的内存最多。
def first(numpy_file,r):
'''compare each value in the array to the entire first column of the csv'''
alternate = collections.defaultdict(int)
for value in np.nditer(numpy_file):
comparison = value < r[:,0]
identifier = r[:,1][comparison.argmax()]
alternate[identifier] += 1
return alternate
def second(numpy_file,r):
'''compare each row of the array to the first column of csv'''
alternate = collections.defaultdict(int)
for row in numpy_file:
comparison = row[...,None] < r[:,0]
indices = comparison.argmax(-1)
id_s = r[:,1][indices]
for thing in id_s:
#adding the frequency of the identifier in numpy_file to a dict
alternate[thing] += 1
return alternate
def third(numpy_file,r):
'''compare the whole array to the first column of csv'''
alternate = collections.defaultdict(int)
other = collections.Counter()
comparison = numpy_file[...,None] < r[:,0]
indices = comparison.argmax(-1)
id_s = r[:,1][indices]
other = collections.Counter(map(int,np.nditer(id_s)))
return other
这些函数需要将 csv 文件读入一个 numpy 数组:
read_file.seek(0) #io.StringIO object from setup
csv_reader = csv.reader(read_file, delimiter=',')
r = np.array([list(map(int,thing)) for thing in csv_reader])
one = first(numpy_file, r)
two = second(numpy_file,r)
three = third(numpy_file,r)
assert zero == one
assert zero == two
assert zero == three
推荐阅读
- java - 在正则表达式或 xpath 提取器中按文本查找选项值
- orbeon - 强制清空重复部分
- webrtc - 有什么方法可以合并多个 TURN 服务器?
- python - 根据键更新字典值列表
- google-chrome - Chrome DevTools 显示一个空的传出二进制 Web 套接字框架,当它不是真的为空时
- excel - 我无法运行我的 if 语句和案例
- jasper-reports - 如何在 Jaspersoft Studio 中传递数组/集合类型的参数并在查询时使用它?
- r - 如何修复 pandoc 转换失败并出现错误 61
- c# - 如果 Apllication.Current 为 NULL,如何获取所有活动 wpf 窗口(多线程)的集合?
- python - Keras保存模型终止python