python - Python多进程池的高效实现

Question

我有两个代码。一个是另一个的池化（多处理）版本。然而，即使是 1 个处理器的并行版本也需要很长时间，而串行版本则需要大约 15 秒才能完成。有人可以帮助加速第二个版本。

1. 串行

    import numpy as np, time
    def mapTo(d):   
        global tree
        for idx, item in enumerate(list(d), start=1):
            tree[str(item)].append(idx)

    data=np.random.randint(1,4, 20000000)
    tree = dict({"1":[],"2":[],"3":[]})
    s= time.perf_counter()
    mapTo(data)
    e = time.perf_counter()
    print("elapsed time:",e-s)

耗时：~15 秒

2. 平行

from multiprocessing import Manager, Pool
from functools import partial
import numpy as np
import time

def mapTo(i_d,tree):
    idx,item = i_d
    l = tree[str(item)]
    l.append(idx)
    tree[str(item)] = l

manager = Manager()
data    = np.random.randint(1,4, 20000000)
# sharedtree= manager.dict({"1":manager.list(),"2":manager.list(),"3":manager.list()})
sharedtree = manager.dict({"1":[],"2":[],"3":[]})
s= time.perf_counter()
with Pool(processes=1) as pool:
    pool.map(partial(mapTo, tree=sharedtree), list(enumerate(data,start=1)))
e = time.perf_counter()
print("elapsed time:",e-s)

标签： pythonmultiprocessing