python - Using multiprocessing pool in Python
问题描述
Can someone point out what is wrong with this code snippet. It does not give any results .
import multiprocessing
results = []
def log_results(result):
results.append(result)
def multiply(x, y):
print(f"Gets here for process name {multiprocessing.current_process().name()}")
return x * y
if __name__ == "__main__":
pool = multiprocessing.Pool()
numbers = [(1,1), (2,2), (3,3)]
for x, y in numbers:
print (f"Checking x {x} and y {y}")
pool.apply_async(multiply, (x, y), callback=log_results)
pool.close()
pool.join()
print(results)
results is an empty list which should not be in this case right ? I have used apply_async and map_async . Both do not give the right output. Can someone please help me here
解决方案
Edit: You made an edit to your code so now my answer below is out of date. The only two things I think need doing are:
- add an
error_callback
because I still think you need to ensure that the pool as written does not fail silently by default. - rewrite
multiprocessing.current_process().name()
asmultiprocessing.current_process().name
.
So:
import multiprocessing
results = []
def log_results(result):
results.append(result)
def log_e(e):
print(e)
def multiply(x, y):
print(f"Gets here for process name {multiprocessing.current_process().name}")
return x * y
pool = multiprocessing.Pool()
numbers = [(1,1), (2,2), (3,3)]
for x, y in numbers:
print (f"Checking x {x} and y {y}")
pool.apply_async(multiply, (x, y), callback=log_results,
error_callback=log_e)
pool.close()
pool.join()
print(results)
Old answer
This drove me crazy for a moment but then it made sense.
If I run it with multiply
changed like this:
def multiply(nums):
print("print")
return nums[0] * nums[1]
It runs fine. You said in the comments "I do not think the function multiply
is called in the first place." This is because there is a callback
specified but no error_callback
specified. The result of omitting an error callback is that your script is failing silently.
You could check this with:
import multiprocessing
results = []
def log_results(result):
print(result)
def log_e(e):
print(e)
def multiply(x, y):
print(f"Gets here for process name {multiprocessing.current_process().name()}")
return x * y
pool = multiprocessing.Pool()
numbers = [[1,1], [2,2], [3,3]]
mapResult = pool.map_async(multiply, numbers, callback=log_results,
error_callback=log_e)
pool.close()
pool.join()
Which gives:
multiply() missing 1 required positional argument: 'y'
And with multiply
like so:
def multiply(nums):
return nums[0] * nums[1]
It then returns [1, 4, 9]
PS I am running Python 3.6.7
推荐阅读
- r - 如何使用 depmixS4 包比较 R 中隐马尔可夫模型中的状态?
- cmake - 启用 CMake 的任何更新?
- ldap - 如何返回嵌套组中的所有用户?
- c# - 如何从其他位置在 Razor 页面中添加模型
- ssl - 具有不同组织的相同通配符 SSL 证书
- python - 如何在小提琴图中复制 kdeplot 的阴影?
- react-native - Mathjax 离线反应原生
- angular - angular 6 组件属性使用
- python - 使用 pandas 迭代数据框中的某些元素时填充新行
- macos - Logstash 6.7.0 无法在 macOS Mojave (10.14.4) 上启动并出现错误“错误:权限被拒绝 - 权限被拒绝”