python - Python Pool.Starmap 未在打印时终止或输出

Question

我尝试了几种不同的方式来执行Pool.starmap。我尝试了各种不同的建议和答案，但无济于事。下面是我尝试运行的代码示例，但是它被捕获并且永远不会终止。我在这里做错了什么？

旁注： 我在python版本3.9.8

if __name__ == '__main__':
with get_context("spawn").Pool() as p:
    tasks = [(1,1),(2,2),(3,3)]
    print(p.starmap(add,tasks))
    p.close()
    p.join()

标签： pythonpython-3.xmultiprocessingpoolstarmap

Answer 1

python 中的多处理具有一些复杂性，您应该知道这使得它取决于您如何运行脚本以及您使用的操作系统和 python 版本。

我经常看到的一个大问题是 Jupyter 和其他“笔记本”风格的 Python 环境并不总是能很好地处理多处理。从技术上讲，有一些方法可以解决这个问题，但我通常只是建议从更正常的系统终端执行代码。共同点是“交互式”解释器不能很好地工作，因为需要有一个“主”文件，而在交互模式下没有文件；它只是等待用户输入。

我无法确切知道您的问题是什么，因为您没有提供所有代码、您正在使用的操作系统以及您正在使用的 IDE，但我至少可以让您继续工作（在我的设置中） ） 例子。（Windows 10；python 3.9；具有运行设置的 Spyder IDE -> 在外部系统终端中执行）

import multiprocessing as mp

def add(a, b): #I'm assuming your "add" function looks a bit like this...
    return a+b

if __name__ == "__main__": 
    #this is critical when using "spawn" so code doesn't run when the file is imported
    #you should only define functions, classes, and static data outside this (constants)
    #most critically, it shouldn't be possible for a new child process to start outside this
    
    ctx = mp.get_context("spawn")
    #This is the only context available on windows, and the default for MacOS since python 3.8.
    #  Contexts are an important topic somewhat unique to python multiprocessing, and you should
    #  absolutely do some additional reading about "spawn" vs "fork". tldr; "spawn" starts a new
    #  process with no knowledge of the old one, and must `import` everything from __main__. 
    #  "fork" on the other hand copies the existing process and all its memory before branching. This is
    #  faster than re-starting the interpreter, and re-importing everything, but sometimes things
    #  get copied that shouldn't, and other things that should get copied don't.
    with ctx.Pool() as p: 
        #using `with` automatically shuts down the pool (forcibly) at the end of the block so you don't have to call `close` or `join`.
        #  It was also pointed out that due to the forcible shutdown, async calls like `map_async` may not finish unless you wait for the results
        #  before the end of the `with` block. `starmap` already waits for the results in this case however, so extra waiting is not needed.
        tasks = [(1,1),(2,2),(3,3)]
        print(p.starmap(add, tasks))