时间戳

首页 > 解决方案 > error with module multiprocessing under python3.8

python - error with module multiprocessing under python3.8

问题描述

I had a script that was multiprocessing fine until today. To reproduce the problem, I simplified the function that I parallelized with the one shown below:

    from multiprocessing import Process, Queue
    import random

    def rand_num():
        num = random.random()
        print(num)

    if __name__ == "__main__":
        queue = Queue()

        processes = [Process(target=rand_num, args=()) for x in range(4)]

        for p in processes:
            p.start()

        for p in processes:
            p.join()

that renders the exact same error message (repeated 4 times, which I omitted repeating for readability):

    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 262, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 232, in _get_code_from_file
    with io.open_code(fname) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/myUserName/<stdin>'

I don't know where to start debugging this error. I'm running python3.8 under mac os Catalina (homebrew install). Please help.

标签： pythonpython-3.xpython-multiprocessing

解决方案

从 Python 3.7 升级到 3.8 时，我遇到了同样的问题。特别是现在在 OSX 10.15.6 上运行 3.8.6，Python 由 pyenv 安装。

Darkonaut 的建议帮助解决了这个问题，但它并不那么明显，所以让我在这里重新表述一下：

默认情况下，MacOS 上的 Python 3.8 现在使用spawn而不是fork作为新进程的启动方法。尝试

multiprocessing.set_start_method("fork")

显然，spawn 的行为是错误的，如下简单示例所示：

import multiprocessing

def parallel_function(x):
    print("Function called with", x)

def test_pool():
    print("Running test_pool")
    with multiprocessing.Pool(4) as pool:
        pool.map(parallel_function, range(10))

print("Starting the test")
test_pool()

这会产生以下输出：

Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path

因此，池不会正确创建工作程序，而是尝试在每个生成的进程中运行整个脚本。

推荐阅读