时间戳

首页 > 解决方案 > Joblib 在 2x n_jobs 后崩溃

python - Joblib 在 2x n_jobs 后崩溃

问题描述

Joblib 因错误而崩溃

  Parallel(n_jobs=-1, prefer="threads", verbose=10)(
  File "/home/developer/.local/lib/python3.8/site-packages/joblib/parallel.py", line 1054, in __call__
    self.retrieve()
  File "/home/developer/.local/lib/python3.8/site-packages/joblib/parallel.py", line 933, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/developer/.local/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "/home/developer/.local/lib/python3.8/site-packages/joblib/parallel.py", line 262, in __call__
    return [func(*args, **kwargs)
  File "/home/developer/.local/lib/python3.8/site-packages/joblib/parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
TypeError: cannot unpack non-iterable function object

在这段代码上（一些名称被更改以隐藏信息）

    with open(inputFile) as file:
        csv_reader = csv.DictReader(
            file, fieldnames=["Header1", "Header2"])
        Parallel(n_jobs=3, prefer="threads", verbose=10)(
            delayed(pullSummaryData(row["Header1"]))
            for row in csv_reader
        )

有趣的部分是它总是在pullSummaryData准确调用 2*后崩溃n_jobs。如果n_jobs=3,pullSummaryData将在崩溃前被调用 6 次。

作业库 v1.0.1

csv v1.0

Python v3.8.5

标签： pythonpython-3.xjoblib

解决方案

尝试更改delayed(pullSummaryData(row["Header1"]))为delayed(pullSummaryData)(row["Header1"]).

参考：文件

根据user696969在原始帖子下的评论回答。

推荐阅读