时间戳

首页 > 解决方案 > 获取 TypeError：无法在 Flask 中腌制环境对象

python - 获取 TypeError：无法在 Flask 中腌制环境对象

问题描述

我正在尝试在烧瓶应用程序中设置 multiprocessing.Pool，但似乎遇到了问题。

我有一个调用服务功能的蓝图。

# blueprint.py
@bp.route(...)
def my_bp_route(...):
    service = get_service_instance()
    service.do_stuff(...)

# service.py
import multiprocessing as mp

def foo(self, ...):
    # do stuff

def do_stuff(self, ...):
    some_tuple_pairs = [
        (CustomType.TypeA.value, IdStr.TypeA.value),
        (CustomType.TypeB.value, IdStr.TypeB.value),
    ]

    cropbox = (5, 5)

    # LTChar is from pdfminer
    self.do_mp_concurrent(some_tuple_pair, [LTChar(), LTChar(), ...], cropbox, [{'key1': 1, ...}, ...])

def do_mp_concurrent(
    self,
    types_to_check: List[Tuple[str, str]],
    pdf_objs: List[LTChar],
    cropbox: Tuple[int, int],
    some_dicts: List[dict]
):
    with mp.Pool(processes=3) as pool:
        results = pool.starmap(
            self.foo,
            [(cus_type, id_str, pdf_objs, cropbox, some_dicts) for cus_type, id_str in types_to_check]
        ...

这不断给我一些环境酸洗错误：

    results = pool.starmap(
File "/usr/lib64/python3.8/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/usr/lib64/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
File "/usr/lib64/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
File "/usr/lib64/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib64/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'Environment' object

我认为 pdfminer 的 LTChar 类是问题所在，所以我改成了非常简单的东西，但仍然出现同样的错误。

def foo(self, one, true):
    print(one, two)

def do_mp_concurrent(
    self,
    types_to_check: List[Tuple[str, str]],
    pdf_objs: List[LTChar],
    cropbox: Tuple[int, int],
    some_dicts: List[dict]
):
    with mp.Pool(processes=3) as pool:
        results = pool.starmap(
            self.foo,
            [(1, 2)]
        ...

奇怪的是，在 Jupyter Notebook 上也有类似的东西。

import multiprocessing as mp

class FooClass:
    def foo(self, one, two):
        print(one, two)

    def _do_mp(self):
        with mp.Pool(processes=3) as pool:
            results = pool.starmap(
                self.foo,
                [(1, 2)]
            )
            print(results)
        
FooClass()._do_mp()

标签： pythonflaskmultiprocessing

解决方案

我认为python在第一种情况下不能腌制self实例。当你有SimpleFoo课程时，简单的课程是可以选择的。

要解决此问题，请尝试将foo(self, one, true)方法转换为不使用的方法self。像这样：


def do_mp_concurrent(
    self,
    types_to_check: List[Tuple[str, str]],
    pdf_objs: List[LTChar],
    cropbox: Tuple[int, int],
    some_dicts: List[dict]
):

    def foo(...):
        # do stuff
        # make sure foo is not using any complicated instance.

    with mp.Pool(processes=3) as pool:
        results = pool.starmap(
            foo,
            [(cus_type, id_str, pdf_objs, cropbox, some_dicts) for cus_type, id_str in types_to_check]
        ...

如果您需要self在调用时使用某些值foo，则将这些值转换为可以腌制的值并将其传递给foo.

推荐阅读