python - 多处理：无法腌制 _thread.RLock 对象

Question

在 Flask 应用程序中，我有一个在 for 循环中调用的函数，我想离开执行该进程，同时继续执行以下进程，直到完成for循环。

这是非常基本的场景的片段：

def get_most_similar_names(self):
    original_name = 'ABC ELEMENTARY EDUCATION'
    name_list = ['ABC ELEMENTARY SCHOOL', 'ABC KINDERGARDEN', 'ABC MIDDLE SCHOOL', 'ABC SOLUTIONS']
    vector1 = self.text_to_vector(original_name)
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
    jobs = []
    for name in name_list:
        print('vector1: ', vector1, "type: ", type(vector1))
        print('name: ', name, " type: ", type(name))
        print("return_dict: ", return_dict, "type: ", type(return_dict))
        # For testing
        a = dill.pickles(vector1)
        b = dill.pickles(name)
        c = dill.pickles(return_dict)

        p = multiprocessing.Process(target=self.get_score, args=(vector1, name, return_dict))
        jobs.append(p)
        p.start()

    for proc in jobs:
        proc.join()

    print(return_dict.values())


def get_score(self, vec1, n, return_d):
    vec2 = self.text_to_vector(n)
    print("in get_cosine function")
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])
    sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])

    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        cosine = 0.0
    else:
        cosine = float(numerator) / denominator

    return_d[n] = cosine

    print("return_dict:", return_d)

    return return_d

目的是执行 get_score 函数，该函数接收 vector1、名称和共享变量以将结果存储在字典中。但是，执行start()时出现以下错误：

File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.RLock objects

我假设这些参数是不可腌制的，所以我测试了它们，但它们是可腌制的。我读过 logger 会产生这个错误，我在其他模块中发现 current_app.logger 执行。

你知道其他因素会影响执行start()吗？

标签： pythonmultiprocessingpython-3.6pickledill