时间戳

首页 > 解决方案 > dask.protocol.core 中的 msgpack 序列化引发 TypeError：无法序列化“变量”对象

python - dask.protocol.core 中的 msgpack 序列化引发 TypeError：无法序列化“变量”对象

问题描述

我已经分发了 dask 作为工作流（完美）的更大努力的一部分，并且在我的 dask 工作人员中看到了一个错误。工人收到它的工作，但很快引发了这个异常：

TypeError: can not serialize 'Variable' object

完整的跟踪是：

distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 25, in dumps
    small_header, small_payload = dumps_msgpack(msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/opt/conda/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object
distributed.comm.utils - INFO - Unserializable Message: {'op': 'variable_get', 'timeout': 0, 'name': <distributed.variable.Variable object at 0x7fec9eafaef0>, 'client': 'Client-worker-365168fe-ccf9-11ea-800b-0242ac110004', 'reply': True}
distributed.comm.utils - ERROR - can not serialize 'Variable' object
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/comm/utils.py", line 35, in _to_frames
    msg, serializers=serializers, on_error=on_error, context=context
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 25, in dumps
    small_header, small_payload = dumps_msgpack(msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/opt/conda/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object

我把它归结为一个非常小的测试用例，我认为它具有相当的代表性。它提出了同样的问题：

from distributed.protocol.core import dumps_msgpack
from distributed.variable import Variable
from distributed.client import Client
client = Client(address='10.72.112.29:38786')
var = Variable(name='test', client=client)
msg = {'op': 'variable_get', 'timeout': 0, 'name': var, 'client': client.id, 'reply': True}
d = dumps_msgpack(msg)

此示例产生错误消息，但也显示了一些版本不匹配（我认为这不是问题）：

/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/distributed/client.py:1136: VersionMismatchWarning: Mismatched versions found
+---------+---------------+---------------+---------------+
| Package | client        | scheduler     | workers       |
+---------+---------------+---------------+---------------+
| numpy   | 1.19.0        | 1.18.1        | 1.18.1        |
| python  | 3.7.8.final.0 | 3.7.0.final.0 | 3.7.0.final.0 |
+---------+---------------+---------------+---------------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Traceback (most recent call last):
  File "<input>", line 7, in <module>
  File "/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object

我错过了一个包裹还是什么？我可以提供我的安装脚本等，但它们有点涉及 - 客户端是本地 python 环境，调度程序和工作人员是码头工人。

标签： pythondaskdask-distributedmsgpack

解决方案

这是通过此 PR使用Prefect 0.12.5修复的： https ://github.com/PrefectHQ/prefect/pull/3009 （供后代使用）。

推荐阅读