python - Python的多处理管理器dict连接错误
问题描述
我正在尝试与多处理共享字典。问题类似于这些:
但是,建议的解决方案对我不起作用(添加 process.join())。一个最小的代码示例是:
import multiprocessing as mp
from multiprocessing import Manager
def test(i, multi_cache):
print(i)
multi_cache[i] = 'Test'
print('inside', multi_cache.items())
with Manager() as manager:
multi_cache = manager.dict()
processes = [mp.Process(target=test, args=(i, multi_cache)) for i in range(3)]
for p in processes:
p.start()
for p in processes:
p.join()
print(multi_cache.items())
输出是:
0
1
inside [(0, 'Test'), (1, 'Test')]
2
之后我得到这个错误:
Process Process-8:
Process Process-6:
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "<ipython-input-3-c78e9bf0d6ec>", line 8, in test
print('inside', multi_cache.items())
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
conn = self._tls.connection
File "<string>", line 2, in items
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 757, in _callmethod
kind, result = conn.recv()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
ConnectionResetError: [Errno 104] Connection reset by peer
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "<ipython-input-3-c78e9bf0d6ec>", line 7, in test
multi_cache[i] = 'Test'
File "<string>", line 2, in __setitem__
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
self._connect()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 741, in _connect
dispatch(conn, None, 'accept_connection', (name,))
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 79, in dispatch
kind, result = c.recv()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-3-c78e9bf0d6ec>", line 21, in <module>
print(multi_cache.items())
File "<string>", line 2, in items
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
self._connect()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
奇怪的是,当只启动一个 Process 时,错误略有不同,并在 print 语句处中断:
Traceback (most recent call last):
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-d7a43009862c>", line 21, in <module>
print(multi_cache.items())
File "<string>", line 2, in items
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
self._connect()
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 493, in Client
answer_challenge(c, authkey)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
message = connection.recv_bytes(256) # reject large message
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
我正在使用 Python 3.6.10 在 ubuntu 16.04 上尝试这个。知道如何解决这个问题吗?谢谢!
解决方案
推荐阅读
- elasticsearch - 线程“Thread-21”中的异常 java.lang.NoSuchFieldError: FAIL_ON_SYMBOL_HASH_OVERFLOW
- c++ - 将二维数组作为类构造函数的参数输入时如何将其作为参考
- sql - 避免重复来自 2 个左连接表的聚合列值
- python - 如何将 celery 集成到 flask + redis 中?
- apache-kafka - 保留.bytes 是否定义了非活动段的最大大小?
- java - jhipster/spring keycloak 集成 - 设置自定义 redirect_uri
- android - 在 AutocompleteSupportFragment 中只想显示机场建议?
- c# - 将 C# UdpClient 连接到代理
- python-3.x - 除了长度之外,这两个代码块之间有区别吗?
- angular - Angular OnChange、DoCheck 和 Observables