python - 调用 fork() 时,多处理导致 Python 崩溃并给出错误可能已在另一个线程中进行
问题描述
我对 Python 比较陌生,并试图为我的 for 循环实现一个多处理模块。
我有一个存储在 img_urls 中的图像 url 数组,我需要下载并应用一些 Google 视觉。
if __name__ == '__main__':
img_urls = [ALL_MY_Image_URLS]
runAll(img_urls)
print("--- %s seconds ---" % (time.time() - start_time))
这是我的 runAll() 方法
def runAll(img_urls):
num_cores = multiprocessing.cpu_count()
print("Image URLS {}",len(img_urls))
if len(img_urls) > 2:
numberOfImages = 0
else:
numberOfImages = 1
start_timeProcess = time.time()
pool = multiprocessing.Pool()
pool.map(annotate,img_urls)
end_timeProcess = time.time()
print('\n Time to complete ', end_timeProcess-start_timeProcess)
print(full_matching_pages)
def annotate(img_path):
file = requests.get(img_path).content
print("file is",file)
"""Returns web annotations given the path to an image."""
print('Process Working under ',os.getpid())
image = types.Image(content=file)
web_detection = vision_client.web_detection(image=image).web_detection
report(web_detection)
当我运行它并且python崩溃时,我收到了这个警告
objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
解决方案
发生此错误是因为在 macOS High Sierra 和更高版本的 macOS 中增加了限制多线程的安全性。我知道这个答案有点晚了,但我使用以下方法解决了这个问题:
设置环境变量.bash_profile
(或.zshrc
最近的 macOS)以允许在新的 macOS High Sierra 安全规则下使用多线程应用程序或脚本。
打开一个终端:
$ nano .bash_profile
将以下行添加到文件末尾:
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
保存,退出,关闭终端并重新打开终端。检查环境变量是否已设置:
$ env
您将看到类似于以下内容的输出:
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/pn/vasdlj3ojO#OOas4dasdffJq/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.E7qLFJDSo/Render
TERM_PROGRAM_VERSION=404
TERM_SESSION_ID=NONE
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
您现在应该能够使用多线程运行您的 Python 脚本。
推荐阅读
- sql - 如何选择数据但没有相似的时间?
- c# - 如何将数据从一个 WPF 应用程序传递到另一个 WPF 应用程序?
- android - 如何修复:Recyclerview 未显示在 Activity 中
- java - 为什么操作系统说内存足够时无法为jvm分配内存
- javascript - 如何排除 Firebase 数据库树中的某些键
- dataframe - 将 Julia DataFrames 列表写入文件
- certificate - 在 WCF 自定义发送适配器上设置 clientCredentials 端点行为
- typescript - 打字稿无法推断函数参数
- webpack-4 - 具有自己的块依赖项的动态模块导入
- python - 无法更改数组某些部分的值