python - 具有独立进程的 Python 多进程日志记录
问题描述
目前,Python 的标准文档推荐使用带有 QueueHandler/QueueListener 的多进程队列来实现多进程日志记录。但是,由于多进程队列是使用未命名管道实现的,因此独立进程无法共享同一个队列。
我对其他可行的解决方案感兴趣;我个人的想法是,可以改为实现基于命名管道的队列,这应该允许队列在多个独立进程之间共享。
解决方案
正常记录对我来说很好:
#!/usr/bin/env python
import uuid
import time
import multiprocessing
import logging
logging.basicConfig(filename='multiprocess.log', level=logging.DEBUG)
def act(p):
start_time = time.monotonic()
work()
logging.info(f"{p} took {time.monotonic() - start_time:.3f} seconds")
def work(start_of_uuid='00000'):
while True:
uid = uuid.uuid4().hex
if uid.startswith(start_of_uuid):
return uid
if __name__ == '__main__':
start_time = time.monotonic()
with multiprocessing.Pool(multiprocessing.cpu_count() - 1) as pool:
pool.map(act, range(24))
logging.info(f"Elapsed time overall was {time.monotonic() - start_time:.3f} seconds")
在 multiprocess.log 中生成:
INFO:root:5 took 0.048 seconds
INFO:root:9 took 2.191 seconds
INFO:root:6 took 3.581 seconds
INFO:root:11 took 4.800 seconds
INFO:root:14 took 0.295 seconds
INFO:root:8 took 5.361 seconds
INFO:root:1 took 5.577 seconds
INFO:root:3 took 10.739 seconds
INFO:root:15 took 5.605 seconds
INFO:root:2 took 16.118 seconds
INFO:root:4 took 16.931 seconds
INFO:root:20 took 3.577 seconds
INFO:root:16 took 15.270 seconds
INFO:root:13 took 17.115 seconds
INFO:root:21 took 4.529 seconds
INFO:root:22 took 4.715 seconds
INFO:root:23 took 4.897 seconds
INFO:root:10 took 25.925 seconds
INFO:root:12 took 24.482 seconds
INFO:root:18 took 16.629 seconds
INFO:root:0 took 28.958 seconds
INFO:root:19 took 20.696 seconds
INFO:root:7 took 34.055 seconds
INFO:root:17 took 36.142 seconds
INFO:root:Elapsed time overall was 41.827 seconds
推荐阅读
- php - 如何避免 Laravel 范围内的重复?
- reactjs - 在 useCallback 中更新状态 - React JS
- visual-studio-code - 如何将 vscode Rust tabsize 设置为 4?
- pytest - 使用 asyncio 加速测试
- python - Y标签的放置plotly python水平条形图
- delphi - 一个虚假的 Delphi 错误 E2200,即使在全新的空包项目中也不断显示
- c# - 如何在 MongoDB 中索引 .NET DateTimeOffset 字段?
- javascript - 从 JavaScript 中的字符串中获取特定项目
- google-analytics-api - 个人用户使用 UA 维度和指标的活动
- android - 为什么 Android Studio 在构建签名包时不创建 native-debug-symbols zip 文件