时间戳

首页 > 解决方案 > Python 多处理 log_to_stderr 在 Windows 上重复

python - Python 多处理 log_to_stderr 在 Windows 上重复

问题描述

我正在关注使用多处理进行日志记录的文档，但我看到工作人员在每个子进程中创建了两个日志。我在某个地方犯了一个愚蠢的错误吗？

环境：

Python 3.6.1 |Anaconda 自定义（64 位）| （默认，2017 年 5 月 11 日，13:25:24）[MSC v.1900 64 位 (AMD64)] 在 win32 上

代码（编辑以修复@georgexsh 推荐的范围问题）：

import logging
import multiprocessing


logger = multiprocessing.log_to_stderr(logging.INFO)


def test(i):
    logger.info(f'worker processing {i}')


if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        metrics = pool.map(test, range(20))

记录输出：

[INFO/SpawnPoolWorker-2] child process calling self.run()
[INFO/SpawnPoolWorker-2] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 0
[INFO/SpawnPoolWorker-3] child process calling self.run()
[INFO/SpawnPoolWorker-3] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 0
[INFO/SpawnPoolWorker-1] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 1
[INFO/SpawnPoolWorker-2] worker processing 2
[INFO/SpawnPoolWorker-1] child process calling self.run()
[INFO/SpawnPoolWorker-6] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 1
[INFO/SpawnPoolWorker-4] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 2
[INFO/SpawnPoolWorker-5] child process calling self.run()
[INFO/SpawnPoolWorker-7] child process calling self.run()
[INFO/SpawnPoolWorker-1] worker processing 3
[INFO/SpawnPoolWorker-6] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 4
[INFO/SpawnPoolWorker-4] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 5
[INFO/SpawnPoolWorker-5] child process calling self.run()
[INFO/SpawnPoolWorker-7] child process calling self.run()
[INFO/SpawnPoolWorker-1] worker processing 3
[INFO/SpawnPoolWorker-6] worker processing 6
[INFO/SpawnPoolWorker-3] worker processing 4
...
[INFO/SpawnPoolWorker-5] worker processing 16
[INFO/SpawnPoolWorker-2] worker processing 12
[INFO/SpawnPoolWorker-7] worker processing 17
[INFO/SpawnPoolWorker-1] worker processing 18
[INFO/SpawnPoolWorker-6] worker processing 13
[INFO/SpawnPoolWorker-3] worker processing 19
[INFO/SpawnPoolWorker-8] worker processing 14
[INFO/SpawnPoolWorker-4] worker processing 15
[INFO/SpawnPoolWorker-5] worker processing 16
[INFO/SpawnPoolWorker-7] worker processing 17
[INFO/SpawnPoolWorker-1] worker processing 18
[INFO/SpawnPoolWorker-3] worker processing 19
[INFO/SpawnPoolWorker-2] process shutting down
[INFO/SpawnPoolWorker-6] process shutting down
[INFO/MainProcess] process shutting down

标签： pythonwindowsloggingduplicatesmultiprocessing

解决方案

移动logger = multiprocessing.log_to_stderr()到全局范围，而不是工作函数内部。以确保它只调用一次。因为每次log_to_stderr被调用，它都会向记录器添加一个新的处理程序：

def test(i):
    logger.info('worker processing %s', i)

if __name__ == '__main__':
    logger = multiprocessing.log_to_stderr(logging.INFO)

请注意，在 windows 下，由于没有fork()，当创建子进程以重建上下文时，整个模块会再次执行，您可以使用Pool'sinitializer初始化记录器，它只运行一次前子进程：

logger = None

def test(i):
    logger.info('worker processing %s', i)

def initializer(level):
    global logger
    logger = multiprocessing.log_to_stderr(level)

if __name__ == '__main__':
    pool = multiprocessing.Pool(4, initializer=initializer, initargs=(logging.INFO,))
    metrics = pool.map(test, range(20))

推荐阅读