时间戳

首页 > 解决方案 > Dask - 警告 - Worker 超出了 95% 的内存预算

python - Dask - 警告 - Worker 超出了 95% 的内存预算

问题描述

我收到错误

Dask - 警告 - Worker 超出了 95% 的内存预算。

我正在使用具有 4 个物理内核和 8 个虚拟内核的本地 PC，我尝试了以下操作：

每...

管理 dask localcluster 上的 worker 内存

...以及此处的文档...

https://distributed.readthedocs.io/en/latest/worker.html#memory-management

...我尝试编辑 .config\dask\distributed.yaml 以取消注释底部五行...

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 0.60  # target fraction to stay below
      spill: 0.70  # fraction at which we spill to disk
      pause: 0.80  # fraction at which we pause worker threads
      terminate: 0.95  # fraction at which we terminate the worker

我还在我的代码中尝试了以下内容：

from dask.distributed import Client, LocalCluster

    worker_kwargs = {
        'memory_limit': '1G',
        'memory_target_fraction': 0.6,
        'memory_spill_fraction': 0.7,
        'memory_pause_fraction': 0.8,
    #     'memory_terminate_fraction': 0.95,
    }

    cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)
    client = Client(cluster, memory_limit='4GB')

...有和没有 Client() 函数的 memory_limit 参数。

有任何想法吗？

标签： pythondask

解决方案

如果您不希望 dask 终止工作人员，则需要在您的分布式.yaml 文件中设置terminate为False：

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 0.60  # target fraction to stay below
      spill: 0.70  # fraction at which we spill to disk
      pause: 0.80  # fraction at which we pause worker threads
      terminate: False  # fraction at which we terminate the worker

（您可能还想设置pause为 False。）

该文件通常位于 ~/.config/dask/distributed.yaml：

警告：不要忘记取消注释distributed:,worker:和memory:行。否则更改将无效。

推荐阅读