时间戳

首页 > 解决方案 > 具有多个参数的 Dask map 方法

python - 具有多个参数的 Dask map 方法

问题描述

我想将该方法Client.map应用于使用多个参数的函数，就像. 这是一个例子Pool.starmapmultiprocessing

from contextlib import contextmanager

from dask.distributed import Client


@contextmanager
def dask_client(**kwargs):
    """some docs"""
    kwargs.setdefault("ip", "localhost:8786")
    client = Client(**kwargs)

    try:
        yield client
    except Exception:
        raise
    finally:
        client.close()


def f(x,y,z):
    return x+y+z
# Dummy function

if __name__ == "__main__":
    with dask_client() as client:
        client.map(f, (1,2,3), (1,2,3))

distributed.worker - WARNING - Compute Failed
Function:  f
args:      (1, 1)
kwargs:    {}
Exception: TypeError("f() missing 1 required positional argument: 'z'")

distributed.worker - WARNING - Compute Failed
Function:  f
args:      (2, 2)
kwargs:    {}
Exception: TypeError("f() missing 1 required positional argument: 'z'")

distributed.worker - WARNING - Compute Failed
Function:  f
args:      (3, 3)
kwargs:    {}
Exception: TypeError("f() missing 1 required positional argument: 'z'")

这是这里接受的答案

我知道每个元组都被视为x我的功能f。如果可能的话，我不想要这样的解决方案

def f(var_list):
    # could be sum(), but this is a dummy example
    return var_list[0] + var_list[1] + var_list[2]

标签： pythondask

解决方案

你已经接近了，请注意应该有与你的函数中的参数相同数量的迭代：

from dask.distributed import Client
client = Client()

def f(x,y,z):
    return x+y+z

futs = client.map(f, *[(1,2,3), (4,5,6), (7,8,9)])

client.gather(futs) # [12, 15, 18]

从评论看来，您似乎想将所有参数存储在一个元组中，在这种情况下，您可以这样做：

# will pass them as x=1, y=2, z=3
long_list = [(1,2,3), (4,5,6), (7,8,9), (10,11,12)]

futs = client.map(f, *zip(*long_list))

client.gather(futs) # [6, 15, 24, 33]

推荐阅读