时间戳

首页 > 解决方案 > 如何查找 dask 数组分区的行索引

python - 如何查找 dask 数组分区的行索引

问题描述

我有一个 2D (4950, 4950) dask 数组，我想并行计算。使用链接：https ://docs.dask.org/en/latest/delayed-best-practices.html#don-t-call-dask-delayed-on-other-dask-collections

print(da.shape)
partitions = da.to_delayed()
print(partitions)
delayed_values = [dask.delayed(funct)(part) for part in partitions]
print(delayed_values)

我得到的结果是：

(4950, 4950)
[[Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 3))]
 [Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 3))]
 [Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 3))]
 [Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 3))]]
[Delayed('funct-c0044e9f-4b8e-4d02-b364-f6a483eaae2f'), 
 Delayed('funct-d2d14dcd-6f0a-4198-b999-221b0609bcaa'), 
 Delayed('funct-1951008c-14f4-43da-bbc1-443e90aae029'), 
 Delayed('funct-a254e3ba-2d45-45f8-bae4-85ba8c37a32f')]

我想找出每个分区的行索引（第一个和最后一个索引），以将每个索引的计算结果保存在最终输出文件中。

我找不到很多与分区相关的文档，非常感谢任何可以帮助查找行索引的帮助/链接。

标签： pythondaskdask-distributeddask-delayeddask-ml

解决方案

对于 Dask 数组，您要查看.chunks属性。特别是我认为你可能会想要类似的东西

[np.cumsum(c) for c in x.chunks]

有关更多信息，请参阅https://docs.dask.org/en/latest/array-design.html#chunks

推荐阅读