pyspark - Dask 等效于 pyspark 超前和滞后功能

Question

是否有可能在数据帧中接收到可以通过或窗口函数dask接收的类似结果？我想转换以下数据框lagleadpyspark

    +-------+
    | value |
    +-------+
    | 1     |
    | 2     |
    | 3     |
    +-------+

像这样

    +-------+------------+------------+
    | value | prev_value | next_value |
    +-------+------------+------------+
    | 1     | NaN        | 2          |
    | 2     | 1          | 3          |
    | 3     | 2          | NaN        |
    +-------+------------+------------+

标签： pysparkpyspark-sqldask

Answer 1

Dask 数据框只是反映了 pandas 界面。在这种情况下，您想要的方法是shift：

In [3]: import pandas as pd

In [4]: df = pd.DataFrame({'a': range(5)})

In [5]: import dask.dataframe as dd

In [6]: ddf = dd.from_pandas(df, npartitions=2)

In [7]: out = ddf.assign(prev_a=ddf.a.shift(1), next_a=ddf.a.shift(-1))

In [8]: out.compute()
Out[8]:
   a  prev_a  next_a
0  0     NaN     1.0
1  1     0.0     2.0
2  2     1.0     3.0
3  3     2.0     4.0
4  4     3.0     NaN

但是，如果您尝试对齐行以进行某种窗口或滚动计算，您可能对map_overlap更感兴趣，它会更高效。