apache-spark - 在 java 中使用 Apache Spark Stream 从刻度数据创建蜡烛数据

Question

我们正在获取 Kafka 上的滴答数据，并将其流式传输到 Apache Spark。我们需要从该流数据创建蜡烛数据。

我想创建数据框的第一个选项，然后从那里运行 sql 查询，例如

SELECT t1.price AS open,
       m.high,
       m.low,
       t2.price as close,
       open_time
FROM (SELECT MIN(timeInMilliseconds) AS min_time,
             MAX(timeInMilliseconds) AS max_time,
             MIN(price) as low,
             MAX(price) as high,
             FLOOR(timeInMilliseconds/(1000*60)) as open_time
      FROM ticks
      GROUP BY open_time) m
JOIN ticks t1 ON t1.timeInMilliseconds = min_time
JOIN ticks t2 ON t2.timeInMilliseconds = max_time

但我不确定是否能够获取旧刻度的数据

是否可以使用 Spark 库的某些方法来创建类似的？

标签： apache-sparkapache-spark-sqlspark-streaming

Answer 1

请查看事件时间的窗口操作https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time 这正是您所需要的。这是一个scatch代码

val windowedCounts = tickStream.groupBy(
   window($"timeInMilliseconds", "1 minutes"))
 ).agg(
    first("price").alias("open"),
    min("price").alias("min"),
    max ("price").alias("max"),
    last("price").alias("close"))