apache-spark - 原因：java.lang.Exception：您不能在 Spark 闭包中使用 GraphFrame 对象

Question

我有一个函数get_paths，它采用 root_node_id 和 leaf_node_id 并使用 graphframe 的 bsf 函数返回这些节点之间的路径。我有一个trans_df1具有 root_node_id 和 leaf_node_id的数据框

我需要为get_paths数据框中的每一行调用该函数。我怎样才能完成这个？

我尝试创建 UDFget_paths并从数据框调用，但出现**You cannot use GraphFrame objects within a Spark closure**异常

def get_paths(from_id: String, to_id:String) : Row =
{
    val paths: DataFrame = grph.bfs.fromExpr("id = '" +from_id+"'").toExpr("id == '"+to_id+"'").run()
    val c = paths.columns.filter(_.startsWith("e")).map(x => x+".src") ++ paths.columns.filter(_.startsWith("to")).map(x => x+".id")
    val return_value = paths.withColumn("path_to_leaf", concat_ws("->", c.map(paths(_)) : _*)).select("path_to_leaf").rdd.collect()(0)
    return return_value
}

print( "Path is: " + get_paths("A", "7091501") ); //Instead of hardcoding, I need to pass root and leaf value from below dataframe. 

scala> trans_df1.show()
+----+--------+---------------+
|root|    leaf|   root_to_leaf|
+----+--------+---------------+
|   A|11637001|[[A, 11637001]]|
|   A|11652801|[[A, 11652801]]|
|   A| 1420901| [[A, 1420901]]|
|   A| 7091501| [[A, 7091501]]|
+----+--------+---------------+

标签： apache-spark