时间戳

首页 > 解决方案 > 带有地图的案例类的 Spark 序列化问题

scala - 带有地图的案例类的 Spark 序列化问题

问题描述

我有scala案例类

case class ProbePoint(
   var providerId: Int = 0,
   //...
   var customFields: Map[String, String] = null
 )

此外，父java类

public class Trip implements Serializable, Cloneable {
     // ...
     private Deque<ProbePoint> points = new ArrayDeque<>();
     // ...
}

我用 Spark 做一些事情Dataset[Trip]并遇到下一个错误：

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 10, localhost, executor driver): java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 289, Column 11: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 289, Column 11: 
Cannot instantiate "scala.collection.Map"

我还尝试使用 HashMap 将 scala 类转换为 java 类，但遇到下一个异常的问题：

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 10, localhost, executor driver): java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 423, Column 28: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 423, Column 28: No applicable constructor/method found for actual parameters "java.util.Map"; 
candidates are: "public void com.inrix.analytics.tapp.data.model.ProbePoint.setCustomFields(java.util.HashMap)"

标签： scalaapache-sparkserialization

解决方案

推荐阅读