java - sqlContext.createDateFrame() 时的 NPE
问题描述
我有以下代码,我想使用 spark 将记录列表更新到 mysql db 中。
DataFrame df = sqlContext.createDataFrame(airlineList, Airline.class);
sparkHelper.saveAirlinesToMysql(df);
Airline.class 是
@Getter @Setter
public class Airline implements Serializable {
private Long id;
private String name;
private String referenceId;
@Getter(AccessLevel.NONE)
@Setter(AccessLevel.NONE)
private Object[] parsedReferenceId;
public Object[] getParsedReferenceId() {
if (parsedReferenceId == null) {
parsedReferenceId = ReferenceHelper.parseReferenceId(this.referenceId);
}
return parsedReferenceId;
}
}
当我运行火花作业以保存到数据库时,它显示了我
ERROR [2019-07-17 05:06:49,505] com.api.command.SparkDriverCommand: Spark run failed due to NullPointerException at org.spark-project.guava.reflect.TypeToken.method(TypeToken.java:465)
java.lang.NullPointerException
at org.spark-project.guava.reflect.TypeToken.method(TypeToken.java:465)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:110)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:109)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:109)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:111)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:109)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:109)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:111)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:109)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:109)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:111)
at org.apache.spark.sql.catalyst.JavaTypeInference$$anonfun$2.apply(JavaTypeInference.scala:109)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.sql.catalyst.JavaTypeInference$.org$apache$spark$sql$catalyst$JavaTypeInference$$inferDataType(JavaTypeInference.scala:109)
at org.apache.spark.sql.catalyst.JavaTypeInference$.inferDataType(JavaTypeInference.scala:54)
at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:941)
at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:603)
at com.api.command.SparkDriverCommand.saveAirlines(SparkDriverCommand.java:485)
at com.api.command.SparkDriverCommand.run(SparkDriverCommand.java:465)
at com.api.command.SparkDriverCommand.run(SparkDriverCommand.java:107)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
at com.api.command.SparkDriverCommand.run(SparkDriverCommand.java:120)
我想知道是不是因为 parsedReferenceId 字段没有 setter 和 getter?请帮忙。提前致谢。
解决方案
推荐阅读
- java - 在一段时间后调用 API 时第一次获取连接重置异常
- vba - 避免出现错误的“打开并修复”消息
- c++ - 将 QWidget 链接到另一个新创建的 QWidget
- php - Java - Android Studio - HttpURLConnection 发布到 PC PHP 文件
- android - 没有组件的android App有什么用?
- python - pymysql IO 期间 CPU 过高
- sql - Hive 查询以获取另一个日期一个月范围内的日期
- flask - Jinja2“无效语法”的数据样式属性
- c++ - 从内存 ImageMagick 解码 svg
- python - 如何在列表中添加字典