java - Spark 1.6 与 Kryo 2.21 和 ArrayList
问题描述
我正在尝试将 Spark 1.6.0 与 Kryo 2.21 一起使用。这真的令人沮丧,因为我无法弄清楚是什么导致了这个错误。
我在 Arraylist 类的第 234 行没有看到任何可能导致空指针的内容。在类路径上是否也有 kryo 2.22 会导致这种情况。请帮忙。谢谢
SparkConf conf = new SparkConf().setAppName("ABC");
conf.set("spark.serializer", KryoSerializer.class.getName());
conf.registerKryoClasses(new Class[] { ArrayList.class, VC.class, DC.class });
堆栈跟踪 :
java.lang.NullPointerException
at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:234)
at java.util.ArrayList.ensureCapacity(ArrayList.java:218)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:89)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:228)
at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171)
at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201)
at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:153)
at org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:58)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:89)
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:98)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
解决方案
推荐阅读
- php - 根原因拥有的 Laravel Storage Cache 文件夹无法清除缓存
- python - Dataframe.to_json() 不呈现所有参数(熊猫)
- python - 从 protobuf 字节数据读取到 msg
- blueprism - Blue Prism 未检测到 Chrome 打开工具中的元素
- spring-boot - 如何使用 spring boot 和 spring secuirty 使用基本身份验证标头中的凭据对 LDAP 中的用户进行身份验证
- c# - 在 C# 中定义 BitVector32 集合时出现命名空间错误
- node.js - 使用 nodejs 向 Discord 频道发送未经请求的消息
- excel - 如何将文件附加到 MailItem
- api - 无法渲染图表,来自 api 的数据
- php - 使用 PHP powershell 脚本更改默认打印机纸张大小