java - FlatMapFunction 没有定义一个 serialversionUid

Question

@Component("loader")
    @Slf4j
    public class LoaderSpark implements Serializable , SparkJob {
            private static final int MAX_VERSIONS = 1;
            private static final long serialVersionUID = 5911491576792653184L;
           spark.sparkContext().getConf().toDebugString());
          HBaseUtil hBaseUtil = new HBaseUtil(props);

        byte[][] prefixes = new byte[][]{toBytes("document"),
                toBytes("dataSource"),
                toBytes("hold:")};

        Filter filter = new MultipleColumnPrefixFilter(prefixes);

        Scan scan = new Scan();
        scan.addFamily(toBytes("data"));        
        scan.setFilter(filter);

    @Override
    public void run(final SparkSession spark, final String... args) {

        JavaPairRDD<ImmutableBytesWritable, Result> ScanRDD
                = hBaseUtil.createScanRdd(spark, "Test", scan);

        ScanRDD.mapToPair(new FlatMapFunction<Tuple2<ImmutableBytesWritable,Result>, String, String>() {


            @Override
            public Iterator<String> call(Tuple2<ImmutableBytesWritable, Result> immutableBytesWritableResultTuple2) throws Exception {
                return null;
            }

        });

我的 ScanRDD.mapToPair 函数抱怨缺少 serialversionUid 并要求在里面添加一个ScanRDD.mapToPair(new FlatMapFunction<Tuple2<ImmutableBytesWritable,Result>, String, String>() {

我想知道为什么我需要将 serialversionUid 添加到我的 FlatMapFunction ？我已经有一个可以上课了。

有什么帮助吗？

标签： javaapache-spark

Answer 1

其背后的原因只是一个静态分析，它验证每个实现 Serializable 的类实例是否具有serialVersionUID字段。

在您的情况下， FlatMapFunction 实现java.io.Serializable并且您正在创建它的匿名类。

所以只需添加serialVersionUID

ScanRDD.mapToPair(new FlatMapFunction<Tuple2<ImmutableBytesWritable,Result>, String, String>() {
//inside the anonymous class
private final serialVersionUID = 1L;

@Override
public Iterator<String> call(Tuple2<ImmutableBytesWritable, Result> immutableBytesWritableResultTuple2) throws Exception {
                return null;
}

})

见https://spark.apache.org/docs/2.2.2/api/java/org/apache/spark/api/java/function/FlatMapFunction.html