时间戳

首页 > 解决方案 > 如何解决java spark中的“线程“main”java.lang.Error中的异常：未解决的编译问题：SparkSession无法解析为类型”

java - 如何解决java spark中的“线程“main”java.lang.Error中的异常：未解决的编译问题：SparkSession无法解析为类型”

问题描述

我在 Azure Windows 虚拟机中安装了“VScode、jdk 8、python 3.8 和 databricks-connect==8.1.*” 。之后，我创建了一个 databricks 集群并使用 cmd配置了 Databricks-connect 。设置完所有路径变量后，我按照 Java 代码执行：

'''

import java.util.ArrayList;
import java.util.List;
import java.sql.Date;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.*;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.Dataset;

public class test_{
    public static void main(String[] args) throws Exception {
        SparkSession spark = SparkSession.builder().appName("Temps Demo").config("spark.master", "local").getOrCreate();


        // Create a Spark DataFrame consisting of high and low temperatures
        // by airport code and date.
        StructType schema = new StructType(new StructField[] {
            new StructField("AirportCode", DataTypes.StringType, false, Metadata.empty()),
            new StructField("Date", DataTypes.DateType, false, Metadata.empty()),
            new StructField("TempHighF", DataTypes.IntegerType, false, Metadata.empty()),
            new StructField("TempLowF", DataTypes.IntegerType, false, Metadata.empty()),
        });
}}'''

==================================================== ======= 在 VScode 终端内我得到低于输出'''

PS C:\Users\gaurav>  & 'c:\Users\gaurav\.vscode\extensions\vscjava.vscode-java-debug-0.35.0\scripts\launcher.bat' 'C:\Program Files\AdoptOpenJDK\jdk-11.0.11.9-hotspot\bin\java.exe' '-Dfile.encoding=UTF-8' '-cp' 'C:\Users\gaurav\AppData\Local\Temp\vscodesws_8933a\jdt_ws\jdt.ls-java-project\bin' 'App' 
    Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
            SparkSession cannot be resolved to a type
            SparkSession cannot be resolved
            StructType cannot be resolved to a type
            StructType cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
    PS C:\Users\gaurav>  & 'c:\Users\gaurav\.vscode\extensions\vscjava.vscode-java-debug-0.35.0\scripts\launcher.bat' 'C:\Program Files\AdoptOpenJDK\jdk-11.0.11.9-hotspot\bin\java.exe' '-Dfile.encoding=UTF-8' '-cp' 'C:\Users\gaurav\AppData\Local\Temp\vscodesws_8933a\jdt_ws\jdt.ls-java-project\bin' 'test_'
    Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
            SparkSession cannot be resolved to a type
            SparkSession cannot be resolved
            StructType cannot be resolved to a type
            StructType cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
    
            at test_.main(testjava.java:22)

'''

当我在配置 databricks 连接后在 cmd 中运行“databricks-connect 测试”时。我得到以下输出... [1]：https ://i.stack.imgur.com/p5WG6.png

标签： javaapache-sparkvisual-studio-codeazure-databricksdatabricks-connect

解决方案

org.apache.spark.sql.types.StructType在spark-catalyst中可以实现，将两者都添加spark-catalyst.jar到spark-sql.jarReferenced Libraries中。

此外，java 扩展需要jdk11 或最新版本来支持。

满足以上条件，StructField cannot be resolved to a type我的项目中没有显示。

推荐阅读