时间戳

首页 > 解决方案 > 虽然我将 setMaster 设置为本地，但我的 spark 应用程序给出了错误

apache-spark - 虽然我将 setMaster 设置为本地，但我的 spark 应用程序给出了错误

问题描述

我在 Windows 中有以下应用程序（我正在启动和停止 spark）。我使用 Scala-IDE(Eclipse)。即使我已在此处设置，我也会收到“必须在您的配置中设置主 URL”错误。我使用 spark-2.4.4 版本。

有人可以帮我解决这个问题。

import org.apache.spark._;
import org.apache.spark.sql._;

object SampleApp {

  def main(args: Array[String]) {    
    val conf =  new SparkConf()
               .setMaster("local[*]")
               .setAppName("Simple Application")

    val sc   =  new SparkContext(conf)

    sc.stop()
  }

}

错误是：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/10/28 22:58:56 INFO SparkContext: Running Spark version 2.4.4
19/10/28 22:58:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/10/28 22:58:56 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
    at com.spark.renga.SampleApp$.main(SampleApp.scala:8)
    at com.spark.renga.SampleApp.main(SampleApp.scala)
19/10/28 22:58:56 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
    at org.apache.spark.SparkContext.postApplicationEnd(SparkContext.scala:2416)
    at org.apache.spark.SparkContext.$anonfun$stop$2(SparkContext.scala:1931)
    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1931)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:585)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
    at com.spark.renga.SampleApp$.main(SampleApp.scala:8)
    at com.spark.renga.SampleApp.main(SampleApp.scala)
19/10/28 22:58:56 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
    at com.spark.renga.SampleApp$.main(SampleApp.scala:8)
    at com.spark.renga.SampleApp.main(SampleApp.scala)

标签： apache-spark

解决方案

如果您使用的是 2.4.4 版，请尝试以下操作：

import org.apache.spark.sql.SparkSession

object SampleApp {

  def main(args: Array[String]) {

    val spark = SparkSession
      .builder
      .master("local[*]")
      .appName("test")
      .getOrCreate()

    println(spark.sparkContext.version)

    spark.stop()
  }
}

推荐阅读