时间戳

首页 > 解决方案 > 连接 hive 和 spark 时出现异常HDFS 上的根暂存目录：/tmp/hive 应该是可写的。当前权限为：rwxrwxr-x

scala - 连接 hive 和 spark 时出现异常HDFS 上的根暂存目录：/tmp/hive 应该是可写的。当前权限为：rwxrwxr-x

问题描述

我正在使用 Spark 2.4.5、Hive 3.1.2、Hadoop 3.2.1。在 spark 中运行 hive 时出现以下异常，

Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxrwxr-x;

这是我的源代码，

package com.spark.hiveconnect

import java.io.File

import org.apache.spark.sql.{Row, SaveMode, SparkSession}

object sourceToHIve {
  case class Record(key: Int, value: String)
  def main(args: Array[String]){
    val warehouseLocation = new File("spark-warehouse").getAbsolutePath

    val spark = SparkSession
      .builder()
      .appName("Spark Hive Example").master("local")
      .config("spark.sql.warehouse.dir", warehouseLocation)
      .enableHiveSupport()
      .getOrCreate()

    import spark.implicits._
    import spark.sql

    sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive")
    sql("LOAD DATA LOCAL INPATH '/usr/local/spark3/examples/src/main/resources/kv1.txt' INTO TABLE src")
    sql("SELECT * FROM src").show()

    spark.stop()
  }

}

这是我的 sbt 文件

name := "SparkHive"

version := "0.1"

scalaVersion := "2.12.10"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.5"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.5"
// https://mvnrepository.com/artifact/mysql/mysql-connector-java
libraryDependencies += "mysql" % "mysql-connector-java" % "8.0.19"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.5"

如何解决这个问题？在观察控制台时，我也看到了这个声明，这就是我遇到这个问题的原因。

20/05/28 14:03:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(UDHAV.MAHATA); groups with view permissions: Set(); users  with modify permissions: Set(UDHAV.MAHATA); groups with modify permissions: Set()

谁能帮我？

谢谢你！

标签： scalaapache-sparkhadoophive

解决方案

您可以尝试从外壳更改权限

hdfs dfs -chmod -R 777 /tmp/hive

或在正确的权限下与另一个用户或应用程序一起运行您的 spark 进程。

您可以使用以下方法更改目录或文件的所有者：

-chown [-R] [OWNER][:[GROUP]] PATH... :
  Changes owner and group of a file. This is similar to the shell's chown command
  with a few exceptions.

  -R  modifies the files recursively.

推荐阅读