时间戳

首页 > 解决方案 > 无法启动 spark start-all.sh 权限被拒绝

apache-spark - 无法启动 spark start-all.sh 权限被拒绝

问题描述

我正在尝试在新的 ubuntu 18.10 VM 上设置 Spark（具体来说是 pySpark）。我已经设置了 Hadoop、没有密码的 SSH (localhost)、名为 hduser 的不同用户，这是 Hadoop 所需要的。现在我将 Spark 文件夹放在它旁边，添加了 bashrc 的路径：

export HADOOP_HOME=/usr/local/hadoop-3.1.2
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Native path
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# Java path
export JAVA_HOME="/usr"
# OS path
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/sbin

export SPARK_HOME=/usr/local/spark-2.4.1-bin-hadoop2.7
export PATH=$PATH:/usr/local/spark-2.4.1-bin-hadoop2.7/bin

所以现在从 hduser 我 cd 进入 $SPARK_HOME 并尝试运行./sbin/start-all.sh，我收到以下消息：

chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
/usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out: Permission denied
failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.master.Master --host cristi-VirtualBox --port 7077 --webui-port 8080
tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out' for reading: No such file or directory
full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.master.Master-1-cristi-VirtualBox.out
localhost: chown: changing ownership of '/usr/local/spark-2.4.1-bin-hadoop2.7/logs': Operation not permitted
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out
localhost: /usr/local/spark-2.4.1-bin-hadoop2.7/sbin/spark-daemon.sh: line 128: /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out: Permission denied
localhost: failed to launch: nice -n 0 /usr/local/spark-2.4.1-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://cristi-VirtualBox:7077
localhost: tail: cannot open '/usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out' for reading: No such file or directory
localhost: full log in /usr/local/spark-2.4.1-bin-hadoop2.7/logs/spark-hduser-org.apache.spark.deploy.worker.Worker-1-cristi-VirtualBox.out

究竟是什么问题？我不明白发生了什么事。

标签： apache-sparkpyspark

解决方案

原来我需要给我的其他用户权限。我用过 sudo chown -R hduser /usr/local/spark-2.4.1-bin-hadoop2.7并且有效

推荐阅读