apache-spark - 火花错误 - 安装错误
问题描述
安装 Spark 并运行后
C:\spark-2.3.1-bin-hadoop2.7\bin>spark-shell
我收到以下错误 - 有什么建议吗?
C:\spark-2.3.1-bin-hadoop2.7\bin>spark-shell
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/C:/spark-2.3.1-bin-hadoop2.7/jars/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2018-08-05 01:29:36 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Failed to initialize compiler: object java.lang.Object in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Failed to initialize compiler: object java.lang.Object in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.NullPointerException
解决方案
我认为您没有正确的 java 或 scala 版本。
请注意 Spark 2.3.1 运行于
Java 8+,
Python 2.7+/3.4+ and
R 3.1+.
对于 Scala API,Spark 2.3.1 使用 Scala 2.11。您将需要使用兼容的 Scala 版本 (2.11.x)。
请检查以下两件事 - 1. 检查您提交 spark 应用程序的机器上安装的 java 版本
sudo update-alternatives --config java
sudo update-alternatives --config javac
检查scala版本
scala -version
推荐阅读
- excel - 压缩文件夹和文件到文件夹
- ionic-framework - 有什么方法可以使 3D 模型的背景在 ionic 的场景中透明?
- mysql - 在 daterange 中计算 daterange 的天数
- sql - 当 ProC SQL 结果为空时,将其获取为 -nan 而不是 0
- javascript - 从 URL 跨浏览器自动下载文件
- rabbitmq - 更改 RabbitMQ 绑定
- regex - RegEx 用于排除重复的单个字符
- reactjs - 如何描述类型滚动事件?
- django - django.urls.exceptions.NoReverseMatch:使用关键字参数反转“change_view”
- java - 默认情况下,如何使用 appium 将 chrome 作为本机应用程序打开?