时间戳

首页 > 解决方案 > IPv6 上的 Apache Spark

apache-spark - IPv6 上的 Apache Spark

问题描述

我正在尝试在 IPv6 上安装 Spark，但是，spark-master 出现在 IPv6 - DNS 主机名上，但即使我通过 IP 或 DNS，spark 工作节点也不会启动，错误是相同的。我需要使用 LOCAL_WORKER_IP=127.0.0.1 来启动 spark worker。

SPARK Worker log:
*fd74:ca9b:3a09:868c:172:18:0:462a spark-master.t253-u000265.svc.cluster.local
21/08/09 16:41:40 INFO Worker: Started daemon with process name: 10@v4-virtio-spark-worker-zjgxs

21/08/09 16:41:40 INFO SignalUtils: Registered signal handler for TERM

21/08/09 16:41:40 INFO SignalUtils: Registered signal handler for HUP

21/08/09 16:41:40 INFO SignalUtils: Registered signal handler for INT

21/08/09 16:41:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your
 platform... using builtin-java classes where applicable
21/08/09 16:41:41 INFO SecurityManager: Changing view acls to: root

21/08/09 16:41:41 INFO SecurityManager: Changing modify acls to: root

21/08/09 16:41:41 INFO SecurityManager: Changing view acls groups to:

21/08/09 16:41:41 INFO SecurityManager: Changing modify acls groups to:

21/08/09 16:41:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()

21/08/09 16:41:42 INFO Utils: Successfully started service 'sparkWorker' on port 37816

21/08/09 16:41:42 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]

java.lang.AssertionError: assertion failed: Expected hostname (not IP) but got fd74:ca9b:3a09:868c:172:18:0:488e
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.util.Utils$.checkHost(Utils.scala:1014)
at org.apache.spark.deploy.worker.Worker.<init>(Worker.scala:60)
at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:811)
at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:779)
at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
21/08/09 16:41:42 INFO ShutdownHookManager: Shutdown hook called*

有谁知道是否可以使用任何配置在 IPv6 上配置 SPARK 工作人员？

标签： apache-spark

解决方案

推荐阅读