时间戳

首页 > 解决方案 > Sparkly 连接到 HDP-Sandbox Spark 实例

r - Sparkly 连接到 HDP-Sandbox Spark 实例

问题描述

我想将 R 实例连接到在一台本地机器上部署在 Docker 上的 HDP-Sandbox 上运行的 Spark。错误消息表明--version调用spark-submit失败。

R 实例

packageVersion("sparklyr")
# [1] ‘1.0.1’

# Set old JAVA version
Sys.setenv("JAVA_HOME" =
               "/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home")

library(sparklyr)

sc <- spark_connect(
    master = " http://localhost:4040",
    spark_home = "/usr/hdp/current/spark2-client",
    app_name = "RStudio",
    method = "shell"
)

错误system2(file.path(spark_home, "bin", "spark-submit"), "--version", ：运行命令出错

高密度板

[root@sandbox-hdp ~]# /usr/hdp/current/spark2-client/bin/spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.1.3.0.1.0-187
      /_/

Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_191
Branch HEAD
Compiled by user jenkins on 2018-09-19T10:10:07Z
Revision fe7bed1ca174a6687ebd2aa0f8ba5fb7bf668399
Url git@github.com:hortonworks/spark2.git
Type --help for more information.

更新

我求助于使用 Livy，因为它似乎是一个更简单的解决方案。我已经打开8999了文件中引用的端口，livy.conf现在可以在http://localhost:8999/ui.

李维会议

尽管如此，我在尝试连接时仍然不断收到错误：

config <- livy_config()
sc <- spark_connect(master = "http://localhost:8999",
                     method = "livy",
                     config = config)
# Error in livy_connection(master = master, config = config, app_name, version,  : 
# Failed to launch livy session, session status is still starting after waiting for 60 seconds


config <- livy_config(username = "root", password = "***")
sc <- spark_connect(master = "http://localhost:8999",
                     method = "livy",
                     config = config)
# Error in livy_connection(master = master, config = config, app_name, version,  : 
# Failed to launch livy session, session status is still starting after waiting for 60 seconds

标签： rapache-sparksparklyrlivyhdp

解决方案

推荐阅读