时间戳

首页 > 解决方案 > Apache Spark - Java -JSCH - 算法协商失败

java - Apache Spark - Java -JSCH - 算法协商失败

问题描述

使用 Spark 2.0 和 JSCH 进行文件传输时出现以下异常。Spark lib 使用 0.1.42 版本，并且还遇到了覆盖内置依赖项的选项。

它在客户端模式下工作正常，但是当提交到 Yarn 集群时，它会失败并出现以下异常。任何建议，请。

    1Connecting to xxxxxxxxxxxx.com port 22
    1Connection established
    1Remote version string: SSH-2.0-OpenSSH_7.4
    1Local version string: SSH-2.0-JSCH-0.1.42
    1CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
    1SSH_MSG_KEXINIT sent
    1SSH_MSG_KEXINIT received
    1Disconnecting from xxxxxxxxxxxxxxx.com port 22
    18/09/26 10:56:50 ERROR Myclass: Error occurred in processing:
    com.jcraft.jsch.JSchException: Algorithm negotiation fail
com.jcraft.jsch.JSchException: Algorithm negotiation fail
    at com.jcraft.jsch.Session.receive_kexinit(Session.java:520)
    at com.jcraft.jsch.Session.connect(Session.java:286)
    at com.jcraft.jsch.Session.connect(Session.java:150)

如果我尝试使用覆盖选项来加载 spark submit 命令中给出的 jar，那么我得到：

java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of org/apache/spark/util/ChildFirstURLClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at org.apache.commons.logging.LogFactory$Slf4jDelegate.createLocationAwareLog(LogFactory.java:174)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:111)

这里已经有一个问题 - JSch 算法协商失败 ，但从 Java 的角度来看，这看起来更多，但从 Yarn 和 Spark 来看。

标签： javaapache-sparkjsch

解决方案

推荐阅读