时间戳

首页 > 解决方案 > Livy Batch API - 无法获取 jar

apache-spark - Livy Batch API - 无法获取 jar

问题描述

我正在尝试使用 Livy Batch API 来调用我的 Spark 应用程序。

curl -X POST -d '{"file": "file:///path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory": "8g","executorCores": 2,"numExecutors": 2,"queue": "default","name": "Livy_sample_prog"}' -H "Content-Type: application/json" http://serverip:8998/batches/

POST/batches 失败并出现以下错误：

"Warning: Local jar /path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar does not exist, skipping.",
  "java.lang.ClassNotFoundException: App",

以下是我迄今为止尝试过的各种事情

1.上传jar到HDFS

curl -X POST -d '{"file": "hdfs://path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory": "8g","executorCores": 2,"numExecutors": 2,"queue": "default","name": "Livy_sample_prog"}' -H "Content-Type: application/json" http://serverip:8998/batches/

例外：

"Warning: Skip remote jar hdfs://path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar.",
  "java.lang.ClassNotFoundException: App"

2.添加 livy.file.local-dir-whitelist 作为包含 jar 文件的目录

3.将文件：///更改为本地：/

我已经多次验证文件是否存在，并且每种情况下提供的路径都是有效的。

我现在没有选择了。请帮助找到解决方案。

谢谢！！

标签： apache-sparklivy

解决方案

我能够将这项工作作为纱线集群的主人。

这是我使用的 --data

{
    "file": "local:/path/to/file/SampleSparkProject-0.0.2-SNAPSHOT.jar",
    "proxyUser": "nrsapp",
    "className": "App",
    "args": [
        "abc"
    ],
    "name": "Livy_sample_prog",
    "conf": {
        "spark.master": "yarn-cluster",
        "spark.deployMode": "client"
    }
}

推荐阅读