apache-spark - AWS EMR 火花提交选项 - 失败
问题描述
我正在尝试在 EMR spark 集群上运行 python 作业。
在 spark 提交选项中,我仅使用以下设置来更改内存:
-- master yarn --driver-memory 4g --executor-memory 2g
但是,每次我尝试提交作业时,它总是失败。
我可以知道如何更正火花提交选项吗?
错误日志如下:
Error: Unrecognized option: --
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]
Options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn,
k8s://https://host:port, or local (Default: local[*]).
--deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or
on one of the worker machines inside the cluster ("cluster")
(Default: client).
--class CLASS_NAME Your application's main class (for Java / Scala apps).
--name NAME A name of your application.
--jars JARS Comma-separated list of jars to include on the driver
and executor classpaths.
--packages Comma-separated list of maven coordinates of jars to include
on the driver and executor classpaths. Will search the local
maven repo, then maven central and any additional remote
repositories given by --repositories. The format for the
coordinates should be groupId:artifactId:version.
--exclude-packages Comma-separated list of groupId:artifactId, to exclude while
resolving the dependencies provided in --packages to avoid
dependency conflicts.
--repositories Comma-separated list of additional remote repositories to
search for the maven coordinates given with --packages.
--py-files PY_FILES Comma-separated list of .zip, .egg, or .py files to place
on the PYTHONPATH for Python apps.
--files FILES Comma-separated list of files to be placed in the working
directory of each executor. File paths of these files
in executors can be accessed via SparkFiles.get(fileName).
--conf PROP=VALUE Arbitrary Spark configuration property.
--properties-file FILE Path to a file from which to load extra properties. If not
specified, this will look for conf/spark-defaults.conf.
--driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
--driver-java-options Extra Java optionCommand exiting with ret '1'
解决方案
我可以看到--
和之间有一个空格master
。
尝试--master
代替-- master
.
推荐阅读
- cmake - 无法在 Ubuntu 18.04 上使用 CMake 配置 QtCreator
- python - 如何从输入分配到类中的对象
- cassandra - Cassandra 在插入后更新前几行
- gitlab - 在没有个人“私人令牌”的情况下调用 GitLab API
- c# - 需要覆盖导航方法
- google-cloud-platform - Google Cloud CDN 开始忽略存储分区的查询字符串
- r - R - 具有不同列的矩阵的总和列表
- r - 如何使用零膨胀回归模型进行预测?
- python - 在加入 2 个列表时添加空字符串 - Python
- android - Android Studio 生成对 .iml 文件的不需要的 java 源文件夹引用