apache-spark - Spark：如何在 spark-submit 中设置 spark.yarn.executor.memoryOverhead 属性

Question

在 Spark 2.0 中。运行 spark 提交时如何设置 spark.yarn.executor.memoryOverhead。

我知道你可以设置 spark.executor.cores 之类的东西--executor-cores 2。这个属性的模式是否相同？例如--yarn-executor-memoryOverhead 4096

标签： apache-sparkpysparkspark-submit

Answer 1

请找例子。这些值也可以在 Sparkconf 中给出。

例子：

./bin/spark-submit \
--[your class] \
--master yarn \
--deploy-mode cluster \
--num-exectors 17
--conf spark.yarn.executor.memoryOverhead=4096 \
--executor-memory 35G \  //Amount of memory to use per executor process 
--conf spark.yarn.driver.memoryOverhead=4096 \
--driver-memory 35G \   //Amount of memory to be used for the driver process
--executor-cores 5
--driver-cores 5 \     //number of cores to use for the driver process 
--conf spark.default.parallelism=170
 /path/to/examples.jar