时间戳

首页 > 解决方案 > Spark 启动很慢

apache-spark - Spark 启动很慢

问题描述

我有一个使用 spark-submit 启动 spark 应用程序的 shell 脚本。

我今天发现火花开始得很慢。日志如下

18/10/07 06:00:53  WARN  SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
18/10/07 06:00:53  WARN  SparkConf: Detected deprecated memory fraction settings: [spark.shuffle.memoryFraction, spark.storage.memoryFraction, spark.storage.unrollFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended).
18/10/07 06:00:54  INFO  log: Logging initialized @47766ms
18/10/07 06:00:54  INFO  Server: jetty-9.2.z-SNAPSHOT
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@4a481728{/jobs,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@4ace284d{/jobs/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@77fa9381{/jobs/job,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@76d72b5b{/jobs/job/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@7cac93fe{/stages,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@479b5066{/stages/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@64deb58f{/stages/stage,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2b1cd7bc{/stages/stage/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@33379242{/stages/pool,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2e32fc22{/stages/pool/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@3f5156a6{/storage,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@72bdd7c7{/storage/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@7d17ee50{/storage/rdd,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@44fdce3c{/storage/rdd/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@712c5463{/environment,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@6d33a66e{/environment/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2f14b0f6{/executors,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5b20706{/executors/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@16aed49f{/executors/threadDump,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@66de00f2{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@6c479fdf{/static,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@79d82f66{/,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@19b206ee{/api,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@ffd4cba{/jobs/job/kill,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5f1483fd{/stages/stage/kill,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ServerConnector: Started Spark@21263314{HTTP/1.1}{10.1.237.119:22640}
18/10/07 06:00:54  INFO  Server: Started @47955ms
18/10/07 06:06:46  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5d5574c7{/metrics/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2328ec73{/SQL,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@58021c3c{/SQL/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@7bda01da{/SQL/execution,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5c997de8{/SQL/execution/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@15d58530{/static/sql,null,AVAILABLE,@Spark}
18/10/07 06:06:48  WARN  SessionState: load mapred-default.xml, HIVE_CONF_DIR env not found!
18/10/07 06:06:50  WARN  SessionState: load mapred-default.xml, HIVE_CONF_DIR env not found!

从日志来看，启动某个服务器大约需要 6 分钟？

18/10/07 06:00:54  INFO  Server: Started @47955ms
18/10/07 06:06:46  INFO  ContextHandler: Started

我会问这里可能会发生什么，为什么需要这么长时间？

标签： apache-spark

解决方案

我认为启动服务器只需要 47 秒，建议您查看链接以解决未找到 HIVE_CONF_DIR env

推荐阅读