首页 > 技术文章 > 启动Flink的yarn-session模式出错

LzMingYueShanPao 2021-08-05 14:41 原文

  问题如下:bin/yarn-session.sh -d 报错

 The program finished with the following exception:

org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
    at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:425)
    at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:606)
    at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
    at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.
    at org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:326)
    at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:534)
    at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:418)
    ... 7 more

  解析:日志中显示的大概原因就行启动失败,资源不够,但是我启动的时候并没有指定任何参数,使用的是默认值,申请的资源大小绝对不会超过可用的资源大小,完全够用,所以报错原因跟运行时指定的参数无关,那么我们该如何解决呢?

  解决:之前测试时启动太多次Filnk的yarn-session模式,所以断定是临时文件过多导致,故删除HDFS中的Flink的临时文件即可,你的文件存在哪就删除哪,我存在/flink/yarn/ha目录下,所以直接删除即可,可以在flink的conf目录下的flink-conf.yaml查看HDFS的存储路径

  再次启动,发现启动成功

bin/yarn-session.sh -d

推荐阅读