apache-flink - 在 EMR 集群中提交 Flink 作业：初始化集群入口点 YarnJobClusterEntrypoint 失败

Question

我正在使用 EMR 5.30.0 并尝试使用以下命令提交 Flink (1.10.0) 作业

flink run -m yarn-cluster /home/hadoop/flink--test-0.0.1-SNAPSHOT.jar

我收到以下错误：

Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 

在查看工作节点和作业管理器日志上的日志后，看起来存在端口冲突

2020-06-17 21:40:51,199 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Could not start cluster entrypoint YarnJobClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnJobClusterEntrypoint.
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:187)
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
        at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:119)
Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
        at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:261)
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:215)
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
        at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
        ... 2 more
Caused by: java.net.BindException: Could not start rest endpoint on any port in port range 8081
        at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:219)
        at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:165)
        ... 9 more

似乎为此打开了 JIRA Ticket ( https://issues.apache.org/jira/browse/FLINK-15394 )（尽管它适用于 Flink 的 1.9 版本），建议的解决方案是使用端口范围进行休息。 Flink 配置文件中的绑定端口。

然而在 1.10 版本的 Flink 中，我们只有以下 Yan Conf YML 文件

rest.port: 8081

我面临的另一个问题是我使用 AWS 控制台并通过添加步骤 ui 提交了多个 Flink 作业（多次相同的作业）。只有一项作业成功，其余作业因上面发布的错误而失败。当我转到 Flink UI 时，它根本没有显示任何作业。

想知道是否每个提交的作业都试图创建 Flink Yarn 会话而不是使用现有的会话。

谢谢萨蒂什

标签： apache-flinkamazon-emrflink-streaming

Answer 1

我能够解决它。似乎有端口冲突，我必须使用一系列端口并注释掉 rest.port: 8081

#rest.port: 8081
rest.bind-port: 50100-50200

谢谢