[Flink] Flink On Yarn（yarn-session.sh）启动错误-Toy模板网

这篇具有很好参考价值的文章主要介绍了[Flink] Flink On Yarn（yarn-session.sh）启动错误。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

在Flink上启动 yarn-session.sh时出现 The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.错误。

版本说明：

Hadoop： 3.3.4

Flink：1.17.1

问题

在Flink On Yarn上启动yarn-session.sh时出现如下错误：

ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli        [] - Error while running the Flink session.

org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
	at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:437) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:608) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:869) ~[flink-dist-1.17.1.jar:1.17.1]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_231]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_231]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:869) [flink-dist-1.17.1.jar:1.17.1]
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.
	at org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:338) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:567) ~[flink-dist-1.17.1.jar:1.17.1]
	at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:430) ~[flink-dist-1.17.1.jar:1.17.1]
	... 7 more
------------------------------------------------------------
 The program finished with the following exception:
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
	at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:437)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:608)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:869)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:869)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.
	at org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:338)
	at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:567)
	at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:430)
	... 7 more

原因

在yarn-site.xml文件中配置了所有可能相关的参数，重启yarn服务，执行yarn-session.sh错误依旧：

	<property>
    <name>yarn.containers.vcores</name>
    <value>8</value>
  </property>

  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>4</value>
  </property>

  <property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>2</value>
  </property>

在看yarn cluster上的信息时突然发现Unhealth Nodes，然后查看了具体信息：
[Flink] Flink On Yarn（yarn-session.sh）启动错误,大数据,Hadoop,Flink,flink,java,hadoop,yarn-session
具体原因就是磁盘使用空间占比超过了90了（yarn默认为90），则认为不健康，不健康相当于这个节点不可用，由于本地只有一个节点，所以相当于整个集群不可用，于是就出现了开头的错误信息。
[Flink] Flink On Yarn（yarn-session.sh）启动错误,大数据,Hadoop,Flink,flink,java,hadoop,yarn-session

解决

根据Health-report的提示，在yarn-site.xml中添加了如下参数：

	<property>
    <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
    <value>99</value>
  </property>

重启yarn，再查看节点状态为正常了，再执行flink的yarn-session.sh就可以正常启动了。
[Flink] Flink On Yarn（yarn-session.sh）启动错误,大数据,Hadoop,Flink,flink,java,hadoop,yarn-session

总结

在Flink中使用yarn-session时，如果出现yarn相关的错误，可以到Yarn的WebUI上查看可能的Unhealth-report和具体的错误信息，再根据具体信息调整配置后不断调试，直到解决问题。文章来源地址https://www.toymoban.com/news/detail-635597.html

到了这里，关于[Flink] Flink On Yarn（yarn-session.sh）启动错误的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！