前几天在本地开发调试Flink程序时,在WebUI页面无法查看jobManager日志或者taskManager日志,点击会在控制台报如下错误:
18:23:07.953 [flink-rest-server-netty-worker-thread-1] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor 9c9607af-a26b-47aa-89e3-7db9d89b89fb.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_341]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_341]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) ~[?:1.8.0_341]
......
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [flink-shaded-netty-4.1.39.Final-11.0.jar:?]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-shaded-netty-4.1.39.Final-11.0.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_341]
Caused by: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByFilePath(TaskExecutor.java:1747) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByType(TaskExecutor.java:1006) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
.......
// #########################
18:28:27.129 [flink-akka.actor.default-dispatcher-2] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler - Failed to transfer file from TaskExecutor dd0863a3-721d-47d2-b5fd-b4f5b778ec45.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_341]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_341]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) ~[?:1.8.0_341]
......
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [akka-actor_2.12-2.5.21.jar:2.5.21]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [akka-actor_2.12-2.5.21.jar:2.5.21]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [akka-actor_2.12-2.5.21.jar:2.5.21]
Caused by: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByFilePath(TaskExecutor.java:1747) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByType(TaskExecutor.java:1006) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
......
解决办法如下:
1、引入日志配置,包括pom文件中的依赖和 src/main/resources 目录下的日志文职文件。下面以log4j2.xml为例展示日志配置:
<?xml version="1.0" encoding="UTF-8"?>
<configuration monitorInterval="5">
<Properties>
<property name="LOG_PATTERN" value="%date{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n" />
<property name="LOG_LEVEL" value="INFO" />
</Properties>
<appenders>
<console name="console" target="SYSTEM_OUT">
<PatternLayout pattern="${LOG_PATTERN}"/>
<ThresholdFilter level="${LOG_LEVEL}" onMatch="ACCEPT" onMismatch="DENY"/>
</console>
<File name="log" fileName="tmp/log/job.log" append="false">
<PatternLayout pattern="%d{HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n"/>
</File>
</appenders>
<loggers>
<root level="${LOG_LEVEL}">
<appender-ref ref="console"/>
<appender-ref ref="log"/>
</root>
</loggers>
</configuration>
2、在flink程序开始初始化env时增加log_path相关的配置项,如下所示:
Configuration conf = new Configuration();
conf.setString(RestOptions.BIND_PORT,"8081");
conf.setString(WebOptions.LOG_PATH,"tmp/log/job.log");
conf.setString(ConfigConstants.TASK_MANAGER_LOG_PATH_KEY,"tmp/log/job.log");
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
然后重新启动程序就可以WebUI正常查看日志了,如图:文章来源:https://www.toymoban.com/news/detail-519969.html
文章来源地址https://www.toymoban.com/news/detail-519969.html
总结:引入日志配置(logback、log4j都可以),开启日志文件的appender和logger,然后将日志文件路径引入到初始化env的conf中就可以了。
到了这里,关于Flink本地运行WebUI日志问题的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!