java - java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider 当 flink 启动时
问题描述
我在 k8s 集群上运行 Flink 1.11,尝试更新 log4j-console.properties 文件时出现以下错误:
Starting Task Manager
Enabling required built-in plugins
Linking flink-s3-fs-hadoop-1.11.1.jar to plugin directory
Successfully enabled flink-s3-fs-hadoop-1.11.1.jar
sed: couldn't open temporary file /opt/flink/conf/sedl2dH0X: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedPLYAzY: Read-only file system
/docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied
sed: couldn't open temporary file /opt/flink/conf/sede0G5LW: Read-only file system
/docker-entrypoint.sh: 120: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting taskexecutor as a console application on host flink-taskmanager-c765c947c-qx68t.
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider
at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:158)
at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:69)
at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:102)
at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:77)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<clinit>(TaskManagerRunner.java:89)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
我的 log4j-console.properties:
rootLogger.level = INFO
#rootLogger.appenderRef.console.ref = ConsoleAppender
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.topic = test
appender.kafka.layout.type = JsonLayout
appender.kafka.layout.complete = false
appender.kafka.additional1.type = KeyValuePair
appender.kafka.additional1.key=app
appender.kafka.additional1.value=TEST
appender.kafka.additional2.type = KeyValuePair
appender.kafka.additional2.key=subsystem
appender.kafka.additional2.value=TEST
appender.kafka.additional3.type = KeyValuePair
appender.kafka.additional3.key=deployment
appender.kafka.additional3.value=TEST
appender.kafka.property.bootstrap.servers=***
rootLogger.appenderRef.console.ref = STDOUT
rootLogger.appenderRef.kafka.ref = Kafka
我使用“flink:1.11.1-scala_2.11-java11”docker 镜像并验证所有 log4j2 依赖项都在类路径中。
我还尝试从上面的基本映像创建一个新的 docker 映像,并将缺少的依赖项添加到其中,但什么也没发生。
解决方案
我也遭受了这个错误。这里的问题是,当任务管理器和作业管理器启动时,它们使用修改后的类路径运行,而不是您通过构建系统构建的 JAR。
请参阅flink 源代码constructFlinkClassPath
中的。为了证明这一点,请恢复 JSON 日志记录模式并在启动时检查 tm/jm 日志中的类路径。您会注意到您的 JAR 不在类路径中。
要解决此问题,您需要向tm/jm 节点内的文件夹提供依赖项(在这种情况下您需要jackson-core
jackson-annotations
and jackson-databind
) (该文件夹默认包含在 flink 类路径中)。lib
lib
如果您使用的是 docker,则可以在构建容器 ( RUN wget...
) 时执行此操作。
推荐阅读
- r - 将 pivot_wider 应用于列表中的数据框
- python - 如何读取(解码).tfrecords 文件,查看里面的图像并进行扩充?
- python - 深度优先搜索以在图中找到路径
- android-gradle-plugin - “无法找到请求目标的有效认证路径”android studio
- python - 更改文本文件中数字的位置 - 在 Python 中从第一到第二到第二?
- apache-camel - 使用 RouteReifer.adviceWith(..) 进行测试时,Camel 路线中的标头值丢失
- javascript - navigator.share 打开一个较小的菜单
- java - RecyclerView 首先在 NestedScrollView 内滚动
- python - Matplotlib - 以正方形绘制数据
- css - 使用 prefers-color-scheme 时防止重复 CSS 自定义属性