log4j - 如何在纱线上配置 log4j？

问题描述

背景：纱线由cloudera manager安装。我使用 yarn 来运行我的 xxx.jar。

我使用 yarn 来运行我的 xxx.jar。众所周知，syslog文件将包含 log4j 相关的日志。stdout文件将包含一些输出，例如 System.out.println(...)。Blow 是stdout文件中的一些输出：

===============================================================
LogType:stdout
LogLastModifiedTime:Sun Oct 11 21:19:27 +0800 2020
LogLength:109238
LogContents:
log4j: Trying to find [container-log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@5c647e05.
log4j: Using URL [jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-3.0.0-cdh6.3.2.jar!/container-log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-3.0.0-cdh6.3.2.jar!/container-log4j.properties
log4j: Hierarchy threshold set to [ALL].
log4j: Parsing for [root] with value=[INFO,CLA, EventCounter].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named "CLA".
log4j: Parsing layout options for "CLA".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n].
log4j: End of parsing for "CLA".
log4j: Setting property [containerLogFile] to [syslog].
log4j: Setting property [totalLogFileSize] to [1048576].
log4j: Setting property [containerLogDir] to [/yarn/container-logs/application_1602420941906_0002/container_1602420941906_0002_01_000001].
log4j: setFile called: /yarn/container-logs/application_1602420941906_0002/container_1602420941906_0002_01_000001/syslog, true
log4j: setFile ended
log4j: Parsed "CLA" options.
log4j: Parsing appender named "EventCounter".
log4j: Parsed "EventCounter" options.
log4j: Parsing for [org.apache.hadoop.mapreduce.task.reduce] with value=[INFO,CLA].
log4j: Level token is [INFO].
log4j: Category org.apache.hadoop.mapreduce.task.reduce set to INFO
log4j: Parsing appender named "CLA".
log4j: Appender "CLA" was already parsed.
log4j: Handling log4j.additivity.org.apache.hadoop.mapreduce.task.reduce=[false]
log4j: Setting additivity for "org.apache.hadoop.mapreduce.task.reduce" to false
log4j: Parsing for [org.apache.hadoop.mapred.Merger] with value=[INFO,CLA].
log4j: Level token is [INFO].
log4j: Category org.apache.hadoop.mapred.Merger set to INFO
log4j: Parsing appender named "CLA".
log4j: Appender "CLA" was already parsed.
log4j: Handling log4j.additivity.org.apache.hadoop.mapred.Merger=[false]
log4j: Setting additivity for "org.apache.hadoop.mapred.Merger" to false
log4j: Finished configuring.
Launcher AM configuration loaded

从日志中，我们可以看到 log4j 属性来自jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-3.0.0-cdh6.3.2.jar!/container-log4j.properties. 我也下载hadoop-yarn-server-nodemanager-3.0.0-cdh6.3.2.jar. 的内容container-log4j.properties是：

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License. See accompanying LICENSE file.
#
# Define some default values that can be overridden by system properties
hadoop.root.logger=DEBUG,CLA
yarn.app.mapreduce.shuffle.logger=${hadoop.root.logger}

# Define the root logger to the system property "hadoop.root.logger".
log4j.rootLogger=${hadoop.root.logger}, EventCounter

# Logging Threshold
log4j.threshold=ALL

#
# ContainerLog Appender
#

#Default values
yarn.app.container.log.dir=null
yarn.app.container.log.filesize=100

log4j.appender.CLA=org.apache.hadoop.yarn.ContainerLogAppender
log4j.appender.CLA.containerLogDir=${yarn.app.container.log.dir}
log4j.appender.CLA.containerLogFile=${hadoop.root.logfile}
log4j.appender.CLA.totalLogFileSize=${yarn.app.container.log.filesize}
log4j.appender.CLA.layout=org.apache.log4j.PatternLayout
log4j.appender.CLA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c: %m%n

log4j.appender.CRLA=org.apache.hadoop.yarn.ContainerRollingLogAppender
log4j.appender.CRLA.containerLogDir=${yarn.app.container.log.dir}
log4j.appender.CRLA.containerLogFile=${hadoop.root.logfile}
log4j.appender.CRLA.maximumFileSize=${yarn.app.container.log.filesize}
log4j.appender.CRLA.maxBackupIndex=${yarn.app.container.log.backups}
log4j.appender.CRLA.layout=org.apache.log4j.PatternLayout
log4j.appender.CRLA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c: %m%n

log4j.appender.shuffleCLA=org.apache.hadoop.yarn.ContainerLogAppender
log4j.appender.shuffleCLA.containerLogDir=${yarn.app.container.log.dir}
log4j.appender.shuffleCLA.containerLogFile=${yarn.app.mapreduce.shuffle.logfile}
log4j.appender.shuffleCLA.totalLogFileSize=${yarn.app.mapreduce.shuffle.log.filesize}
log4j.appender.shuffleCLA.layout=org.apache.log4j.PatternLayout
log4j.appender.shuffleCLA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c: %m%n

log4j.appender.shuffleCRLA=org.apache.hadoop.yarn.ContainerRollingLogAppender
log4j.appender.shuffleCRLA.containerLogDir=${yarn.app.container.log.dir}
log4j.appender.shuffleCRLA.containerLogFile=${yarn.app.mapreduce.shuffle.logfile}
log4j.appender.shuffleCRLA.maximumFileSize=${yarn.app.mapreduce.shuffle.log.filesize}
log4j.appender.shuffleCRLA.maxBackupIndex=${yarn.app.mapreduce.shuffle.log.backups}
log4j.appender.shuffleCRLA.layout=org.apache.log4j.PatternLayout
log4j.appender.shuffleCRLA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c: %m%n

################################################################################
# Shuffle Logger
#
log4j.logger.org.apache.hadoop.mapreduce.task.reduce=${yarn.app.mapreduce.shuffle.logger}
log4j.additivity.org.apache.hadoop.mapreduce.task.reduce=false
# Merger is used for both map-side and reduce-side spill merging. On the map
# side yarn.app.mapreduce.shuffle.logger == hadoop.root.logger
#
log4j.logger.org.apache.hadoop.mapred.Merger=${yarn.app.mapreduce.shuffle.logger}
log4j.additivity.org.apache.hadoop.mapred.Merger=false
#
# Event Counter Appender
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
#
log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter

所以，我认为 log4j 相关的日志将保存在syslog文件中。但是在纱线中运行我的 xxx.jar 后，syslog 中没有任何内容，我的 xxx.jar 还包含一些代码，例如：System.out.println("my demo");。运行我的 xxx.jar 后，我可以在标准输出中找到输出“我的演示”。

所以我的问题是为什么log4j相关的输出不能打印出来？是否需要其他配置？

标签： log4jhadoop-yarn

log4j - 如何在纱线上配置 log4j？

问题描述

解决方案

推荐阅读