时间戳

首页 > 解决方案 > 过滤 Apache Spark Log4J 消息以仅保留包含特定字符串的消息

java - 过滤 Apache Spark Log4J 消息以仅保留包含特定字符串的消息

问题描述

我有一些使用 Apache Spark 用 Ja​​va 编写的代码，我想禁用所有 Log4j 日志消息级别（ERROR、WARN 等）并只保留包含特定字符串的 INFO 级别。换句话说，我有这些日志：

19/04/21 19:09:40 INFO Instrumentation: [e10c0eb5] {"seed":26,"impurity":"entropy","featuresCol":"indexedFeatures","maxDepth":5,"labelCol":"indexedLabel","numTrees":10}
19/04/21 19:09:40 INFO Instrumentation: [752ad4c3] {"seed":26,"impurity":"entropy","featuresCol":"indexedFeatures","maxDepth":5,"labelCol":"indexedLabel","numTrees":12}
19/04/21 19:09:40 INFO Instrumentation: [d9d09329] {"seed":26,"impurity":"entropy","featuresCol":"indexedFeatures","maxDepth":5,"labelCol":"indexedLabel","numTrees":11}
19/04/21 19:09:40 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:112
19/04/21 19:09:40 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:112
19/04/21 19:09:40 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:112
19/04/21 19:09:40 INFO DAGScheduler: Got job 5 (take at DecisionTreeMetadata.scala:112) with 1 output partitions
19/04/21 19:09:40 INFO DAGScheduler: Final stage: ResultStage 6 (take at DecisionTreeMetadata.scala:112)
19/04/21 19:09:40 INFO DAGScheduler: Parents of final stage: List()
19/04/21 19:09:40 INFO DAGScheduler: Missing parents: List()

我只想保留以“INFO Instrumentation”开头的那些。我有这个示例代码：

    /*Logger.getLogger("org").setLevel(Level.OFF);
    Logger.getLogger("akka").setLevel(Level.OFF);*/








    SparkSession sparkSession = new SparkSession
            .Builder()
            .appName("Random Forest Classifier")
            .master("local[*]")
            .config("spark.ui.port", "40000")
            .getOrCreate();

我想更改前两个注释行以应用我的过滤器，这是否可能，如果可以，该怎么做？

我已经解决了我的问题；事实证明，我不需要任何过滤器或附加程序；我只是禁用了“org”和“akka”的所有日志，然后我只为这个类启用了 INFO 级别：“org.apache.spark.ml.util”，如下所示：

    Logger.getLogger("org").setLevel(Level.OFF);
    Logger.getLogger("akka").setLevel(Level.OFF);


    Logger.getLogger("org.apache.spark.ml.util").setLevel(Level.INFO);

谢谢你的帮助。

标签： javalogginglog4j

解决方案

您可以向包含您希望忽略的消息的记录器添加过滤器。
过滤器实现将过滤掉不符合给定谓词的消息，例如：

import org.apache.log4j.Level;
import org.apache.log4j.spi.Filter;
import org.apache.log4j.spi.LoggingEvent;

public class MyLog4jFilter extends Filter {

    /**
     * Custom filter to only log INFO events with the 'Instrumentation:' prefix in their message
     */
    @Override
    public int decide(LoggingEvent event) {
        if(event.getLevel() == Level.INFO && event.getMessage().trim().startsWith("Instrumentation:"))
           return ACCEPT;
        else 
           return DENY;
    }

}

参考：

推荐阅读