junit - Java GC Overhead Limit Error Using JUnit 5 TestFactory with large stream of dynamicTest (DynamicContainer / DynamicNode)

问题描述

尝试使用 maven-surefire-plugin 2.21.0 创建和运行大量 JUnit 5 动态测试时，我面临以下问题

[错误] 超过 GC 开销限制 [错误] org.apache.maven.surefire.booter.SurefireBooterForkException：分叉进程中出现错误 [错误] org.apache.maven.plugin.surefire 超过 GC 开销限制 [错误] .booterclient.ForkStarter.fork(ForkStarter.java:673) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:535) [ERROR] at org.apache.maven.plugin .surefire.booterclient.ForkStarter.run(ForkStarter.java:280) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245) [ERROR] at org.apache.maven .plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1124) [错误] 在 org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954）[错误]在org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute（AbstractSurefireMojo.java:832）[错误]在org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo（DefaultBuildPluginManager.java:134）[错误] 在 org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) [错误] 在 org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154) [错误] 在org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146) [错误] 在 org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) [错误] 在 org. org.apache.maven.lifecycle.internal.builder.singlethreaded 中的 apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) [错误]。SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) [错误] 在 org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

下面是我的代码，它是一个数据库比较工具。基本上我正在做的是，从两个数据库流式传输数据并比较记录，如果两条记录中的任何一条存在差异，则测试失败，因此在数据库中创建的动态测试数量与记录的数量一样多（~14M），相当大，不确定是否要在这种规模上使用动态测试。非常感谢这方面的任何帮助:)

 @TestFactory
Stream<DynamicNode> dynamicTestsWithContainers() throws SQLException {
    return tableNameProvider()//List of db tables ~100
            .map(tableName -> dynamicContainer(tableName, dynamicNodeStream(tableName)));
}

private Stream<DynamicNode> dynamicNodeStream(String tableName) {

    try {
        System.out.println("Testing " + tableName);
        Stream<Row> rows = sourceRepo.rows(tableName);
        Stream<List<Row>> batchRows = batch(rows, 10000);
        Optional<TableSchema> tableInfo = sourceRepo.getTableSchema(tableName);


        final Stream<DynamicNode> dynamicNodeStream = batchRows
                .flatMap(batch -> {

                    Map<String, Row> sourceRowsMap = buildRowMap(batch, tableInfo.get());//HashMap with 10000 Objects
                    Map<String, Row> targetRowsMap = targetRepo.getTargetDBRows(sourceRowsMap, tableInfo.get());//HashMap with 10000 Objects
                    Set<String> commonKeys = Sets.intersection(sourceRowsMap.keySet(), targetRowsMap.keySet());

                    final Stream<DynamicTest> dynamicTestStream = Streams.concat(
                            Stream.of(
                                    dynamicTest("All source records should be present in target DB", () -> assertThat(targetRowsMap.keySet())
                                            .as("Comparing " + sourceRepo.getServerName() + " against " + targetRepo.getServerName())
                                            .hasSameElementsAs(sourceRowsMap.keySet()))
                            ),
                            commonKeys
                                    .stream()
                                    .map(rowKey -> dynamicTest(tableName + " Row with #" + rowKey + " should be same in target DB",
                                            () -> assertThat(targetRowsMap.get(rowKey).getRowData())
                                                    .isEqualToComparingFieldByFieldRecursively(sourceRowsMap.get(rowKey).getRowData())
                                    ))

                    );

                    return dynamicTestStream;
                });

        return dynamicNodeStream;
    } catch (Exception e) {
        throw new RuntimeException("Error running tests on table " + tableName, e);
    }
}

标签： junitjava-8java-streammaven-surefire-pluginjunit5

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <argLine>-XX:+UseConcMarkSweepGC</argLine> </configuration> </plugin>

junit - Java GC Overhead Limit Error Using JUnit 5 TestFactory with large stream of dynamicTest (DynamicContainer / DynamicNode)

问题描述

解决方案

推荐阅读