linkedin - WhereHows, wherehows-common 构建错误（创建的工作太多）
问题描述
尝试WhereHows根据此处的文档（https://github.com/linkedin/WhereHows/blob/master/wherehows-docs/getting-started.md#build-the-source-code）在 Centos7 上安装 Linkedin 并运行构建使用 gradle 构建时与 wherehows-common:test 相关的错误。
> Task :wherehows-common:test
Putting task artifact state for task ':wherehows-common:test' into context took 0.001 secs.
Executing task ':wherehows-common:test' (up-to-date check took 0.041 secs) due to:
No history is available.
Starting process 'Gradle Test Executor 2'. Working directory: /home/me/WhereHows/wherehows-common Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -javaagent:build/tmp/expandedArchives/org.jacoco.agent-0.7.1.201405082137.jar_r1v2o0qii4g3mwygeto49hza/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,dumponexit=true,output=file,jmx=false -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/me/.gradle/caches/4.0.2/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 2'
Successfully started process 'Gradle Test Executor 2'
Gradle suite > Gradle test > FileWriterTest. STANDARD_OUT
hdfs:///a/b/cnull{"sample": [aaa, bbb, ccc] }
Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs FAILED
java.lang.AssertionError: expected [2] but found [47]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:513)
at org.testng.Assert.assertEqualsImpl(Assert.java:135)
at org.testng.Assert.assertEquals(Assert.java:116)
at org.testng.Assert.assertEquals(Assert.java:389)
at org.testng.Assert.assertEquals(Assert.java:399)
at wherehows.common.utils.JobsUtilTest.testGetEnabledJobs(JobsUtilTest.java:91)
.....
更改引用的 JobsUtilTest.java 中的 testGetEnabledJobs() 函数（https://github.com/linkedin/WhereHows/blob/master/wherehows-common/src/test/java/wherehows/common/utils/JobsUtilTest.java #L77）看起来像（注意我添加了一些打印语句，以查看运行测试时作业字典实际看到的内容）：
public void testGetEnabledJobs() throws IOException, ConfigurationException {
String propertyStr1 = "job.class=test\n" + "job.cron.expr=0 0 1 * * ? *\n" + "#job.disabled=1\n" + "job.type=TEST1";
String propertyStr2 = "job.class=test\n" + "#job.disabled=1\n";
Path path1 = createPropertiesFile(propertyStr1);
Path path2 = createPropertiesFile(propertyStr2);
String filename1 = jobNameFromPath(path1);
String filename2 = jobNameFromPath(path2);
String dir = path1.getParent().toString();
Map<String, Properties> jobs = getEnabledJobs(dir);
System.out.println("Displaying jobs map");
jobs.forEach((key, value) -> System.out.println(key + ":" + value));
Assert.assertEquals(jobs.size(), 2);
Assert.assertEquals(jobs.get(filename1).getProperty("job.class"), "test");
Assert.assertEquals(jobs.get(filename1).getProperty("job.disabled", ""), "");
Assert.assertEquals(jobs.get(filename2).getProperty("job.class"), "test");
Files.deleteIfExists(path1);
Files.deleteIfExists(path2);
}
我看到了输出
> Task :wherehows-common:test
Gradle suite > Gradle test > FileWriterTest. STANDARD_OUT
hdfs:///a/b/cnull{"sample": [aaa, bbb, ccc] }
Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs STANDARD_OUT
temp1911939516358009277:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp6499017376697657088:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp5966987645065306263:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
temp2019798894975335038:{job.class=test}
temp7378838424107087948:{job.class=test}
temp8013153217448339252:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp4015785892016530294:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8886920588183826697:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp7303423395129181364:{job.class=test}
temp7377443665642582850:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp8117994429105024533:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp1240481287927638671:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp7839672993673219735:{job.class=test}
temp7259952657422506556:{job.class=test, job.type=TEST3}
temp7162637183580461694:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp3707419465874500296:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp7235208969315338463:{job.class=test, job.type=TEST3}
temp3853444188071964131:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp4656083272550419936:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp4446014595794536877:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp9150954469641724228:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp1130156704067895902:{job.class=test, job.type=TEST3}
temp780755426601798559:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2434470881148405176:{job.class=test}
temp3689536441531931750:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
temp5615428018432710013:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp8234020029970009016:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2936711116252973718:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2794065451765912556:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8170223748305820520:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8108053620028090977:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp7690999652582845774:{job.class=test, job.type=TEST3}
temp8403527686775796761:{job.class=test}
temp1264073914844226751:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp4824707996781445098:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp4743553641259724497:{job.class=test, job.type=TEST3}
temp6976197742641904750:{job.class=test}
temp1730302984164232378:{job.class=test, job.type=TEST3}
temp1842348735636276811:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp1493041975452434193:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
temp7759939924299343315:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2004080563168505783:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp1339889112521909105:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp930690436274101959:{job.class=test, job.type=TEST3}
temp8950010240587776997:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp1690669672245318742:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp7618814921091278264:{job.class=test}
temp5153364346961309871:{job.class=test}
temp2706126941897920939:{job.class=test, job.type=TEST3}
temp7879422791151165293:{job.class=test, job.type=TEST3}
temp3500152346117134646:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp749236937622369939:{job.class=test}
temp2402271667647981403:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp3731569061193901687:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8046137399375181735:{job.class=test, job.type=TEST3}
temp2787658903538848691:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp2549570320110302073:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp4717503488348299098:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
temp4558936186853777244:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
temp4977131264038603527:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp7320554126669992802:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8291109043177891111:{job.class=test}
temp3078133305981963798:{job.class=test, job.type=TEST3}
temp4497083049478812420:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp3224402460504056397:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8457758602058485743:{job.class=test, job.type=TEST3}
temp1336150142768316636:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp577381353130877703:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp1077045599840249203:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp1960639765588269368:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp4407187188971325393:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2303367009745054129:{job.class=test, job.type=TEST3}
temp8807915018871869160:{job.class=test}
temp8784520992289559963:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2670455510663453671:{job.class=test}
temp1910488462419291875:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp7195699649615422636:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp1493552070734553408:{job.class=test}
temp9092541414990236719:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp6892241678814671661:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8422662429932217118:{job.class=test}
temp7706138265443041124:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp5607003154298079581:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8785154796880304252:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp9219444719015202876:{var3=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var2=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/, var1=foo}
temp1177833530220968687:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp8577396298139877100:{job.class=test, job.type=TEST2,TEST3, job.cron.expr=0 0 1 * * ? *}
temp979103679152008854:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp2369961801884435918:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp3349449102692687060:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp430974220547583703:{job.class=test, job.type=TEST3}
temp7799011812497499444:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs FAILED
java.lang.AssertionError: expected [2] but found [92]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:513)
at org.testng.Assert.assertEqualsImpl(Assert.java:135)
at org.testng.Assert.assertEquals(Assert.java:116)
at org.testng.Assert.assertEquals(Assert.java:389)
at org.testng.Assert.assertEquals(Assert.java:399)
at wherehows.common.utils.JobsUtilTest.testGetEnabledJobs(JobsUtilTest.java:93)
所以应该只有 2 个工作，但有一堆重复的工作。这里似乎有什么问题？
使用此工具的其他人是否知道这里可能出现的问题？
** 有用的系统和环境。信息：
[me@mapr07 WhereHows]$ tree -L 2 wherehows-etl/extralibs/
wherehows-etl/extralibs/
├── dist
│ ├── changes.txt
│ ├── compile_run_demo.txt
│ ├── demos
│ ├── GeneralSQLParser_UserGuide.pdf
│ ├── gsp.jar
│ ├── javadoc
│ └── test
├── gsp.jar
├── gsp_java_trial_1_9_2_3.zip
├── ojdbc7.jar
├── orai18n.jar
├── README.md
├── readme.txt
├── tdgssconfig.jar
├── terajdbc4.jar
└── TeraJDBC__indep_indep.16.20.00.06.tar
4 directories, 13 files
[me@mapr07 WhereHows]$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
[me@mapr07 WhereHows]$ javac -version
javac 1.8.0_171
[me@mapr07 WhereHows]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre
[me@mapr07 WhereHows]$ cat /proc/version
Linux version 3.10.0-693.21.1.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Wed Mar 7 19:03:37 UTC 2018
解决方案
解决了这个问题。在测试函数的代码中添加一些打印语句，如下所示：
@Test
public void testGetEnabledJobs() throws IOException, ConfigurationException {
String propertyStr1 = "job.class=test\n" + "job.cron.expr=0 0 1 * * ? *\n" + "#job.disabled=1\n" + "job.type=TEST1";
String propertyStr2 = "job.class=test\n" + "#job.disabled=1\n";
Path path1 = createPropertiesFile(propertyStr1);
Path path2 = createPropertiesFile(propertyStr2);
String filename1 = jobNameFromPath(path1);
String filename2 = jobNameFromPath(path2);
System.out.println("filenames: " + filename1 + "\n" + filename2);
String dir = path1.getParent().toString();
System.out.println("dir="+dir);
Map<String, Properties> jobs = getEnabledJobs(dir);
System.out.println("Displaying jobs map");
jobs.forEach((key, value) -> System.out.println(key + ":" + value));
Assert.assertEquals(jobs.size(), 2);
Assert.assertEquals(jobs.get(filename1).getProperty("job.class"), "test");
Assert.assertEquals(jobs.get(filename1).getProperty("job.disabled", ""), "");
Assert.assertEquals(jobs.get(filename2).getProperty("job.class"), "test");
Files.deleteIfExists(path1);
Files.deleteIfExists(path2);
}
我看到了输出：
> Task :wherehows-common:test
Putting task artifact state for task ':wherehows-common:test' into context took 0.0 secs.
Executing task ':wherehows-common:test' (up-to-date check took 0.02 secs) due to:
No history is available.
Starting process 'Gradle Test Executor 3'. Working directory: /home/rvillanueva/WhereHows/wherehows-common Command: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -javaagent:build/tmp/expandedArchives/org.jacoco.agent-0.7.1.201405082137.jar_r1v2o0qii4g3mwygeto49hza/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,dumponexit=true,output=file,jmx=false -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/rvillanueva/.gradle/caches/4.0.2/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 3'
Successfully started process 'Gradle Test Executor 3'
Gradle suite > Gradle test > FileWriterTest. STANDARD_OUT
hdfs:///a/b/cnull{"sample": [aaa, bbb, ccc] }
Gradle suite > Gradle test > wherehows.common.utils.JobsUtilTest.testGetEnabledJobs STANDARD_OUT
filenames: temp7961565693796243498
temp1135071152526160059
dir=/tmp
Displaying jobs map
temp7961565693796243498:{job.class=test, job.type=TEST1, job.cron.expr=0 0 1 * * ? *}
temp1135071152526160059:{job.class=test}
......
查看 /tmp 目录。我看到了一堆旧的 temp*.job 文件。删除这些并重新运行似乎使我能够成功构建。
我认为的问题是这样的：
- 最初运行 gradlew build 命令，由于未设置而出错并退出
JAVA_HOME。此时，我认为 testGetEnabledJobs() 已经创建了它的断言期望并成功通过的 2 个 temp*.job 文件。 - 然后设置
JAVA_HOME并尝试再次构建。这次 testGetEnabledJobs() 在 /tmp 中创建了更多的 temp*.job 文件（在上一次构建尝试中未删除的文件之上）并得到了 testGetEnabledJobs() 错误，因为现在 /tmp 中的 temp*.job 文件比预期，导致错误并退出。 - 此时，每次重试构建都会表明断言计数越来越高于预期值。
无论如何，我认为应该更改此测试以删除它创建的 temp*.job 文件，如果它在相关断言处出错，否则我们将在 /tmp 目录中遇到复合问题。我刚刚修好了。
推荐阅读
- c# - 如何使用 C# 更改更新 XML 中的元素值？
- c++ - 具有 r 值的类似数组的功能
- javascript - 如何操作 CSS PaintWorklet 上下文的像素数据？
- laravel - 广播 laravel 事件和多个频道
- configuration - TYPO3的配置涉及哪些语言？
- python - 如何将python中的列表转换为numpy数组以放入张量流
- python - 如何使用另一个数组中的值更新 numpy 数组的所有值
- string - 如何将集合传递给函数参数？
- uwp - 未检测到 UWP Enter 键按下
- c# - 在 Angular 7 中配置和使用 Dotnet Core Web API