mapreduce - MapReduce - 用 Mokito 模拟
问题描述
我有一个要编写测试用例的减速器类:
减少类:
public class MyReducer extends Reducer<Text, Text, NullWritable, Text> {
private static final Logger LOG = LogManager.getLogger(MyReducer.class);
public static List<String> l1 = new ArrayList<String>();
String id = null;
private MultipleOutputs<NullWritable, Text> mos;
@Override
public void setup(final Context context) throws IOException, InterruptedException {
mos = new MultipleOutputs<NullWritable, Text>(context);
final Path[] uris = DistributedCache.getLocalCacheFiles(context.getConfiguration());
try {
final BufferedReader readBuffer1 = new BufferedReader(new FileReader(uris[0].toString()));
String line;
while ((line = readBuffer1.readLine()) != null) {
l1.add(line);
}
readBuffer1.close();
} catch (Exception e) {
LOG.error(e);
}
}
public void reduce(final Text key, final Iterable<Text> values, final Context context)
throws IOException, InterruptedException {
final String[] key1 = key.toString().split("-");
final String keyA = key1[10];
final String date = key1[1];
/* Some condition check */
mos.write(NullWritable.get(), new Text(inputEventValue), keyA + "//date=" +
date.substring(0, 4) + "-" + date.substring(4, 6));
}
@Override
public void cleanup(final Context context) throws IOException, InterruptedException {
mos.close();
}
}
测试用例看起来像:
@RunWith(MockitoJUnitRunner.class)
public class MyTest {
@Mock
private MyReducer.Context mockContext;
MyReducer reducer;
MultipleOutputs<NullWritable, Text> mos;
@Before
public void setUp() {
reducer = new MyReducer();
}
@Test
public void myReducerTest() throws Exception {
MyReducer spy = PowerMockito.spy(new MyReducer());
doNothing().when(spy).setup(mockContext);
mos = new MultipleOutputs<NullWritable, Text>(mockContext);
List<Text> sline = new ArrayList<>() ;
List<String> l1 = new ArrayList<String>();
l1.add(“1234”);
sline.add(new Text(“xyz”));
Whitebox.setInternalState(MyReducer.class,”l1", l1);
Whitebox.setInternalState(MyReducer.class,"mos",mos);
reducer.reduce(new Text(“xyz-20200101-1234),sline,mockContext);
}
@After
public void tearDown() throws Exception {
/*
* this will do the clean up part
*/
verifyNoMoreInteractions(mockContext);
}
在 Debug 模式下运行时,它会转到 reducer 的 reduce 方法,并因 NullPointerException 而失败,mos write 语句在哪里?
完整的堆栈跟踪:
java.lang.NullPointerException
at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getNamedOutputsList(MultipleOutputs.java:196)
at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.<init>(MultipleOutputs.java:324)
at MyTest.myeducerTest
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:86)
at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:94)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:294)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:127)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:282)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:207)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:146)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:120)
at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:118)
at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:101)
at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:53)
at org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:53)
将 mos 错误模拟为 mos 不是静态的。
任何建议。
Junit - ReduceDriver, withInput, withOutput,testRun doesn't work.
谢谢。
我尝试按照建议模拟多个输出:
导入 org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
@Mock private MyReducer.Context mockContext;
List<String> namedOut = new ArrayList<>();
namedOut.add("NM1");
namedOut.add("NM2");
MultipleOutputs spy = PowerMockito.spy(new MultipleOutputs<>(mockContext)); when(spy, "getNamedOutputsList(mockContext)").thenReturn(namedOut);
但这给了我错误:org.powermock.reflect.exceptions.MethodNotFoundException:没有找到名称为“getNamedOutputsList(()anyObject())”的方法,参数类型:[]在类org.apache.hadoop.mapreduce.lib.output .多个输出。
解决方案
看起来你没有定义mockContext.getContext()
你的测试应该返回什么,所以它返回null
并失败。
基于此源代码,方法如下所示(因此您可能会使用不同的版本):
private static List<String> getNamedOutputsList(JobContext job) {
List<String> names = new ArrayList<String>();
StringTokenizer st = new StringTokenizer(
job.getConfiguration().get(MULTIPLE_OUTPUTS, ""), " ");
while (st.hasMoreTokens()) {
names.add(st.nextToken());
}
return names;
}
JobContext
似乎是指您的 mock Reducer.Context mockContext
,因此您需要定义适当的行为,以便它返回应该返回的内容。
请注意,此调用源自 的构造函数MultipleOutputs
。
还要注意getCountersEnabled
从构造函数调用并与上下文交互的静态方法。
将 mos 错误模拟为 mos 不是静态的。
您可能可以使用反射将模拟版本mos
放入您的MyReducer
课程中。
在此处查看有关如何模拟私有静态字段的一些示例。
编辑:
如果您尝试模拟 conig,请这样做:
Configuration config = Mockito.mock(Configuration.class);
when(mockContext.getConfiguration()).thenReturn(config);
据我所见get
,在配置对象上调用的总是提供默认值,因此键/值对是否在其中并不重要。
推荐阅读
- angular - 角 2 号管给出空白值
- java - 如何在字符串资源中设置值
- javascript - JS无法正常工作的列表项的反向顺序
- json - 如何将列表转换为 JSON?
- reactjs - 无法调用 React ExoticComponent
- kubernetes - Kubernetes 无法访问同一命名空间内的 Cassandra
- wordpress - 无论如何我可以在 web.config 文件中将网站从 .co.uk 更改为 .com
- shiny - 减少 TextInput 小部件中的输入
- python - 在python中将CSV循环导入无穷大
- reactjs - React 应用程序是如何构建的?