java - Mapreduce:平均计算不起作用
问题描述
我正在尝试这个 map reduce 代码来计算平均值,但由于某种原因,平均值没有得到正确计算。这个想法是计算每年的平均电影评分
映射器代码
public class AverageRatingMapper extends Mapper<LongWritable, Text, Text, DoubleWritable>
{
//initialize the writable datatype variables
private final static DoubleWritable tempWritable = new DoubleWritable(0);
private Text ReleaseYear = new Text();
//Override the original map methods
@Override
//map takes in three parameters
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException
{
//creating an array called line of type String
//Split the values in the fields and store each value in one array element
String[] line = value.toString().split("\t");
//create a variable ID with type String and store the the 4th element (index 3) from array line. This contains the Year in th data file
String Year = line[3];
//Set the value of the year object created from the Text class to be the value of the Year read from the data file
ReleaseYear.set(Year);
//Create a variable, temp, of type double and convert the value stored in the 15th element (14th index) of the line array from String to Double and store it in temp
double temp = Double.parseDouble(line[14].trim());
//Store temp in tempWritable
tempWritable.set(temp);
//Emit Year and the average rating in tempWritable to the Reducer class
context.write(ReleaseYear, tempWritable);
}
}
减速机代码:
public class AverageRatingReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable>
{
//Create an arraylist of type double called ratingList
ArrayList<Double> ratingList = new ArrayList<Double>();
//Override reduce method
@Override
//reduce takes in thre parameters
public void reduce(Text key, Iterable<DoubleWritable> values, Context context)
throws IOException, InterruptedException
{
//create a variable SumofRatings of type double and initialize it to 0
double SumofRating = 0.0;
//use a for loop to store ratings in the ratingList array and sum the ratings in the SumofRatings variable
for(DoubleWritable value : values)
{
ratingList.add(value.get());
//calculate the cumulative sum
SumofRating = SumofRating + value.get();
}
//get the number of rating in the arrayList
int size = ratingList.size();
//calculate the average rating
double averageRating = SumofRating/size;
//Emit the year and the average rating to the output file
context.write(key, new DoubleWritable(averageRating));
}
}
主要课程:
public class AverageRating
{
public static void main(String[] args) throws Exception
{
//Create an object, conf, from the configuration class
Configuration conf = new Configuration();
if (args.length != 3)
{
System.err.println("Usage: MeanTemperature <input path> <output path>");
System.exit(-1);
}
//create an object, job, from the Job class
Job job;
//configure the parameters for the job
job = Job.getInstance(conf, "Average Rating");
//specify the driver class in the JAR file
job.setJarByClass(AverageRating.class);
//setting the input and output paths for the job
FileInputFormat.addInputPath(job, new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));
//Set the mapper and reducer for the job
job.setMapperClass(AverageRatingMapper.class);
job.setReducerClass(AverageRatingReducer.class);
//Set the key class (Text) and value class (DoubleWritable) for te job output data
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(DoubleWritable.class);
//Delete output if it exists
FileSystem hdfs = FileSystem.get(conf);
Path outputDir = new Path(args[2]);
if(hdfs.exists(outputDir))
{
hdfs.delete(outputDir, true);
}
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
根据要求提供数据样本:
imdbtitle | 标题 | 年 | 平均评分 |
---|---|---|---|
tt0000009 | XYZ | 1911 | 6.9 |
tt0001892 | 二维码 | 1912年 | 6.2 |
tt0002154 | 美国广播公司 | 1912年 | 8.2 |
tt0000458 | JKL | 1913年 | 6.3 |
tt0015263 | 生长激素 | 1913年 | 7.1 |
tt0000053 | 巴解组织 | 1912年 | 4.9 |
注意:还有更多列。我刚刚添加了重要的
第一年的结果显示正确,但第二年和第三年完全不准确。
- 1911 年 6.9
- 1912 5.25
- 1913 2.2
有人可以帮帮我吗!
解决方案
您实际上并不需要ArrayList
Reduce 函数中的 an,因为每个 reducer 都会获取按给定键分组的所有值(因此在这种情况下,每个 reducer 都有一年的所有评级)。
此外,声明Reduce 函数体的ArrayList
外部类型会带来麻烦,因为您使用此列表只是为了计算 reducer 上已经存在的值的数量。我的猜测是,在不同的 reducer 扫描之间,列表中会不断填充来自下一个键(又名年份,此处)的评级的评级。所以第一个键值对是正确的,但后面的一切都不是因为列表中的元素数量不断增加。
您可以保持它更传统,方法是使用一个简单的int
变量命名numOfRatings
,该变量只计算值的数量(即此处给出的特定年份的评分),并使用该变量进行除以找到每年的平均评分。
public static class AverageRatingReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable>
{
public void reduce(Text key, Iterable<DoubleWritable> values, Context context) throws IOException, InterruptedException
{
// create a variable SumofRatings of type double and initialize it to 0
double sumOfRating = 0.0;
int numOfRatings = 0;
// use a for loop to store ratings in the ratingList array and sum the ratings in the SumofRatings variable
for(DoubleWritable value : values)
{
sumOfRating += value.get();
numOfRatings++;
}
// calculate the average rating
double averageRating = sumOfRating/numOfRatings;
// emit the year and the average rating to the output file
context.write(key, new DoubleWritable(averageRating));
}
}
推荐阅读
- cuda - NSight 调试在任何内存分配上都非常慢
- c# - 连接两个列表但保留源对象信息
- javascript - 在 vue / vuex 中对页面进行刷新后如何保持数据持久化?
- llvm - 为什么 LLVM llc 编译器会抛出指令号错误?
- powershell - 无法从电子邮件中检索全文以进行比较,导出为 .csv 文件,仅获取正文的第一行
- php - Laravel id 从 URL 到变量
- c# - C# How to change width of border rapidly?
- python - 如何使用 x 轴作为“日期”的 seaborn 实现 Lineplot
- sql - Spring JPA : No converter data from DB in to DTO
- python - 是否可以使用外部 EXE 转换文件并在 python 中作为类文件对象未保存到磁盘输出?