时间戳

首页 > 解决方案 > MapReduce 作业继续以 map = 0%、reduce = 0% 运行数小时

sql - MapReduce 作业继续以 map = 0%、reduce = 0% 运行数小时

问题描述

我正在运行一个 Hive 查询，它看起来像

create table table1 as select split(comments,' ') as words from table2;

评论栏有以空格分隔的字符串形式的评论评论。

当我运行此查询时，MapReduce 作业开始并继续以 Map 0% 运行数小时。在此过程中它不会给出任何错误。

hive> create table jw_1 as select split(comments,' ') from removed_null_values;
Query ID = xxx-190418201314_7781cf59-6afb-4e82-ab75-c7e343c4985e
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555607912038_0013, Tracking URL = http://xxx-VirtualBox:8088/proxy/application_1555607912038_0013/
Kill Command = /usr/local/bin/hadoop-3.2.0/bin/mapred job  -kill job_1555607912038_0013
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-18 20:13:30,568 Stage-1 map = 0%,  reduce = 0%
2019-04-18 20:14:31,140 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 39.6 sec
2019-04-18 20:15:31,311 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 101.64 sec
2019-04-18 20:16:31,451 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 146.5 sec
2019-04-18 20:17:31,684 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 212.08 sec

但是，当我尝试

select split(comments,' ') from table2;

我可以在 shell 中看到数组形式的注释。

["\"Lauren","was","promptly","responsive","in","advance","of","our","booking.","providing","a","lot","of","helpful","info.","And","she","stayed","in","contact","and","was","readily","available","prior","to","and","during","our","stay.","which","was","awesome.","The","location.","price","and","privacy","were","the","real","benefits."]

我还运行了一些其他查询，其中 MapReduce 作业完成并产生了所需的结果

我目前正在使用 Hive 3.1.1

基本上，我想创建一个包含单词的数组的新表，然后标记该列

我是 Hive 的新手，我正在对 35MB 的数据文件进行情感分析。

标签： sqlhadoophivemapreduce

解决方案

在第一种情况下，当转换为 MapReduce 时，您很可能没有完成 Hive 查询所需的资源。您必须查看 YARN 或 MR1 以确定您是否有足够的计算资源来运行 MapReduce 作业。

在第二个查询中，一些 Hive 查询触发器不会触发 MapReduce 作业，这就是它返回的原因。请参阅Hive 如何决定何时使用 map reduce 以及何时不使用？了解更多信息。

推荐阅读