hive - 当查询具有 GROUP BY 子句时,Hive 查询抛出“来自 org.apache.hadoop.hive.ql.exec.tez.TezTask 的代码 2”异常
问题描述
我在 HDP 3.1.4 上有 Hive + LLAP
Hive 和 Tez 配置是:
yarn.nodemanager.resource.memory-mb = 40960
yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 40960
hive.tez.container.size = 4096
num_llap_nodes=4
hive.llap.daemon.num.executors=8
hive.llap.daemon.yarn.container.mb = 35840
llap_headroom_space=2048
llap_heap_size=32768
hive.llap.io.memory.size=1024
tez.am.resource.memory.mb=4096
hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps **-Xmx3276m**
tez.runtime.io.sort.mb= 1638
tez.runtime.unordered.output.buffer.size-mb=409
以下查询正常运行:
select count(*) from balance;
但是在以下查询中使用 group by 表达式时:
select count(*),jobdate from balance group by jobdate;
我已经尝试了很多配置,但是抛出了这个很长的异常:
ERROR: Error while processing statement: **FAILED: Execution Error,
return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask.**
Vertex failed, vertexName=Map 1,
vertexId=vertex_1617520101397_0014_1_00, diagnostics=[Task
failed, taskId=task_1617520101397_0014_1_00_000013,
diagnostics=[TaskAttempt 0 failed, **info=[Error: Error while
running task ( failure ) : java.lang.NoClassDefFoundError: Could
not initialize class
org.apache.tez.runtime.library.api.TezRuntimeConfiguration** at
**BLABLA**
at java.lang.Thread.run(Thread.java:748) ]], Task failed,
taskId=task_1617520101397_0014_1_00_000006,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while
running task ( failure ) : java.lang.NoClassDefFoundError: Could
not initialize class
org.apache.tez.runtime.library.api.TezRuntimeConfiguration at
at java.lang.Thread.run(Thread.java:748) ]], Task failed,
taskId=task_1617520101397_0014_1_00_000005,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while
running task ( failure ) : java.lang.NoClassDefFoundError: Could
not initialize class
org.apache.tez.runtime.library.api.TezRuntimeConfiguration at
org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:111)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) ]], **Vertex did not
succeed due to OWN_TASK_FAILURE, failedTasks:9 killedTasks:31761,
Vertex vertex_1617520101397_0014_1_00 [Map 1] killed/failed due
to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2,
vertexId=vertex_1617520101397_0014_1_01, diagnostics=[Vertex
received Kill while in RUNNING state., Vertex did not succeed due
to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:18, Vertex
vertex_1617520101397_0014_1_01 [Reducer 2] killed/failed due
to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
VERTEX_FAILURE. failedVertices:1 killedVertices:1 Error Code: 2**
解决方案
在 Ambari Hive Config 页面中设置hive.tez.container.size有两个部分。其中一个出现在SETTINGS选项卡中,另一个与 LLAP 相关的位于 ADVANCED 选项卡中的Advanced hive - interactive-site下。我正在尝试使用hive.tez.container.size值设置选项卡而不是高级 hive-interactive-site部分。最后,我设置了以下配置并解决了错误:
set hive.tez.container.size=10240;
set hive.tez.java.opts=-Xmx9216m;
set tez.runtime.io.sort.mb=3072;
set tez.runtime.unordered.output.buffer.size-mb=1024;
推荐阅读
- javascript - GET 请求不在 json 内容类型烧瓶中
- javascript - 为什么 __proto__ 引用名称对于继承的对象看起来是错误的?
- google-apps-script - 更新 Google 域共享联系人照片
- mysql - mysql中出现频率最高的值
- python - 比 Pandas 的 isin 函数更快地提取条件行的方法
- python - python:我可以重新导入外部配置文件并让程序读取它而不重新启动程序本身吗?
- android - 将视图折叠到选项卡布局
- youtube-api - 如何列出 YouTube 频道的所有视频,包括每个视频的观看次数
- r - 当某些日期为 NA 时,按日期合并 R 中的两个数据框
- r - 在 R 中创建平均值列