hadoop - Impyla 在查询 HiveServer2 时从 org.apache.hadoop.hive.ql.exec.mr.MapRedTask 返回代码 1
问题描述
我正在使用Impyla从 HIVE 查询一些结果,但是,我遇到了这个问题:
来自因皮拉:
impala.error.OperationalError: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
并来自 HiveServer2:
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = hduser_20180827031927_fdb148b0-725b-434c-a0f8-98b6843d4348
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
我的源代码是:
from impala.dbapi import connect
import sys
dbName = sys.argv[1:][0]
query = sys.argv[1:][1]
conn = connect(host='192.168.0.10', port=10000, database=dbName , auth_mechanism='NOSASL' , use_ssl=True)
cursor = conn.cursor()
cursor.execute(query, configuration={'hive.exec.reducers.bytes.per.reducer': '100000', 'hive.auto.convert.join.noconditionaltask':'false','mapreduce.job.reduces':'1','hive.auto.convert.join':'false'})
returnData = []
for row in cursor:
returnData.append(row[0])
pprint(returnData)
如您所见,我添加了许多配置,但它不起作用
解决方案
从您的错误中,无法知道发生了什么。
我不确定在 impyla 中启用调试日志记录,因此您需要转到 YARN UI 来查找查询。
如果 YARN 没有运行,我认为您会收到更具描述性的错误,例如“无法提交作业”,尽管该错误可能没有从 HiveServer 传播
推荐阅读
- dart - 如何创建带有圆角和渐变背景的 RaisedButton?
- python - Many2one Odoo 11 中的域
- c++ - 对 Eigen 对象的自动引用不是用 GCC 编译,而是用 MSVC 编译
- matlab - 按用户参数过滤合适的内容
- kotlin - kotlin 中的 RecyclerView onClick
- .net - 如何在 asp.net 核心中将托管模型更改为 InProcess?
- swift - 拓扑排序 Swift
- octobercms - 监控日期并向用户发送电子邮件通知
- php - 根据发货国家/地区将 Woocommerce 结帐电话字段设为可选
- php - OnChange AJAX 似乎对动态值很复杂