apache-spark - 如何在 pyspark 中读取 .hql 文件（运行 hive 查询）

Question

我有包含大量查询的 .hql 文件。它在蜂巢中运行缓慢。我想使用 pyspark/sparksql 读取和运行 .hql 文件。

我试过count = sqlContext.sql(open("file.hql").read()).count()但给出以下错误：

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
  File "/data/CDH-5.7.1-1.cdh5.7.1/lib/spark/python/pyspark/sql/context.py", line 580, in sql
    return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
  File "/data/CDH-5.7.1-1.cdh5.7.1/lib/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/data/CDH-5.7.1-1.cdh5.7.1/lib/spark/python/pyspark/sql/utils.py", line 51, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u"missing EOF at ';' near 'db'; line 1 pos 36"

标签： apache-sparkhivepyspark