时间戳

首页 > 解决方案 > 无法处理 Py4JJavaError

exception - 无法处理 Py4JJavaError

问题描述

尝试处理向 spark.sql() 输入无效 sql 代码时引发的 Py4JJavaError。我的功能如下：

import py4j

def sql_to_df_and_create_view(sql_string: str, view_name: str):
    ''' Takes in a SQL command in string format and creates a spark df from the command. 
Secondly, creates a temporary view with name specified under view_name parameter. Ensures that 
the SQL code is valid and that the view is created. '''
    try:
        df = spark.sql(sql_string)
    except py4j.protocol.Py4JJavaError:
        raise Exception(f'Invalid SQL code passed in by {sql_string}.')
        return
         
    df.createOrReplaceTempView(view_name)
    return df

错误代码为：

org.apache.spark.sql.catalyst.parser.ParseException: 
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, 
gateway_client, target_id, name)
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:

Py4JJavaError: An error occurred while calling o213.sql.
: org.apache.spark.sql.catalyst.parser.ParseException: 
mismatched input 'sd' expecting {'(', 'SELECT', 'FROM', 'ADD', 'DESC', 'WITH', 'VALUES', 
'CREATE', 'TABLE', 'INSERT', 'DELETE', 'DESCRIBE', 'EXPLAIN', 'SHOW', 'USE', 'DROP', 'ALTER', 
'MAP', 'SET', 'RESET', 'START', 'COMMIT', 'ROLLBACK', 'MERGE', 'UPDATE', 'CONVERT', 'REDUCE', 
'REFRESH', 'CLEAR', 'CACHE', 'UNCACHE', 'DFS', 'TRUNCATE', 'ANALYZE', 'LIST', 'REVOKE', 
'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT', 'LOAD', 'OPTIMIZE', 'COPY'}(line 1, pos 
0)

== SQL ==
sd
^^^

当我在 sql_string 参数中使用无效的 sql 运行函数时，错误不会被处理，它仍然会引发相同的错误代码，而不是 'Exception(f'Invalid SQL code pass in by {sql_string}.')'。如果有人能弄清楚为什么没有正确处理，我将不胜感激:)

标签： exceptionpysparkpy4j

解决方案

推荐阅读