python - z:net.snowflake.spark.snowflake.Utils.runQuery。: java.lang.NullPointerException
问题描述
根据我的用例,我正在尝试将数据从 s3 写入雪花。我的应用程序是用 Python/Pyspark 编写的,并且在 AWS EKS 集群上运行。
代码是:
def write_from_audience_parquet_to_snowflake(self):
self.logger.info('write_from_audience_parquet_to_snowflake started')
path = os.path.dirname(os.path.abspath(__file__))
parquet_stage = read_template(path + '/sql/staging_audience_attribute.sql')
input_base_s3 = self.base_input_s3_path.replace("s3a://", "s3://")
self.logger.info("input_base_s3 = {}".format(input_base_s3))
parquet_stage = parquet_stage.render(input_base_s3=input_base_s3, mab_id=self.mab_id,
mab_send_id=self.mab_send_id, access_key=self.aws_access_key,
secret_key=self.aws_secret_key, session_token=self.aws_session_key)
create_table_template = read_template(path + '/sql/create_audience_table.sql')
create_table_sql = create_table_template.render(audience_table=self.audience_table)
upload_stage = read_template(path + '/sql/upload_audience_stage.sql')
upload_stage = upload_stage.render(audience_table=self.audience_table)
try:
self.utilSf.runQuery(self.sfoptions, parquet_stage)
self.utilSf.runQuery(self.sfoptions, create_table_sql)
self.utilSf.runQuery(self.sfoptions, upload_stage)
except Exception as e:
self.logger.exception(e)
raise e
self.logger.info('write_from_audience_parquet_to_snowflake completed successfully')
错误是:
An error occurred while calling z:net.snowflake.spark.snowflake.Utils.runQuery.
: java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:460)
at net.snowflake.spark.snowflake.JDBCWrapper.getConnector(SnowflakeJDBCWrapper.scala:141)
at net.snowflake.spark.snowflake.Utils$.getJDBCConnection(Utils.scala:244)
at net.snowflake.spark.snowflake.Utils$.getJDBCConnection(Utils.scala:249)
at net.snowflake.spark.snowflake.Utils$.runQuery(Utils.scala:391)
at net.snowflake.spark.snowflake.Utils.runQuery(Utils.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
请在此处找到完整的错误日志
我已经读到这是由于 spark 提供的 scala 版本控制问题而发生的。我正在使用带有外部添加的 hadoop 3.1.2 的 spark 2.4.5。另外,我正在使用这些雪花罐:
spark-snowflake_2.11-2.4.10-spark_2.4.jar
雪花摄取 SDK-0.9.5.jar
雪花-jdbc-3.6.15.jar
如果需要更多详细信息,请通过评论告诉我。
解决方案
当它尝试连接到 Snowflake 时会发生错误。我建议检查“self.sfoptions”中的值,以确保正确指定所有必需的选项。下面是一个例子:
sfOptions = {
"sfURL" : "<account_name>.<cloud_domain>",
"sfAccount" : "<account_name>",
"sfUser" : "<user_name>",
"sfPassword" : "<password>",
"sfDatabase" : "<database>",
"sfSchema" : "<schema>",
"sfWarehouse" : "<warehouse>",
}