时间戳

首页 > 解决方案 > 在 AWS EMR Notebook 上使用来自 sagemaker_pyspark 的 XGBoostSageMakerEstimator 的运行时错误

python - 在 AWS EMR Notebook 上使用来自 sagemaker_pyspark 的 XGBoostSageMakerEstimator 的运行时错误

问题描述

我正在尝试在 EMR (Jupyter) Notebook 上使用 SageMaker Python SDK 和 PySpark。当尝试使用如下所示的 XGBoostSageMakerEstimator 时，

from sagemaker_pyspark.algorithms import XGBoostSageMakerEstimator

xgboost_estimator = XGBoostSageMakerEstimator(
    sagemakerRole=IAMRole(someRoleArn),
    trainingInstanceType='ml.m4.xlarge',
    trainingInstanceCount=1,
    endpointInstanceType='ml.m4.xlarge',
    endpointInitialInstanceCount=1)

我收到以下错误，我无法找到解决方案。

Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.ScalaMap object at 0x7fd3d9e96240>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'ScalaMap' object has no attribute '_java_obj'
Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.ScalaMap object at 0x7fd3d9e96240>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'ScalaMap' object has no attribute '_java_obj'
Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.Option object at 0x7fd3d9e9d3c8>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'Option' object has no attribute '_java_obj'
Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.Option object at 0x7fd3d9e9d128>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'Option' object has no attribute '_java_obj'
Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.Option object at 0x7fd3d9e9d0f0>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'Option' object has no attribute '_java_obj'
Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.Option object at 0x7fd3d9e9d080>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'Option' object has no attribute '_java_obj'
Exception ignored in: <bound method JavaWrapper.__del__ of <sagemaker_pyspark.wrapper.Option object at 0x7fd3d9e96ef0>>
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 40, in __del__
AttributeError: 'Option' object has no attribute '_java_obj'

任何解决此问题的帮助将不胜感激。

使用：

标签： pythonapache-sparkpysparkamazon-emramazon-sagemaker

解决方案

我遇到了同样的错误。我相信 sagemaker_pyspark 与 Spark 版本 > 2.3.2 不兼容（来源：https ://github.com/aws/sagemaker-spark/commit/4055f1e05be7d5e764f2abc8b3d6fc2c252ae272 ）。我能够与为该项目做出贡献的人确认这一点。

我用 Spark 2.3.2 运行了我的代码，再也没有看到异常。

推荐阅读