apache-spark - 使用 sparkmeasure 时 Java 对象不可调用

Question

我正在尝试使用 Sparkmeausre 检查我的 Pyspark 代码的性能。我在 Windows 10 上使用 Pycharm 社区版，并正确配置了 Pyspark。我做了“ pip install sparkmeasure ”并且成功安装了sparkmeasure。现在，当我尝试运行这段代码时。

from pyspark import  SparkConf , SparkContext
from pyspark.context import  SparkContext
from pyspark.sql.session import  SparkSession
from sparkmeasure import StageMetrics


sc = SparkContext(master = "local" , appName = "sparkdemo")
spark = SparkSession(sc)
sm = StageMetrics(spark)

我收到错误消息。

File "C:/Users/nj123/PycharmProjects/pythonProject/sparkdemo.py", line 9, in <module>
sm = StageMetrics(spark)
File "C:\Users\nj123\PycharmProjects\pythonProject\venv\lib\site- 
packages\sparkmeasure\stagemetrics.py", line 15, in __init__
self.stagemetrics = self.sc._jvm.ch.cern.sparkmeasure.StageMetrics(self.sparksession._jsparkSession)
TypeError: 'JavaPackage' object is not callable

如何解决此错误并将 sparkmeasure 配置为 Pycharm？

标签： apache-sparkpysparkpycharm

Answer 1

感谢@user238607。以下是我为解决此问题而执行的步骤。

1. First download Sparkmeasure jar file from Maven Central.

2. Then move this jar file to the spark jar folder. Mine location was, C:\Spark\spark-3.0.1-bin-hadoop2.7\jars

3. Now, Go to pycharm again, and rerun the same code.

下载jar文件的链接。