时间戳

首页 > 解决方案 > PySpark 2.4.5 与 Python 3.8.3 不兼容，我该如何解决？

python - PySpark 2.4.5 与 Python 3.8.3 不兼容，我该如何解决？

问题描述

代码

from pyspark import SparkContext,SparkConf

conf=SparkConf().setMaster('local').setAppName('Test App')
sc=SparkContext(conf)

错误信息

    Traceback (most recent call last):
      File "C:\Users\Test\PycharmProjects\python-test\MainFile.py", line 5, in <module>
        from pyspark import SparkContext,SparkConf
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\__init__.py", line 51, in <module>
        from pyspark.context import SparkContext
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\context.py", line 31, in <module>
        from pyspark import accumulators
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\accumulators.py", line 97, in <module>
        from pyspark.serializers import read_int, PickleSerializer
      File "C:\Test\Python_3.8.3_Latest\lib\sit`enter code here`e-packages\pyspark\serializers.py", line 72, in <module>
        from pyspark import cloudpickle
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\cloudpickle.py", line 145, in <module>
        _cell_set_template_code = _make_cell_set_template_code()
      File "C:\Test\Python_3.8.3_Latest\lib\site-packages\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
        return types.CodeType(
    TypeError: an integer is required (got type bytes)

标签： pythonapache-sparkpyspark

解决方案

虽然最新的 Spark 文档说它支持Python 2.7+/3.4+，但实际上还不支持 Python 3.8。根据此PR，预计 Spark 3.0 将支持 Python 3.8。因此，您可以试用 Spark 3.0 预览版（假设您不打算进行生产部署）或“暂时”回退到用于 Spark 2.4.x 的 Python 3.6/3.7。

推荐阅读