pyspark - 如何在 PySpark 中保存列计算语句？

Question

我有一段 pyspark 代码，它是以编程方式生成的，而且生成起来很昂贵。我想生成状态并保存它。但是pickle不能用来保存语句。有没有办法做到这一点？

from pyspark.sql import functions as F

# the real statement is more complex and expensive to generate
statement = F.when(F.col("col") < 0.5).when(F.col("col2")<0.6)

import pickle
pickle.dump(statement, "statement.pickle")

#TypeError: file must have a 'write' attribute

标签： pyspark