pyspark - 错误：AttributeError：“DataFrame”对象没有属性“_jdf”

Question

我想使用 pyspark 执行 k 折交叉验证来微调参数，我正在使用 pyspark.ml。我收到属性错误。

AttributeError：“DataFrame”对象没有属性“_jdf”

我最初尝试使用 pyspark.mllib 但未能成功执行 k 折交叉验证

import pandas as pd
from pyspark import SparkConf, SparkContext
from pyspark.ml.classification import DecisionTreeClassifier

data=pd.read_csv("file:///SparkCourse/wdbc.csv", header=None)
type(data)
print(data)

conf = SparkConf().setMaster("local").setAppName("SparkDecisionTree")
sc = SparkContext(conf = conf)

# Create initial Decision Tree Model
dt = DecisionTreeClassifier(labelCol="label", featuresCol="features", 
maxDepth=3)

# Train model with Training Data
dtModel = dt.fit(data)

# I expect the model to be trained but I'm getting the following error 
AttributeError: 'DataFrame' object has no attribute '_jdf'

注意：我可以打印数据。错误在 dtModel

标签： pyspark

Answer 1

将 Panadas 转换为 Spark

from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)

spark_dff = sqlContext.createDataFrame(panada_df)