时间戳

首页 > 解决方案 > 我不断收到错误：值 toDF 不是 org.apache.spark.rdd.RDD 的成员

scala - 我不断收到错误：值 toDF 不是 org.apache.spark.rdd.RDD 的成员

问题描述

我已经写了“import sqlContext.implicits._”；但是它仍然不起作用。它就在 spark-shell 中。为什么在这种情况下不正确？我见过许多其他将 rdd 转换为数据框的方法，但我的大部分代码都写成 toDF()。如何使 toDF 工作？错误：

import org.apache.spark.ml.evaluation.RegressionEvaluator
import org.apache.spark.ml.recommendation.ALS
import org.apache.spark.ml.tuning.{ParamGridBuilder, CrossValidator}
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.DoubleType
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
import sys.process._

"rm -f ./ml-1m.zip".!
"wget http://files.grouplens.org/datasets/movielens/ml-1m.zip".!

"ls ./ml-1m.zip".!

"rm -r ./ml-1m".!
"unzip ml-1m.zip".!

"ls ./ml-1m".!

val ratings_raw = sc.textFile("./ml-1m/ratings.dat")
ratings_raw.takeSample(false,10, seed=0).foreach(println)

case class Rating(userId: Int, movieId: Int, rating: Float)
val ratings = ratings_raw.map(x => x.split("::")).map(r => Rating(r(0).toInt, r(1).toInt, r(2).toFloat)).toDF().na.drop()

标签： scalaapache-spark

解决方案

如果你在 spark-shell 中，你不需要创建新的 SQLContext

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

你可以直接使用火花


scala> import spark.implicits._

scala> val ratings_raw = sc.textFile("./ml-1m/ratings.dat")
ratings_raw: org.apache.spark.rdd.RDD[String] = ./ml-1m/ratings.dat MapPartitionsRDD[1] at textFile at <console>:38

scala> case class Rating(userId: Int, movieId: Int, rating: Float)
defined class Rating

scala> val ratings = ratings_raw.map(x => x.split("::")).map(r => Rating(r(0).toInt, r(1).toInt, r(2).toFloat)).toDF().na.drop()
ratings: org.apache.spark.sql.DataFrame = [userId: int, movieId: int ... 1 more field]

scala> ratings
res3: org.apache.spark.sql.DataFrame = [userId: int, movieId: int ... 1 more field]

scala> ratings.printSchema
root
 |-- userId: integer (nullable = false)
 |-- movieId: integer (nullable = false)
 |-- rating: float (nullable = false)

推荐阅读