时间戳

首页 > 解决方案 > val df_read_old = spark.read.format("csv").option("header", "true").option("delimiter","^").schema(old_schema).load("D:\\TESTSCALA\ \"+"${lstfile}")

scala - val df_read_old = spark.read.format("csv").option("header", "true").option("delimiter","^").schema(old_schema).load("D:\\TESTSCALA\ \"+"${lstfile}")

问题描述

val df_read_old = spark.read.format("csv").option("header", "true").option("delimiter","^").schema(old_schema).load("D:\\TESTSCALA\\"+"${lstfile}") // is not able to read file from variable.

任何其他方式来做到这一点，或者我错过了一些东西。

当文件在给定路径中可用时出现以下错误：

Exception in thread "main" org.apache.spark.sql.AnalysisException: Path does not exist: file:/D:/TESTSCALA/${lstfile};
    at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:621)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:350)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.immutable.List.flatMap(List.scala:355)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:349)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156)
    at com.tmobile.dev.compare.serializedmaterial11$.main(test.scala:31)
    at com.tmobile.dev.compare.serializedmaterial11.main(test.scala)

标签： scalaapache-spark

解决方案

    val lstfile = ??? // your file here
  val old_schema = "" // your schema here
  val df_read_old = spark.read.format("csv").option("header", "true").option("delimiter","^").schema(old_schema)
    .load(s"D:\TESTSCALA\${lstfile}")

您必须s像上面的示例一样正确使用 scala 字符串替换。

推荐阅读