时间戳

首页 > 解决方案 > Py Spark No 这样的文件或目录问题

python - Py Spark No 这样的文件或目录问题

问题描述

我想使用带有 pyspark 的 SQL 表。我可以看到这样的 sparkDF；

sparkDF
DataFrame[SPRAS: string, PRCTR: string, DATBI: string, KOKRS: string, KTEXT: string, LTEXT: string, MCTXT: string]

但是当我 sparkDF.show() 出现错误时。这是我的代码；

import pyodbc
import pandas
import os

import findspark as fs
fs.init()
os.environ["JAVA_HOME"] = "C:/Program Files/Java/jdk1.8.0_181"
os.environ["PYTHONPATH"]=os.environ["PYTHON"]
from pyspark import SparkContext, SparkConf

conf = SparkConf().set("spark.cores.max", 3).set("spark.executor.memory", "3G").setAppName('hello').setMaster(
    'spark://xx.xx.xx.xx:7077')
sc = SparkContext(conf=conf)

sqlContext = SQLContext(sc)
spark = sqlContext.sparkSession

conn = pyodbc.connect('Driver={SQL Server};'
                      'Server=xx.xx.xx.xx;'
                      'Database=DEP;'
                      'Trusted_Connection=yes;')
sql = 'SELECT * FROM dep.xxxx'
pdf  = pandas.read_sql(sql, conn)
rdd=sc.parallelize(pdf)
sparkDF =  spark.createDataFrame(rdd)
sparkDF.show()
sparkDF.createOrReplaceTempView("sparkDF")
sqlDF = spark.sql("SELECT * FROM sparkDF")
sqlDF.show()

结果; 

9/11/21 09:42:45 WARN TaskSetManager: Lost task 0.0 in stage 12.0 (TID 63, xx.xx.xx.xx, executor 2): java.io.IOException: Cannot run program "C:\ProgramData\Anaconda3\envs\deneme\python.exe": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:163)
        at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
        at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:65)
        at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
        at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128)
        at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)

        ......

你对此有什么想法吗？（我是 Python 世界的新手）

标签： pythonapache-sparkpyspark

解决方案

推荐阅读