apache-spark - How to get Great_Expectations to work with Spark Dataframes in Apache Spark ValueError: Unrecognized spark type: string

Question

I have a Apache Spark dataframe which as a 'string' type field. However, Great_Expectations doesn't recognize the field type. I have imported the modules that I think are necessary, but not sure why Great_Expectations doesn't recognize the field

import great_expectations as ge
import great_expectations.dataset.sparkdf_dataset
from great_expectations.dataset.sparkdf_dataset import SparkDFDataset
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, BooleanType

The following code reads in the csv as a dataframe

test = spark.read.csv('abfss://[e-mail]/RAW/LANDING/customers.csv', inferSchema=True, header=True)

The following shows the schema:

test.printSchema()
Command executed in 2 sec 64 ms by carlton on 1:53:28 PM, 6/17/21
root
 |-- first_name: string (nullable = true)

I think the following line of code creates Great_Expectation dataframe from the above Spark Dataframe

test2 = ge.dataset.SparkDFDataset(test)

I then code in the following expectation:

test2.expect_column_values_to_be_of_type(column='first_name', type_='string')

However, I get the following error:

ValueError: Unrecognized spark type: string
Traceback (most recent call last):

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/great_expectations/data_asset/util.py", line 80, in f
    return self.mthd(obj, *args, **kwargs)

Not sure why Great_Expectations cannot recognize the Spark Type?

标签： apache-sparkpysparkgreat-expectations

Answer 1

You need to use names like, StringType, LongType, etc. - the same names as specified in the documentation. It should be like this:

test2.expect_column_values_to_be_of_type("first_name", "StringType")

see screenshot: