apache-spark - SparkSQL：在 Transform Lambda 中使用 SQL

Question

我无法访问 Spark SQL 转换函数的内联 SQL 命令中的 lambda 变量。

我不想爆炸并执行连接。我希望执行内联连接以获取电子邮件的值。

错误。

AnalysisException: cannot resolve '`y.email`' given input columns: [lkp.id, lkp.display_name, lkp.email_address, lkp.uuid]

代码。

transform (
  x.license_admin,
  y -> struct (
    get_uuid(lower(trim(y.email))) as email,
    ( select coalesce(lkp.uuid, get_uuid(lower(trim(y.email))))
      from  lkp_uuid_email_map lkp
      where lkp.email_address = lower(trim(y.email))
    ) as email_02,
    generic_hash(y.first_name) as first_name
  )
) as license_admin,

列模式

|-- order_line_items: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- additional_notes: string (nullable = true)
 |    |    |-- amount: double (nullable = true)
 |    |    |-- discounts: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- discount_amount: double (nullable = true)
 |    |    |    |    |-- discount_percentage: string (nullable = true)
 |    |    |    |    |-- discount_reason: string (nullable = true)
 |    |    |-- id: string (nullable = true)
 |    |    |-- license_admin: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- email: string (nullable = true)
 |    |    |    |    |-- first_name: string (nullable = true)
 |    |    |    |    |-- last_name: string (nullable = true)
 |    |    |    |    |-- license_admin_id: string (nullable = true)

提前致谢。

标签： apache-sparklambdaapache-spark-sql