时间戳

首页 > 解决方案 > 如何使 Spark DataFrameReader jdbc 接受自定义类型的 Postgres 数组

postgresql - 如何使 Spark DataFrameReader jdbc 接受自定义类型的 Postgres 数组

问题描述

使用DataFrameReader函数Unsupported type ARRAY读取表时得到：jdbc()

java.sql.SQLException: Unsupported type ARRAY
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$getCatalystType(JdbcUtils.scala:251)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$8.apply(JdbcUtils.scala:316)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$8.apply(JdbcUtils.scala:316)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:315)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:63)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:210)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.apply(JDBCRelation.scala:225)
    at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:313)

表是这样的：

mydb=> \d check_type;
              Table "public.check_type"
 Column |   Type    | Collation | Nullable | Default
--------+-----------+-----------+----------+---------
 id     | integer   |           |          |
 types  | my_type[] |           |          |

mydb=> \d my_type;
          Composite type "public.my_type"
 Column |  Type   | Collation | Nullable | Default
--------+---------+-----------+----------+---------
 id     | integer |           |          |

有解决方法吗？或者这是一个预期的行为？

火花版本：

Spark-core = 2.4.0;
Spark-sql = 2.4.0;

标签： postgresqlapache-sparkjdbcsqlexceptioncustom-type

解决方案

推荐阅读