pyspark - DELTA TABLES - 指定的属性与现有属性不匹配

Question

我在 DataBricks Delta Tables 上使用一组用户属性进行元数据管理。问题是当我需要更改其中一个属性时，我收到“失败错误：指定的属性与 /mnt/silver/... 中的现有属性不匹配”错误消息。

Databricks 文档仅声明将引发异常，并且我没有找到任何参数来强制它接受新值。

是否可以只更新表属性？

有什么建议么？

示例代码：

query = f'''
CREATE  TABLE if not exists  {tableMetadataDBName}.{tableMetadataTableName}
(
... my columns ...
-- COMMON COLUMNS
,Hash  string
,sourceFilename STRING
,HashWithFileDate  string
,Surrogate_Key  STRING
,SessionId  STRING
,SessionRunDate TIMESTAMP
,Year INT GENERATED ALWAYS AS ( YEAR(fileDate))
,Month INT  GENERATED ALWAYS AS ( MONTH(fileDate))
,fileDate   DATE
)
USING DELTA 
COMMENT '{tableDocumentationURL}'
LOCATION "{savePath}/parquet"
OPTIONS(  "compression"="snappy")
PARTITIONED BY (Year, Month, fileDate )
 TBLPROPERTIES ("DataStage"="{txtDataStage.upper()}"
,"Environment"="{txtEnvironment}"
,"source"="{tableMetadataSource}"
,"CreationDate"             = "{tableMetadataCreationDate}"
,"CreatedBy"                = "{tableMetadataCreatedBy}"
,"Project"                  = "{tableMetadataProject}"
,"AssociatedReports"        = "{tableMetadataAssociatedReports}"
,"UpstreamDependencies"     = "{tableMetadataUpstreamDependencies}"
,"DownstreamDependencies"   = "{tableMetadataDownstreamDependencies}"
,"Source"                   = "{tableMetadataSource}"
,"PopulationFrequency"      = "{tableMetadataPopulationFrequency}"
,"BusinessSubject"          = "{tableMetadataBusinessSubject}"
,"JiraProject"              = "{tableMetadataJiraProject}"
,"DevOpsProject"            = "{tableMetadataDevOpsProject}"
,"DevOpsRepository"         = "{tableMetadataDevOpsRepository}"
,"URL"                      = "{tableMetadataURL}") '''

spark.sql(query)

标签： pysparkdatabricksdelta-lake

Answer 1

是的，可以只更改属性 - 您需要为此使用“ ALTER TABLE [table_name] SET TBLPROPERTIES ... ”：

query = f"""ALTER TABLE {table_name} SET TBLPROPERTIES ( 
'CreatedBy' = '{tableMetadataCreatedBy}
,'Project'  = '{tableMetadataProject}'
....
)"""
spark.sql(query)