azure - Azure 数据工厂管道复制数据错误。将 CSV 导入 SQL 表。已添加具有相同密钥的项目
问题描述
我整天都在做这件事,我可以让它从 CSV 导入单个列,但是一旦有多个列,我每次都会收到这个错误。我不知道它是否重要,但分隔符是管道 (|)。总共有 260 列和 880 万行我正在尝试导入所有字段都是 nvarcher(255)。
{
"errorCode": "2200",
"message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'An item with the same key has already been added.'.,Source=,''Type=System.ArgumentException,Message=An item with the same key has already been added.,Source=mscorlib,'",
"failureType": "UserError",
"target": "Copy data1",
"details": []
}
非常感谢任何可以阐明这个问题的人。
在输出下也发现了这个:
{
"dataRead": 4194304,
"dataWritten": 0,
"filesRead": 1,
"sourcePeakConnections": 10,
"sinkPeakConnections": 2,
"rowsRead": 0,
"rowsCopied": 0,
"copyDuration": 4,
"throughput": 1024,
"errors": [
{
"Code": 11402,
"Message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property '' is invalid: 'An item with the same key has already been added.'.,Source=,''Type=System.ArgumentException,Message=An item with the same key has already been added.,Source=mscorlib,'",
"EventType": 0,
"Category": 5,
"Data": {
"PropertyName": "UnknownParameterName",
"InvalidParameter": "An item with the same key has already been added."
},
"MsgId": null,
"ExceptionType": null,
"Source": null,
"StackTrace": null,
"InnerEventInfos": []
}
],
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West Europe)",
"usedDataIntegrationUnits": 4,
"billingReference": {
"activityType": "DataMovement",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.06666666666666667,
"unit": "DIUHours"
}
]
},
"usedParallelCopies": 1,
"executionDetails": [
{
"source": {
"type": "AzureBlobStorage",
"region": "West Europe"
},
"sink": {
"type": "AzureSqlDatabase",
"region": "West Europe"
},
"status": "Failed",
"start": "2020-10-08T22:23:53.9269314Z",
"duration": 4,
"usedDataIntegrationUnits": 4,
"usedParallelCopies": 1,
"profile": {
"queue": {
"status": "Completed",
"duration": 1
},
"transfer": {
"status": "Completed",
"duration": 2,
"details": {
"listingSource": {
"type": "AzureBlobStorage",
"workingDuration": 0
},
"readingFromSource": {
"type": "AzureBlobStorage",
"workingDuration": 0
},
"writingToSink": {
"type": "AzureSqlDatabase",
"workingDuration": 0
}
}
}
},
"detailedDurations": {
"queuingDuration": 1,
"transferDuration": 2
}
}
],
"dataConsistencyVerification": {
"VerificationResult": "NotVerified"
},
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}
解决方案
所以问题是元数据。即使复制数据在 CSV 本身上为标题提供了唯一名称,它们也不是唯一的,因为这就是我们接收数据的方式。只是让它跳过一行并修复它。感谢任何正在寻找和解决这个问题的人。
推荐阅读
- excel - 需要帮助根据 VBS 中的范围将复制的范围从 excel 文件正确保存到单独的文件中
- json - 迭代 json 时 FlatList 不起作用
- amazon-web-services - AWS Cognito 用户活动
- ruby-on-rails - bundle 无法加载此类文件 bundler-1.17.1/exe/bundle
- solr - SOLR DataImportHanlder (DIH) 完整索引 - 有时我们的索引在导入期间显示接近零的文档
- sql - 使用 cte 时子查询返回超过 1 个值错误
- sql-server - 更改作为合并结果的视图列的精度
- php - 我在 hMailServer 和 XAMPP 的 PHPWebAdmin 页面中不断获得“致命错误:找不到类 COM”
- microsoft-translator - 用户如何使用 Word Online Preview 翻译文档?
- android - 膨胀类 com.google.android.material.bottomnavigation.BottomNavigationView 时出错