首页 > 解决方案 > Visual Studio 数据从平面文件加载到 Postgres 数据库

问题描述

我正在使用 Visual Studio 将数据从 CSV 文件传输到 Postgres 数据库。我的数据库安装在 windows server 2012 上,我正在使用本地计算机传输数据。我的进程成功运行而没有抛出任何错误,但不知何故它没有加载我的 CSV 文件的所有行。该文件包含 382,​​363 行,但在我检查我的数据库后,仅加载了 26000 行。

我直接从 Postgres 的导入向导加载了 CSV,它成功加载了所有 382,​​363 行,但是当我通过 Visual Studio 加载数据时,它只加载了 26000 行而不会引发任何错误。我只收到两条警告信息。有没有人遇到过任何问题,如果是这样,我该如何解决?

粘贴在我的流程的整个输出下方

SSIS package "C:\Users\Shivam SARIN\source\repos\Integration Services Project3\Integration Services Project3\Package.dtsx" starting.

Information: 0x4004300A at Data Flow Task, SSIS.Pipeline: Validation phase is beginning.

Information: 0x4004300A at Data Flow Task, SSIS.Pipeline: Validation phase is beginning.

Warning: 0x80049304 at Data Flow Task, SSIS.Pipeline: Warning: Could not open global shared memory to communicate with performance DLL; data flow performance counters are not available.  To resolve, run this package as an administrator, or on the system's console.

Warning: 0x80047076 at Data Flow Task, SSIS.Pipeline: The output column "T_CTRY_DESTINATION" (115) on output "Flat File Source Output" (6) and component "Flat File Source" (2) is not subsequently used in the Data Flow task. Removing this unused output column can increase Data Flow task performance.

Information: 0x40043006 at Data Flow Task, SSIS.Pipeline: Prepare for Execute phase is beginning.

Information: 0x40043007 at Data Flow Task, SSIS.Pipeline: Pre-Execute phase is beginning.

Information: 0x402090DC at Data Flow Task, Flat File Source [2]: The processing of file "C:\Users\Shivam SARIN\Documents\Excel-csv\MS 2018 Q3.csv" has started.

Information: 0x4004300C at Data Flow Task, SSIS.Pipeline: Execute phase is beginning.

Information: 0x402090DE at Data Flow Task, Flat File Source [2]: The total number of data rows processed for file "C:\Users\Shivam SARIN\Documents\Excel-csv\MS 2018 Q3.csv" is 382364.

Information: 0x402090DF at Data Flow Task, OLE DB Destination [275]: The final commit for the data insertion in "OLE DB Destination" has started.

Information: 0x402090E0 at Data Flow Task, OLE DB Destination [275]: The final commit for the data insertion in "OLE DB Destination" has ended.

Information: 0x40043008 at Data Flow Task, SSIS.Pipeline: Post Execute phase is beginning.

Information: 0x402090DD at Data Flow Task, Flat File Source [2]: The processing of file "C:\Users\Shivam SARIN\Documents\Excel-csv\MS 2018 Q3.csv" has ended.

Information: 0x4004300B at Data Flow Task, SSIS.Pipeline: "OLE DB Destination" wrote 382363 rows.

Information: 0x40043009 at Data Flow Task, SSIS.Pipeline: Cleanup phase is beginning.

SSIS package "C:\Users\Shivam SARIN\source\repos\Integration Services Project3\Integration Services Project3\Package.dtsx" finished: Success.

The program '[21956] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).

标签: postgresqlvisual-studiossis

解决方案


根据我使用 postgres 的经验,我发现使用 oledb 目标插入行非常慢,而且正如您在上面提到的那样,似乎发生了奇怪的事情。它很慢,因为提供程序不支持批量插入操作。

我的建议是用Execute Process Task. 这会更快,并且您将使用 postgres 原生的工具。

更多信息在这里:https ://www.postgresql.org/docs/9.6/app-psql.html


推荐阅读