c# - NPGSQL 开始文本导入在较大的 CSV 文件上引发异常

问题描述

编辑：经过更多测试后，此错误与 HAPROXY 直接相关。当我把它从堆栈中取出时，它按预期工作。

我已经为这个问题苦苦挣扎了好几天。此代码适用于较小的 CSV 文件 (25 KB)，但一旦您尝试处理较大的文件 (5000 KB)，它就会引发异常。

我的堆栈是从 asp.net core 3.0、NPGSQL 4.1.1 到 HAPROXY 到 PostgreSQL 12 实例的以下代码。

这是代码：

      string query = "COPY " + seedingDataApi.tableName + " FROM STDIN CSV ENCODING 'UTF8' QUOTE '\"'; ";
                using (var conn = postgresContext.GetDBConnectionPG())
                {

                    conn.Open();
                    using (var writer = conn.BeginTextImport(query))
                    {
                        _logger.LogInformation("Text import started");
                        using (StreamReader sr = new StreamReader(seedingDataApi.tableName + ".csv"))
                        {

                            _logger.LogInformation("Read stream started.");
                            while (!sr.EndOfStream)
                            {
                                string testString = "";

                                testString = sr.ReadLine();
                                while ("<EOL>" != testString.Substring(Math.Max(0, testString.Length - 5)))
                                {
                                    testString += sr.ReadLine();
                                }
                                testString = testString.Remove(testString.Length - 5);
                                writer.WriteLine(testString);
                            }

                        }
                    }
                }

应用程序抛出的异常是：

 System.Exception: Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
 ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)
   at Npgsql.NpgsqlRawCopyStream..ctor(NpgsqlConnector connector, String copyCommand)
   at Npgsql.NpgsqlConnection.BeginTextImport(String copyFromCommand)
   at MiniAppApiServer.Repositories.TableSeedRepo.TableSeedRepository.SeedTableDataAsync(TableSeedDataParams seedingDataApi, CancellationToken ct) in /source/MiniAppApiServer/Repositories/TableSeedRepo/TableSeedRepository.cs:line 306System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   at MiniAppApiServer.Repositories.TableSeedRepo.TableSeedRepository.SeedTableDataAsync(TableSeedDataParams seedingDataApi, CancellationToken ct) in /source/MiniAppApiServer/Repositories/TableSeedRepo/TableSeedRepository.cs:line 339
   at MiniAppApiServer.Supervisors.TableSeedSupervisor.TableSeedSupervisor.SeedTableDataAsync(Stream data, String tablename, CancellationToken ct) in /source/MiniAppApiServer/Supervisors/TableSeedSupervisor/TableSeedSupervisor.cs:line 86
   at MiniAppApiServer.Controllers.TableSeedingController.PostTableDataSeed(IFormFile file, CancellationToken ct) in /source/MiniAppApiServer/Controllers/TableSeedingController.cs:line 114

非常感谢！

标签： c#postgresqlasp.net-corehaproxynpgsql

c# - NPGSQL 开始文本导入在较大的 CSV 文件上引发异常

问题描述

解决方案

推荐阅读