时间戳

首页 > 解决方案 > mongoimport 大型 csv 文件运行时：5131534336 字节的 VirtualAlloc 失败，errno=1455

mongodb - mongoimport 大型 csv 文件运行时：5131534336 字节的 VirtualAlloc 失败，errno=1455

问题描述

我正在尝试使用 mongoimport 在 mongo 中导入一个大型 csv 文件，如下所示：

mongoimport -d dbname -c collectionName --type csv --file filename -f "a,b" --numInsertionWorkers 9

但是一旦达到 100%，我就会收到以下错误。

runtime: VirtualAlloc of 5131534336 bytes failed with errno=1455
fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw(0x85cfd4, 0x30)
        C:/go/src/runtime/panic.go:566 +0x9c
runtime.sysMap(0xc452210000, 0x131dd0000, 0xc0420e6301, 0xa52d18)
        C:/go/src/runtime/mem_windows.go:116 +0x12c
runtime.(*mheap).sysAlloc(0xa39620, 0x131dd0000, 0x0)
        C:/go/src/runtime/malloc.go:407 +0x381
runtime.(*mheap).grow(0xa39620, 0x98ee4, 0x0)
        C:/go/src/runtime/mheap.go:726 +0x69
runtime.(*mheap).allocSpanLocked(0xa39620, 0x98ee4, 0xc04215d000)
        C:/go/src/runtime/mheap.go:630 +0x4f9
runtime.(*mheap).alloc_m(0xa39620, 0x98ee4, 0x100000000, 0x572fd88)
        C:/go/src/runtime/mheap.go:515 +0xee
runtime.(*mheap).alloc.func1()
        C:/go/src/runtime/mheap.go:579 +0x52
runtime.systemstack(0x572fd88)
        C:/go/src/runtime/asm_amd64.s:314 +0xb5
runtime.(*mheap).alloc(0xa39620, 0x98ee4, 0x100000000, 0xc0421083b0)
        C:/go/src/runtime/mheap.go:580 +0x7a
runtime.largeAlloc(0x131dc7294, 0x572fe00, 0x4522a1)
        C:/go/src/runtime/malloc.go:774 +0x9a
runtime.mallocgc.func1()
        C:/go/src/runtime/malloc.go:669 +0x45
runtime.systemstack(0xc042016a00)
        C:/go/src/runtime/asm_amd64.s:298 +0x7e
runtime.mstart()
        C:/go/src/runtime/proc.go:1079

goroutine 35 [running]:
runtime.systemstack_switch()
        C:/go/src/runtime/asm_amd64.s:252 fp=0xc04211bc20 sp=0xc04211bc18
runtime.mallocgc(0x131dc7294, 0x0, 0x1000, 0x0)
        C:/go/src/runtime/malloc.go:670 +0x942 fp=0xc04211bcc0 sp=0xc04211bc20
runtime.rawstring(0x131dc7294, 0x0, 0x0, 0x0, 0x0, 0x0)
        C:/go/src/runtime/string.go:289 +0x8c fp=0xc04211bcf8 sp=0xc04211bcc0
runtime.rawstringtmp(0x0, 0x131dc7294, 0xc04211bd80, 0x5c92c4, 0xc0421023c0, 0xc042042040, 0xa0da20)
        C:/go/src/runtime/string.go:111 +0xa8 fp=0xc04211bd38 sp=0xc04211bcf8
runtime.slicebytetostring(0x0, 0xc24a210000, 0x131dc7294, 0x207ffffff, 0x0, 0x0)
        C:/go/src/runtime/string.go:93 +0x45 fp=0xc04211bd90 sp=0xc04211bd38
github.com/mongodb/mongo-tools/mongoimport/csv.(*Reader).parseRecord(0xc04211e280, 0x0, 0x0, 0x0, 0x0, 0x0)

我不确定文件中的记录数，但文件大小约为 4.5GB。我在这里做错了什么吗？请帮忙！我已按照以下教程从 https://www.khalidalnajjar.com/insert-200-million-rows-into-mongodb-in-minutes开始

标签： mongodbcsvbigdatamongoimport

解决方案

推荐阅读