go - go sql driver + context cancelled 似乎“泄漏”了 goroutines?
问题描述
背景
当我在我正在构建的应用程序上运行负载测试时,一切都运行顺利,直到机器几乎耗尽内存并且应用程序遇到问题。突然间所有的上下文都被取消了(很可能是被 hystrix 库取消了),因为处理请求的时间太长了。到目前为止,没有什么真正令人惊讶的。
问题
我觉得奇怪的是,当上下文被取消时,它似乎开始泄漏 goroutines。
在健康的条件下,大约有 20 个 goroutine 一直在运行。但是在这种情况发生后,它会保留 20 多个 goroutine:准确地说是 98 个。这个数字不会随时间下降。我使用带有读/写超时的 http.Server、带有上下文的 go-sql-driver/mysql 和 hystrix。下面是运行时间超过所有超时时间的 goroutines 转储。几乎所有这些都来自 go-sql-driver 或被 go-sql-driver 调用。MySQL 进程列表中没有任何查询。好奇其他人是否经历过这种情况并知道如何解决它。
应用程序在这之后仍然可以顺利运行,并且可以以 100% 的成功率处理超过 2000r/s,但是在测试完成后,之前挂起的 goroutines 仍然存在。
这是 goroutines 的转储(不是全部,但更多的是相同的)。runtime_pollWait 来自 go-mysql-driver:
goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7fd0fbc007b8, 0x72, 0x0)
goroutine 18 [sleep, 43 minutes]:
runtime.goparkunlock(...)
goroutine 19 [select, 2 minutes]:
database/sql.(*DB).connectionOpener(0xc000188240, 0xb835a0, 0xc00017b2c0)
goroutine 20 [select]:
database/sql.(*DB).connectionResetter(0xc000188240, 0xb835a0, 0xc00017b2c0)
goroutine 34 [select, 2 minutes]:
database/sql.(*DB).connectionOpener(0xc0001883c0, 0xb835a0, 0xc00016a140)
goroutine 35 [select]:
database/sql.(*DB).connectionResetter(0xc0001883c0, 0xb835a0, 0xc00016a140)
goroutine 8 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc0000ba370, 0xdf8475800)
goroutine 11 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc0000ba420, 0xdf8475800)
goroutine 14 [chan receive]:
github.com/go-redis/redis/internal/pool.(*ConnPool).reaper(0xc0000ba4d0, 0xdf8475800)
goroutine 98 [sleep, 43 minutes]:
runtime.goparkunlock(...)
goroutine 87 [chan receive]:
github.com/afex/hystrix-go/hystrix.(*metricExchange).Monitor(0xc002a1b5c0)
goroutine 99 [syscall]:
syscall.Syscall6(0xe8, 0xd, 0xc0002b9a58, 0x64, 0x64, 0x0, 0x0, 0x42bfcf, 0x8, 0xacdd80)
goroutine 100 [sleep]:
runtime.goparkunlock(...)
goroutine 88 [chan receive]:
github.com/afex/hystrix-go/hystrix.(*poolMetrics).Monitor(0xc003ac08a0)
goroutine 91 [chan receive]:
github.com/afex/hystrix-go/hystrix.(*metricExchange).Monitor(0xc002a1b680)
goroutine 92 [chan receive]:
github.com/afex/hystrix-go/hystrix.(*poolMetrics).Monitor(0xc003ac0f00)
goroutine 25921733 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06f298, 0x72, 0xffffffffffffffff)
goroutine 25921439 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0f983fe18, 0x72, 0xffffffffffffffff)
goroutine 25927120 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc003aa7620, 0xc01c2ef980, 0xc01a841bc0)
goroutine 25928175 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc012abbf80, 0xc01c2ee600, 0xc01144f8c0)
goroutine 25927727 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc014ae4300, 0xc0166a3c80, 0xc0030e8a80)
goroutine 24567262 [select]:
database/sql.(*DB).connectionCleaner(0xc000188240, 0x2540be400)
goroutine 25927990 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc012ababa0, 0xc0166a3ec0, 0xc01144f380)
goroutine 25921926 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa0702d8, 0x72, 0xffffffffffffffff)
goroutine 25921958 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06eb48, 0x72, 0xffffffffffffffff)
goroutine 24567381 [select]:
database/sql.(*DB).connectionCleaner(0xc0001883c0, 0x2540be400)
goroutine 25921798 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0f9882038, 0x72, 0xffffffffffffffff)
goroutine 25921751 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa6b6820, 0x72, 0xffffffffffffffff)
goroutine 25921757 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa6b6410, 0x72, 0xffffffffffffffff)
goroutine 25927113 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc003aa7320, 0xc01c2efbc0, 0xc01a8419e0)
goroutine 25921966 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06e8d8, 0x72, 0xffffffffffffffff)
goroutine 25921964 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06e9a8, 0x72, 0xffffffffffffffff)
goroutine 25921323 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06f0f8, 0x72, 0xffffffffffffffff)
goroutine 25928466 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e552240, 0xc01c2ef740, 0xc01144f920)
goroutine 25928011 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc0156599e0, 0xc01c2ee3c0, 0xc01baa8fc0)
goroutine 25919812 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06ea78, 0x72, 0xffffffffffffffff)
goroutine 25928014 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc015659b00, 0xc01177e900, 0xc01baa90e0)
goroutine 25922001 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa070548, 0x72, 0xffffffffffffffff)
goroutine 25925675 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa6b8150, 0x72, 0xffffffffffffffff)
goroutine 25928529 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc003aa7680, 0xc01c2ef8c0, 0xc01a841c20)
goroutine 25928467 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e552360, 0xc01c2ef680, 0xc01144f980)
goroutine 25928468 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e552480, 0xc01c2ef5c0, 0xc01144f9e0)
goroutine 25919817 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06e808, 0x72, 0xffffffffffffffff)
goroutine 25925511 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0f983e4e8, 0x72, 0xffffffffffffffff)
goroutine 25920658 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa070208, 0x72, 0xffffffffffffffff)
goroutine 25920661 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa070138, 0x72, 0xffffffffffffffff)
goroutine 25920667 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06e738, 0x72, 0xffffffffffffffff)
goroutine 25920860 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06f368, 0x72, 0xffffffffffffffff)
goroutine 25920790 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa070478, 0x72, 0xffffffffffffffff)
goroutine 25920793 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa06fec8, 0x72, 0xffffffffffffffff)
goroutine 25928006 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc015659800, 0xc01c1603c0, 0xc01baa8de0)
goroutine 25921085 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa6b7fb0, 0x72, 0xffffffffffffffff)
.....
goroutine 25925779 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa6b7ba0, 0x72, 0xffffffffffffffff)
goroutine 25926131 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fd0fa6b69c0, 0x72, 0xffffffffffffffff)
goroutine 25927988 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc012abaae0, 0xc01c2ee000, 0xc01144f2c0)
goroutine 25928470 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e5526c0, 0xc01c2ef440, 0xc01144faa0)
goroutine 25928471 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e5527e0, 0xc01c2ef380, 0xc01144fb00)
goroutine 25928472 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e552900, 0xc01c2ef2c0, 0xc01144fb60)
goroutine 25928473 [select, 3 minutes]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc00e552a20, 0xc01c2ef200, 0xc01144fbc0)
goroutine 26252345 [IO wait]:
internal/poll.runtime_pollWait(0x7fd0e7054fe8, 0x72, 0xffffffffffffffff)
解决方案
回答我自己的问题;问题很可能是系统过载。我对并发连接数的限制和使用少得多的 MySQL 连接 25%。这似乎可以解决问题。我现在的吞吐量减少了,但系统更稳定了。
它仍然没有回答为什么即使系统再次平静下来,netpoller 仍会继续阻塞。他们只是永远不会恢复,我认为这不好。但可能有一个很好的理由。我在 Stackoverflow 上通过网络连接多次看到此症状,例如 http.Client。
推荐阅读
- kubernetes - 如果发布大型 json,用于自动审查应用程序(gitlab-managed-apps)的入口 NGINX 非常慢
- ios - SwiftUI“线程1:EXC_BAD_INSTRUCTION(代码=EXC_I386_INVOP,子代码=0x0)”弹出时
- python - 将新项目添加到 Windows 上下文菜单(右键菜单)
- angular - 添加路由器存储配置 RouterState.Full 会引发神秘的错误消息
- java - java.lang.AssertionError:状态 404
- c++ - 不要使用静态转换进行算术转换(cpp-core-guidelines)
- javascript - 找不到变量 atob React Native+Firebase 错误
- oracle - 将 JSON 动态检索到数据表 ORacle 12c 版本 1
- powerbi - Power-Query/Power-Bi:仅当列类型存在时如何更改它们?
- postgresql - Postgres View,alter table 更改表名后,View 还在查询吗?