multithreading - Golang中偏移值的循环实现,用于从couchbase n1ql查询中获取的记录数
问题描述
如何为偏移限制添加 RoundRobin 逻辑?只有使用 10 个线程,我才能在 10 个线程之间创建偏移值并以循环方式分配偏移值。我从 couchbase 获取 300 万条记录,这些记录在内部导致 n1ql 超时。
query := "select count(1) as count from `" + configuration.BucketName + "` where APPID='" + configuration.APPID + "'"
var countVariable int = 0
countVariable = countVariable + 1
fmt.Println("countVariable value: ", countVariable)
rows, err := cluster.Query(query, &gocb.QueryOptions{})
if err != nil {
panic(err)
}
var row map[string]interface{} //To fetch n1ql query output
for rows.Next() {
err := rows.Row(&row)
if err != nil {
panic(err)
}
}
countTotal := int64(row["count"].(float64))
fmt.Println("countTotal num value : ", countTotal)
if countTotal == 0 {
fmt.Println("No rows returned")
os.Exit(3)
}
float_offset_count := math.Ceil(float64(int(countTotal) / configuration.Limit))
offset_count := int(float_offset_count) + 1
fmt.Println("offset_count :", offset_count)
for i := 0; i < 10; i++ { // should be only 10 threads
wg.Add(1)
//fmt.Println("Goroutine :",i)
offset_limit := configuration.Limit * i // limit is 20000
go n1qlFetch(i, &wg, offset_limit) // This function contains n1ql query which fetches 3 to 4 ,million records
}
我正在尝试实现如下逻辑
total documents - 2000000
limit :20000
offset : 2000000/20000 = 100
threads :10
so offset value will be as below:
0
20000
40000
60000
80000
100000
.
.
.
.
2000000
thread 1 [0,20000,40000,60000,80000,100000,120000,140000,160000,180000,200000] --> 0 to 2lac records
thread 2 [220000,240000,260000,280000,300000,320000,340000,360000,380000,400000,420000]
thread 3 ""should be divided same as above
thread 4 ""should be divided same as above
thread 5 ""should be divided same as above
thread 6 ""should be divided same as above
thread 7 ""should be divided same as above
thread 8 ""should be divided same as above
thread 9 ""should be divided same as above
thread 10 ""should be divided same as above
for each ele in offsetlist:
offset = ele
解决方案
推荐阅读
- sql - 如何在 Oracle 查询中找到从开始日期到本周结束的周数?
- image-processing - 从零开始学习图像处理
- postgresql - PostgreSQL 函数总是在事务内部调用
- windows - 如何在 Windows 上使用 x86 perl 系统启动 x64 shell
- c++ - 如何将 Openh264 DecodeFrameNoDelay 输出格式转换为 OpenCV 矩阵?
- wordpress - wordpress 导出是否可以作为整个站点的备份?
- php - 使用ajax将值显示到组合框中?
- asp.net-mvc - 在 javascript 前端或 MVC 后端进行身份验证?
- javascript - 节点 Js 类语法
- javascript - JavaScript - 如何根据此函数的结构删除事件侦听器?