powershell - 在 Powershell 中运行并行加载到 Clickhouse Docker 容器中
问题描述
我正在尝试通过 powershell 加快将数据加载到 Windows 10 上 docker 中托管的 clickhouse 的速度,我想知道是否可以利用并行进程同时加载 4 个文件。我有兴趣获得一些帮助以了解这是否可能,或者有关如何处理的一些指示。下面是我用来加载数据的当前脚本:
$files = Get-ChildItem "my_directory" | Sort-Object
foreach ($f in $files){
$outfile = $f.FullName | Write-Host
Get-Date | Write-Host
"Start loading" + $f.FullName | Write-Host
`cat $f.FullName | docker run -i --rm --link ch:clickhouse-client yandex/clickhouse-client -m --host ch --query="INSERT INTO my_table FORMAT CSV"`
Get-Date | Write-Host
"End loading" + $f.FullName | Write-Host
[GC]::Collect()
}
我正在一个一个地加载文件,我想一次加载 4 个。基于此链接:
我试图将代码放在一起,但可以使用一些帮助来查看我是否走在正确的轨道上:
#I am assuming this is the code block of what to do
$block = {
$outfile = $f.FullName | Write-Host
Get-Date | Write-Host
"Start loading" + $f.FullName | Write-Host
`cat $f.FullName | docker run -i --rm --link ch:clickhouse-client yandex/clickhouse-client -m --host ch --query="INSERT INTO my_table FORMAT CSV"`
Get-Date | Write-Host
"End loading" + $f.FullName | Write-Host
[GC]::Collect())
}
#my directory of files
$files = Get-ChildItem "my_directory" | Sort-Object| Sort-Object
#Remove all jobs
Get-Job | Remove-Job
$MaxThreads = 4
#Start the jobs. Max 4 jobs running simultaneously.
foreach($f in $files){
While ($(Get-Job -state running).count -ge $MaxThreads){
Start-Sleep -Milliseconds 3
}
Start-Job -Scriptblock $Block -ArgumentList $f
}
#Wait for all jobs to finish.
While ($(Get-Job -State Running).count -gt 0){
start-sleep 1
}
#Get information from each job.
foreach($job in Get-Job){
$info= Receive-Job -Id ($job.Id)
}
#Remove all jobs created.
Get-Job | Remove-Job
powershell 新手,感谢帮助。
解决方案
我相信我解决了这个问题:
#create my block
$direc = "my_direc"
$block = {
param([string]$file)
`cat $direc/$file | docker run -i --rm --link ch:clickhouse-client yandex/clickhouse-client -m --host ch --query="INSERT INTO test FORMAT CSV"`
[GC]::Collect()
}
#Remove all jobs
Get-Job | Remove-Job
$MaxThreads = 4
#Start the jobs. Max 6 jobs running simultaneously.
foreach($file in $files){
While ($(Get-Job -state running).count -ge $MaxThreads){
Start-Sleep -Milliseconds 3
}
Start-Job -Scriptblock $Block -ArgumentList $file
}
#Wait for all jobs to finish.
While ($(Get-Job -State Running).count -gt 0){
start-sleep 1
}
#Get information from each job.
foreach($job in Get-Job){
$info= Receive-Job -Id ($job.Id)
}
#Remove all jobs created.
Get-Job | Remove-Job
推荐阅读
- reactjs - 停止模式关闭外部点击
- c# - Android - Java.Lang.NoClassDefFoundError: android.support.v7.widget.AppCompatImageButton
- javascript - 通过angularjs中的2个指令将函数作为参数传递
- javascript - 将变量设置为等于从 JavaScript 中的异步函数返回数据
- javascript - 无法读取未定义的属性“arrayToDataTable”
- javascript - npm 'websocket' 服务器向客户端发送消息
- java - @HystrixCommand 中未调用后备方法
- java - 使用反射获取内部类数据
- angular - 使用角度和 aspnet 核心获取 CORS 错误
- excel - 数据字典格式