python - 当 csv 具有大列时，Logstash 进程被终止

Question

我有一个 6gb ram 和存储的 pod，我一直在使用 logstash 将数据从 csv 同步到 Elastic Search，但同步一些数据后，logstash 进程被终止。我也用最少 1 个工人 500 批尝试过

**CSV chunk** {
Rows=> 5 million 
columns=> 641
}

**pod configuration** 
{
ram=> 6gb
storage => 6gb
}

**logstash config** :- 
input {  
        file {
                path => "/app/table1.csv" #ssv of 5m rows rows and 641 columns  
                start_position => "beginning"
                sincedb_path => "/dev/null"
                ignore_older => 36000000 
                close_older => 36000000
            } 
        }
        
        filter {
            csv {
                separator => ";"#csv separator 
                remove_field => ["message"]
                
}}

output { 
                elasticsearch {
                    hosts => ["localhost:9200"] 
                    index => "activity_test04232_3"
                    retry_max_interval => 5
                    retry_initial_interval => 30
                }#output ES host 
            }

标签： pythonlinuxkubernetesoperating-systemlogstash