时间戳

首页 > 解决方案 > 如何为 K8s 上的 Elasticsearch 部署应用配置？

elasticsearch - 如何为 K8s 上的 Elasticsearch 部署应用配置？

问题描述

我在更改部署在 K8s 中的 Elasticsearch 配置时遇到了麻烦。我想将此配置应用于我的 Elasticsearch 节点

# Force all memory to be locked, forcing the JVM to never swap
bootstrap.mlockall: true

## Threadpool Settings ##

# Search pool
threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: 100

# Bulk pool
threadpool.bulk.type: fixed
threadpool.bulk.size: 60
threadpool.bulk.queue_size: 300

# Index pool
threadpool.index.type: fixed
threadpool.index.size: 20
threadpool.index.queue_size: 100

 # Indices settings
indices.memory.index_buffer_size: 30%
indices.memory.min_shard_index_buffer_size: 12mb
indices.memory.min_index_buffer_size: 96mb

# Cache Sizes
indices.fielddata.cache.size: 15%
indices.fielddata.cache.expire: 6h
indices.cache.filter.size: 15%
indices.cache.filter.expire: 6h

# Indexing Settings for Writes
index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000

要在 K8s 中部署 Elasticsearch，我遵循两个步骤： 第 1 步：创建 .yml 文件，如下所示

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: esc-lamtest
  namespace: lamtest
spec:
  version: 7.9.0
  nodeSets:
    - name: basic-1
    count: 3
    config:
        node.master: true
        node.data: true
        node.ingest: true
        # Search pool
        thread_pool.search.queue_size: 50
        thread_pool.search.size: 20
        thread_pool.search.min_queue_size: 10
        thread_pool.search.max_queue_size: 100  
volumeClaimTemplates:
- metadata:
    name: elasticsearch-data
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 150Gi
    storageClassName: vnptit-nfs
podTemplate:
  spec:
    containers:
    - name: elasticsearch
      env:
      - name: ES_JAVA_OPTS
        value: -Xms4g -Xmx4g
      resources:
        requests:
            memory: 0Gi
            cpu: 0Gi
        limits:
            memory: 0Gi
            cpu: 0Gi

http：tls：selfSignedCertificate：禁用：真

第2步：我运行命令行：

kubectl apply -f lamtest.yaml

但我只能为“搜索池”应用配置。当我为 应用配置时# Bulk pool # Index pool # Indices settings # Cache Sizes and # Indexing Settings for Writes，我的 Elasticsearch 失败，这是我的日志

"Suppressed: java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.queue_size] did   you mean any of [thread_pool.get.queue_size, thread_pool.write.queue_size, thread_pool.analyze.queue_size, thread_pool.search.queue_size, thread_pool.listener.queue_size]?", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.node.Node.<init>(Node.java:385) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]", 
"\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "Suppressed: java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.size] did you mean any of [thread_pool.get.size, thread_pool.write.size, thread_pool.analyze.size, thread_pool.search.size]?", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.node.Node.<init>(Node.java:385) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.0.jar:7.9.0]", 
 "\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.0.jar:7.9.0]"] } 
 uncaught exception in thread [main] 
 java.lang.IllegalArgumentException: unknown setting [thread_pool.bulk.type] please check that any  required plugins are installed, or check the breaking changes documentation for removed settings 
 at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) 
 at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) 
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) 
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) 
at org.elasticsearch.node.Node.<init>(Node.java:385) 
at org.elasticsearch.node.Node.<init>(Node.java:277) 
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) 
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) 
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) 
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) 
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) 
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) 
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) 
at org.elasticsearch.cli.Command.main(Command.java:90) 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) 
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

有关完整的错误详细信息，请参阅 /usr/share/elasticsearch/logs/esc-lamtest.log 中的日志

标签： elasticsearch

解决方案

错误信息非常清楚，您没有为其他线程池定义正确的配置，请仔细注意错误信息的第一行。

“抑制：java.lang.IllegalArgumentException：未知设置 [thread_pool.bulk.queue_size] 你的意思是 [thread_pool.get.queue_size, thread_pool.write.queue_size, thread_pool.analyze.queue_size, thread_pool.search.queue_size, thread_pool。 listener.queue_size]?",

您已将批量队列大小定义为，这是不正确的，因为您可以在 ESthread_pool.bulk.queue_size中读取线程池，所以没有线程池 for ，而是使用线程池来处理来自同一文档的请求，因此将其更改为适用于此配置。bulkwritebulkthread_pool.write.queue_size

write用于单文档索引/删除/更新和批量请求。线程池类型的大小固定为分配的处理器数，queue_size 为 10000。此池的最大大小为 1 + 分配的处理器数。

现在为了解决这个问题bulk，Index以及其他设置，请验证您使用的配置名称是否正确，threadpools可以从任何正在运行的 ES 实例中获取，并且您可以轻松构建相应的配置。

http://localhost:9200/_nodes/stats?pretty 给出了所有这些threadpools，下面列出了一些。

"thread_pool": {
"analyze": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 1,
"completed": 1
},
"ccr": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
}

推荐阅读