elasticsearch - 更新/创建文档后 ElasticSearch 崩溃
问题描述
当我想用大约 80 000 个具有 11 个字段的产品来更新我的数据库时，我遇到了问题，ES 崩溃了。我使用 symfony，并且每次更新或创建产品时，我都有一个订阅者来更新 ES 数据库。
但是当我一次更新所有数据库时，我遇到了这个错误：
[2021-10-12T02:11:22,931][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [30256ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:12:19,821][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [48557ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:12:34,931][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@78185ebe] took [6292ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:13:46,250][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [69667ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:14:04,113][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@7f02cfbb] took [7152ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:15:21,661][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [76026ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:15:43,542][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@3600a0f1] took [7618ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:17:26,994][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [100731ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:17:43,165][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@31403959] took [5589ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:18:41,718][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [57778ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:18:57,466][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@5b704f64] took [6443ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:20:00,132][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [61752ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:20:19,545][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@58d5ebe1] took [6910ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:21:44,007][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [83065ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:22:52,863][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [60421ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:23:02,766][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@b97d247, interval=1s}] took [6754ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:23:35,325][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@706bd30d] took [14295ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:23:47,068][WARN ][o.e.m.f.FsHealthService ] [new-host-name] health check of [/var/lib/elasticsearch/nodes/0] took [5288ms] which is above the warn threshold of [5s]
[2021-10-12T02:26:33,254][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [172616ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:26:47,312][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@b97d247, interval=1s}] took [7565ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:27:28,363][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5704/0x00000008018c6db8@77ba4c1] took [17452ms] which is above the warn threshold of [5000ms]
[2021-10-12T02:27:46,940][WARN ][o.e.m.f.FsHealthService ] [new-host-name] health check of [/var/lib/elasticsearch/nodes/0] took [10560ms] which is above the warn threshold of [5s]
[2021-10-12T02:30:55,163][WARN ][o.e.t.ThreadPool ] [new-host-name] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@2e1c6a7d, interval=5s}] took [198915ms] which is above the warn threshold of [5000ms]
[2021-10-12T09:21:03,323][INFO ][o.e.n.Node ] [new-host-name] version[7.15.0], pid[2069072], build[default/deb/79d65f6e357953a5b3cbcc5e2c7c21073d89aa29/2021-09-16T03:05:29.143308416Z], OS[Linux/5.4.0-88-generic/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
[2021-10-12T09:21:03,339][INFO ][o.e.n.Node ] [new-host-name] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-10-12T09:21:03,339][INFO ][o.e.n.Node ] [new-host-name] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms2g, -Xmx2g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4446609679435738103, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
我的索引设置：
$product_param = [
'index' => $this->indexName,
'body' => [
'settings' => [
'number_of_shards' => 2,
'number_of_replicas' => 2,
'analysis' => [
'char_filter' => [
'pre_negs' => [
'type' => 'pattern_replace',
'pattern' => 'a \w',
'replacement' => ''
],
],
'filter' => [
'autocomplete_filter' => [
'type' => 'edge_ngram',
'min_gram' => '1',
'max_gram' => '20',
]
],
"analyzer" => [
"autocomplete" => [
"filter" => [
"lowercase",
"autocomplete_filter"
],
"type" => "custom",
"tokenizer" => "standard",
]
]
]
],
'mappings' => [
'properties' => [
'pickRef' => [
'type' => 'keyword',
'boost' => 5
],
'name' => [
'type' => 'text',
'boost' => 3,
'fields' => [
"keyword" => [
"type" => "keyword",
"ignore_above" => 256
]
]
],
'brand' => [
'type' => 'keyword',
'boost' => 3
],
'description' => [
'type' => 'text',
],
'catalogue' => [
'type' => 'keyword',
],
'category' => [
'type' => 'text',
'analyzer' => 'standard'
],
'slug' => [
'type' => 'text',
],
'image' => [
'type' => 'text',
],
'enabled' => [
'type' => 'boolean',
],
'replaced' => [
'type' => 'boolean',
],
'synonym' => [
'type' => 'text',
'store' => 'false',
],
]
]
]
];
我的内存大约 2Gb。我以为只有 11 个字段的 80 000 个产品不会占用这么多资源。
我怎样才能避免这个错误，所以 ES 崩溃了？
解决方案
推荐阅读
- selenium - 如何根据提供的 HTML 定位元素？
- java - jframe 总是在另一个之上
- elasticsearch - Elasticsearch过滤器与许多ID的术语查询
- amazon-web-services - 如何在没有任何 lambda 函数的情况下创建 API Gateway HTTP 代理
- laravel - 我如何将用户、卖家和管理员重定向到他们在 laravel 5 中的面板
- php - AMP action-xhr 加载文件失败 403（禁止）
- .net - 在 .NET 中验证 RSA 签名
- reactjs - Redux Saga 发生错误时获取 axios 响应对象
- ms-access - 通过 Access 2010 VBA 运行 SQL 存储过程时出错
- oracle - Oracle ADF 登录（有时工作并主要重定向到登录页面本身）