cassandra - Cassandra 节点不断抛出此错误
问题描述
我有 6 个节点 Cassandra 集群,其中 3 个是数据节点,3 个是种子节点
他们都在使用 Cassandra 3.7 版
以下是我在数据节点上遇到的例外情况:
java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
at org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:220) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getEstimatedDroppableTombstoneRatio(StatsMetadata.java:115) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.io.sstable.format.SSTableReader.getEstimatedDroppableTombstoneRatio(SSTableReader.java:1829) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:387) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:99) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:183) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.compaction.CompactionStrategyManager.getNextBackgroundTask(CompactionStrategyManager.java:119) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258) ~[apache-cassandra-3.7.jar:3.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_222]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_222]
下面是 nodetool cfhistograms 的输出
I have one keyspace with 4 tables below is the output of nodetool cfhistorgrams
ubuntu@ankur:~$ nodetool cfhistograms keyspace.tabelname
Column counts are larger than 1996099046, unable to calculate percentiles
keyspace.tabelenamehistograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 20.00 29.52 14530.76 9887 NaN
75% 24.00 35.43 25109.16 88148 NaN
95% 24.00 42.51 36157.19 4866323 NaN
98% 24.00 51.01 52066.35 36157190 NaN
99% 24.00 61.21 52066.35 155469300 NaN
Min 2.00 6.87 61.22 61 0
Max 24.00 10299432.64 268650.95 765256466109223372036854776000
ubuntu@ankur:~$ nodetool cfhistogramskeyspace.tablename
keyspace.tabelename histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 0.00 20.50 0.00 215 8
75% 0.00 24.60 0.00 310 14
95% 0.00 29.52 0.00 446 24
98% 0.00 29.52 0.00 535 29
99% 0.00 35.43 0.00 535 29
Min 0.00 3.31 0.00 73 0
Max 0.00 10299432.64 0.00 4055269 219342
ubuntu@ankur:~$ nodetool cfhistograms keyspace.tabelename
keyspace.tabelename histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 24.00 24.60 1629.72 446 24
75% 29.00 29.52 2816.16 535 29
95% 29.00 35.43 7007.51 924 50
98% 29.00 42.51 12108.97 1109 60
99% 29.00 42.51 14530.76 1331 86
Min 2.00 5.72 20.50 51 0
Max 35.00 10299432.64 557074.61 4866323 654949
ubuntu@ankur:~$ nodetool cfhistograms keyspace.tabelename
keyspace.tabelename histograms
Percentile SSTables Write Latency Read Latency Partition Size Cell Count
(micros) (micros) (bytes)
50% 0.00 14.24 0.00 124 2
75% 0.00 17.08 0.00 149 2
95% 0.00 24.60 0.00 215 4
98% 0.00 24.60 0.00 372 7
99% 0.00 29.52 0.00 535 12
Min 0.00 1.33 0.00 36 0
Max 0.00 10299432.64 0.00 8409007 182785
下面是我们为所有表格遵循的表格结构
CREATE TABLE keyspace.tablename (
xxxx uuid,
xxxx timeuuid,
xxxx text,
xxxx text,
xxxx text,
xxxx uuid,
xxxx text,
city text,
xxxx text,
xxxx text,
xxxx text,
xxxx timeuuid,
xxxx text,
xxxx text,
xxxx text,
xxxx text,
xxxx text,
xxxx text,
xxxx text,
xxxx text,
convtype text,
xxxx float,
xxxx float,
xxxx float,
xxxx float,
xxxx text,
xxxx text,
xxxx ascii,
xxxx text,
device text,
eventxxxxaction text,
xxxx text,
xxxx text,
xxxx int,
xxxx int,
xxxx text,
xxxx uuid,
xxxx int,
ip inet,
xxxx float,
xxxx int,
xxxx uuid,
xxxx float,
xxxx boolean,
xxxx uuid,
xxxx text,
params map<text, text>,
referrer text,
servicelist set<frozen<service_item>>,
xxxx text,
xxxx text,
uxxxxseragent text,
xxxx text,
xxxx text,
PRIMARY KEY (xxx)
) WITH CLUSTERING ORDER BY (created DESC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 86400
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
任何线索或建议将不胜感激
谢谢 Ankur
解决方案
推荐阅读
- php - 来自数据库的 Symfony4 数据不是 utf8 编码的
- python - 如何以编程方式访问流编辑器?
- java - 如何正确使用具有多个值的枚举
- angular - 如何在 flickr Rest API 上使用照片 url 获取照片信息?
- java - RxAndroidBle - 长写和收听通知特性
- json - 多行字符串作为 Json 中的值
- elixir - Phoenix 模块未实现 Ecto Queryable
- python - pytest:如何测试项目相关目录的创建?
- swift - Vapor Swift - 从两个模型中获取数据
- javascript - Firestore 云函数对子集合值求和