时间戳

首页 > 解决方案 > 将 ClickHouse 从 1.1.54231 升级到 19.6.2.11

clickhouse - 将 ClickHouse 从 1.1.54231 升级到 19.6.2.11

问题描述

我们最终决定将我们的生产服务器 ClickHouse 版本 1.1.54231 更新为新的 19.6 。新版本启动，拾取了所有配置，也看到了服务器上的数据，但无法在 Zookeeper 上拾取新数据。在日志错误中

    Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: user_files_path (version 19.6.2.11 (official build))
    Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: format_schema_path (version 19.6.2.11 (official build))
    Include not found: clickhouse_remote_servers
    Include not found: clickhouse_compression
    Logging error to /var/log/clickhouse-server/clickhouse-server.log
    Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
    Logging error to console
    2019.05.16 17:06:10.801816 [ 1 ] {} <Error> Application: Listen [0.0.0.0]: 98: Net Exception: Address already in use: 0.0.0.0:8123  If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
    2019.05.16 17:06:10.859589 [ 25 ] {} <Error> analytics..inner.main_page_hits (StorageReplicatedMergeTree): Code: 40, e.displayText() = DB::Exception: Checksums of parts don't match: hash of uncompressed files doesn't match (version 19.6.2.11 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons: 
1. Using newer version of compression library after server update. 
2. Using another compression method. 
3. Non-deterministic compression algorithm (highly unlikely). 
4. Non-deterministic merge algorithm due to logical error in code. 
5. Data corruption in memory due to bug in code. 
6. Data corruption in memory due to hardware issue. 
7. Manual modification of source data after server startup. 
8. Manual modification of checksums stored in ZooKeeper. We will download merged part from replica to force byte-identical result.
    2019.05.16 17:06:10.874886 [ 23 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8., Stack trace:

    0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x767c3f6]
    1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x39294a2]
    2. /usr/bin/clickhouse-server() [0x6d7fc8c]
    3. /usr/bin/clickhouse-server(DB::DataPartsExchange::Fetcher::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, DB::ConnectionTimeouts const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x950) [0x6d830b0]
    4. /usr/bin/clickhouse-server() [0x6d2d7a4]
    5. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, unsigned long)+0x97f) [0x6d48d5f]
    6. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&)+0x7f2) [0x6d4b012]
    7. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&)+0x26b) [0x6d4bd5b]
    8. /usr/bin/clickhouse-server() [0x6d4c31f]
    9. /usr/bin/clickhouse-server(DB::ReplicatedMergeTreeQueue::processEntry(std::function<std::shared_ptr<zkutil::ZooKeeper> ()>, std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::function<bool (std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>)+0x64) [0x6e6d324]
    10. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::queueTask()+0x189) [0x6d280d9]
    11. /usr/bin/clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x44e) [0x6d7da4e]
    12. /usr/bin/clickhouse-server() [0x6d7e4c9]
    13. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x1ab) [0x768238b]
    14. /usr/bin/clickhouse-server() [0xb5488bf]
    15. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f553fd076db]
    16. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f553f28688f]
     (version 19.6.2.11 (official build))
    2019.05.16 17:06:10.886495 [ 25 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8.,

有任何想法吗？

标签： clickhouse

解决方案

看起来您只更新了少数副本中的一个。

在复制场景中，后台合并结果使用校验和进行交叉检查。如果 2 个副本的合并结果不是字节相同的副本，则将该消息发送到日志并从其他副本重新下载有问题的部分。在您更新其他副本后，问题将消失。

推荐阅读