java - 配置 Hazelcast CPSubsystem 重试超时
问题描述
目前我在CPSubsystem
.
-----
| I1* | * Leader
-----
---- ----
| I2 | | I3 |
---- ----
当所有实例都运行起来时,所有实例都已注册并在所有内容上相互查看,CPSubsystem
一切都按预期工作。以下调用用于在所有实例之间执行分布式锁:
getHazelcastInstance().getCpSubsystem().getLock(lockDefinition.getLockEntryName())
当其中两个实例死亡时,我注意到一个问题,并且没有领导者或其他可用于执行领导者选举的实例:
-----
| XXX | * DEAD
-----
---- -----
| I2 | | XXX | * DEAD
---- -----
正在运行的实例然后尝试获取分布式锁,但请求冻结执行该getLock
方法,导致请求排队几分钟(当实例成为子系统中唯一的实例时,需要配置超时)。
我还注意到以下日志被永久打印:
2019-08-16 10:56:21.697 WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:23.737 WARN 1337 --- [cached.thread-8] c.h.nio.tcp.TcpIpConnectionErrorHandler : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 106
2019-08-16 10:56:23.927 WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:26.006 WARN 1337 --- [onMonitorThread] c.h.s.i.o.impl.Invocation : [127.0.0.1]:5702 [dev] [3.12.1] Retrying invocation: Invocation{op=com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp{serviceName='hz:core:raft', identityHash=1295439737, partitionId=81, replicaIndex=0, callId=1468, invocationTime=1565963786004 (2019-08-16 10:56:26.004), waitTimeout=-1, callTimeout=60000, groupId=CPGroupId{name='default', seed=0, commitIndex=6}, membersCommitIndex=0, member=CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, membershipChangeMode=REMOVE}, tryCount=250, tryPauseMillis=500, invokeCount=130, callTimeoutMillis=60000, firstInvocationTimeMs=1565963740657, firstInvocationTime='2019-08-16 10:55:40.657', lastHeartbeatMillis=0, lastHeartbeatTime='1969-12-31 21:00:00.000', target=[127.0.0.1]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.core.MemberLeftException: Member [127.0.0.1]:5702 - ab45ea09-c8c9-4f03-b3db-42b7b440d161 this has left cluster!
2019-08-16 10:56:26.232 WARN 1337 --- [cached.thread-8] c.h.nio.tcp.TcpIpConnectionErrorHandler : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 107
2019-08-16 10:56:26.413 WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:27.143 WARN 1337 --- [onMonitorThread] c.h.s.i.o.impl.Invocation : [127.0.0.1]:5702 [dev] [3.12.1] Retrying invocation: Invocation{op=com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp{serviceName='hz:core:raft', identityHash=1295439737, partitionId=81, replicaIndex=0, callId=1479, invocationTime=1565963787142 (2019-08-16 10:56:27.142), waitTimeout=-1, callTimeout=60000, groupId=CPGroupId{name='default', seed=0, commitIndex=6}, membersCommitIndex=0, member=CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, membershipChangeMode=REMOVE}, tryCount=250, tryPauseMillis=500, invokeCount=140, callTimeoutMillis=60000, firstInvocationTimeMs=1565963740657, firstInvocationTime='2019-08-16 10:55:40.657', lastHeartbeatMillis=0, lastHeartbeatTime='1969-12-31 21:00:00.000', target=[127.0.0.1]:5703, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.spi.exception.TargetNotMemberException: Not Member! target: CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, partitionId: 81, operation: com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp, service: hz:core:raft
2019-08-16 10:56:28.835 WARN 1337 --- [cached.thread-6] c.h.nio.tcp.TcpIpConnectionErrorHandler : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 108
2019-08-16 10:56:28.941 WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:31.038 WARN 1337 --- [cached.thread-3] c.h.nio.tcp.TcpIpConnectionErrorHandler : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 109
2019-08-16 10:56:31.533 WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:31.555 WARN 1337 --- [.async.thread-3] c.h.s.i.o.impl.Invocation : [127.0.0.1]:5702 [dev] [3.12.1] Retrying invocation: Invocation{op=com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp{serviceName='hz:core:raft', identityHash=1295439737, partitionId=81, replicaIndex=0, callId=1493, invocationTime=1565963791554 (2019-08-16 10:56:31.554), waitTimeout=-1, callTimeout=60000, groupId=CPGroupId{name='default', seed=0, commitIndex=6}, membersCommitIndex=0, member=CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, membershipChangeMode=REMOVE}, tryCount=250, tryPauseMillis=500, invokeCount=150, callTimeoutMillis=60000, firstInvocationTimeMs=1565963740657, firstInvocationTime='2019-08-16 10:55:40.657', lastHeartbeatMillis=0, lastHeartbeatTime='1969-12-31 21:00:00.000', target=[127.0.0.1]:5702, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.cp.exception.NotLeaderException: CPMember{uuid=ab45ea09-c8c9-4f03-b3db-42b7b440d161, address=[127.0.0.1]:5702} is not LEADER of CPGroupId{name='default', seed=0, commitIndex=6}. Known leader is: N/A
有没有办法识别实例现在单独运行,如果是,在获取新锁期间不要阻塞应用程序?
我一直在寻找某种机制来不以任何方式阻塞应用程序的流程,即使应用程序单独运行,我也会使用常规j.u.c.l.ReentrantLock
而不是FencedLock
.
解决方案
CP 子系统被设计为在没有足够的可用成员首先形成 CP 子系统时阻止对属于 CP 子系统族的数据结构的所有操作。此属性由CPSubsystemConfig.setCPMemberCount(int)
hazelcastInstance.getCPSubsystem().getCPSubsystemManagementService().getCPMembers()
将为您提供集群中的 CP 成员。
要确定集群成员计数,您可以hazelcastInstance.getCluster().getMembers()
对加入或离开事件的成员使用和/或使用 MembershipListener。
推荐阅读
- jwt - 如何描述在 OpenAPI 3 的响应正文中发送的 JWT?
- php - mysqli_fetch_all 在 PHP 7.3 中引发未定义函数错误
- python - Django3.0:访问模型表单时出现GDALException
- c++ - 如何将 std::swap 与数组一起使用?
- python - 将 Datetime.Date 转换为 DT 格式 Int
- r - 以“.”开头的变量名
- mysql - 如何将 MySQL 5.6 DB 中的排序规则更改为 utf8mb4_unicode_ci?
- scripting - 我在 IronPython、Spotfire 中有一个存储为字符串的列表。如何以原始列表格式检索它?
- python - 在函数中使用比较语句作为参数,然后在while循环中作为条件传递
- java - 为什么scanner.hasNext() 在我放入emtry 行时不会停止循环?