时间戳

首页 > 解决方案 > 将 Couchbase operator 1.1 与 Istio 服务网格一起使用

kubernetes - 将 Couchbase operator 1.1 与 Istio 服务网格一起使用

问题描述

我正在尝试使用 Autonomous Operator 1.1 在 GKE 中使用 Istio（特使代理）创建一个 Couchbase 集群。操作员启动正常，运行 yaml 以创建 couchbasecluster 后，第一个节点启动，然后第二个节点启动。问题是第二个节点似乎无法加入集群，并且没有启动其他节点。

我不确定如何调试正在发生的事情或需要做什么才能让集群在我的 gke 集群中启动。任何帮助表示赞赏。

谢谢

以下是来自 couchbase 节点 pod 的一些日志：

I  [2019-04-02T14:58:00.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "782bde60-c611-4bfb-a0f4-9975300c71a4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:37221

I  [2019-04-02T14:58:05.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "382b6163-e8bc-4259-baaa-e854c36af1bd" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:55515

I  [2019-04-02T14:58:10.707Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "390e417e-b179-4bbf-81d8-02cc28d2bc98" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:34377

I  [2019-04-02T14:53:13.605Z] - 210 4281 300015 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:45756 10.36.8.13:8091 10.36.9.12:49792

I  [2019-04-02T14:58:15.709Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "037d1791-9feb-47be-b699-10269aaf36e9" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:55307

I  [2019-04-02T14:58:20.708Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "5ca29b59-ff25-4a13-a0c1-62668d40c681" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:51205

I  [2019-04-02T14:58:25.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "9e21bc4d-1367-4d25-b674-39ae6341c9b4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:41435

I  [2019-04-02T14:58:30.710Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "c2f8e866-e0a5-43ff-b54f-e5c504b17cdf" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:40203

I  [2019-04-02T14:58:35.708Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "4b02e855-cc72-49dc-99e1-a8644fdf1af8" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:56433

I  [2019-04-02T14:53:13.641Z] - 16628 40061 324989 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:45760 10.36.8.13:8091 10.36.9.12:49796

I  [2019-04-02T14:56:45.698Z] - 9490 13635 112934 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:46218 10.36.8.13:8091 10.36.9.12:50534

I  [2019-04-02T14:56:45.665Z] - 210 4281 112967 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:46216 10.36.8.13:8091 10.36.9.12:50528

以及来自 couchbase 容器内部的 error.log 的一部分。

[ns_server:error,2019-04-03T16:09:47.398Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.23572.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.23572.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1@cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.23562.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.23516.68>,
                                         {no_connection,
                                          "index-service_api"}}}}
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:<0.24979.68>:service_agent:wait_for_connection_loop:299]No connection with label "index-service_api" after 60000ms. Exiting.
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:handle_info:231]Linked process <0.24979.68> died with reason {no_connection,
                                              "index-service_api"}. Terminating
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:terminate:260]Terminating abnormally
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.25043.68>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.48.97712>,process,
                                        <0.24974.68>,
                                        {linked_process_died,<0.24979.68>,
                                         {no_connection,"index-service_api"}}}
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.25043.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.25043.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1@cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.25042.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.24979.68>,
                                         {no_connection,
                                          "index-service_api"}}}}
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:<0.26456.68>:service_agent:wait_for_connection_loop:299]No connection with label "index-service_api" after 60000ms. Exiting.
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:handle_info:231]Linked process <0.26456.68> died with reason {no_connection,
                                              "index-service_api"}. Terminating
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:terminate:260]Terminating abnormally
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.26515.68>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.48.106235>,process,
                                        <0.26461.68>,
                                        {linked_process_died,<0.26456.68>,
                                         {no_connection,"index-service_api"}}}
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.27939.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.26515.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.26515.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1@cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.26517.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.26456.68>,
                                         {no_connection,
                                          "index-service_api"}}}}

这是来自 couchbase-operator 日志的当前部分：

I  [2019-04-03T16:15:13.959Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "cc976505-818a-4930-9fc8-8bdcb047185d" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59280

I  [2019-04-03T16:15:13.963Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "bfd981b2-9356-4132-a7f8-2a6c0d8ba15f" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57624

I  [2019-04-03T16:15:14.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37568 10.36.9.12:8080 10.36.9.1:44810

I  [2019-04-03T16:15:17.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37574 10.36.9.12:8080 10.36.9.1:44816

I  [2019-04-03T16:15:18.959Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "997c061a-d5d2-425d-b123-bf76073d148a" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59298

I  [2019-04-03T16:15:18.962Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "b03def85-726f-4107-8b27-9fc8b5bddea7" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57642

I  [2019-04-03T16:15:20.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37586 10.36.9.12:8080 10.36.9.1:44828

E  time="2019-04-03T16:15:26Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

E  time="2019-04-03T16:15:31Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

I  [2019-04-03T16:15:23.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37592 10.36.9.12:8080 10.36.9.1:44834

I  [2019-04-03T16:15:26.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37604 10.36.9.12:8080 10.36.9.1:44846

I  [2019-04-03T16:15:26.987Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "c20d057e-646d-4eb3-8931-a220126c27d5" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59326

I  [2019-04-03T16:15:26.991Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "c46e9753-1249-4b8c-8fc9-889e74a0d70b" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57670

I  [2019-04-03T16:15:29.939Z] - 119 135 1 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37616 10.36.9.12:8080 10.36.9.1:44858

I  [2019-04-03T16:15:31.986Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "30c8522f-5797-488c-b234-d1c5a43d9826" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59338

I  [2019-04-03T16:15:31.990Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "a4f74a5e-c026-46b4-b9e3-8b20f65477b4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57682

E  time="2019-04-03T16:15:36Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

E  time="2019-04-03T16:15:41Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

I  [2019-04-03T16:15:32.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37624 10.36.9.12:8080 10.36.9.1:44866

标签： kubernetescouchbasegoogle-kubernetes-engineistio

解决方案

推荐阅读