nginx - 使用 gRPC 和 HTTP 的 kubernetes 入口

Question

我有一个部署到 kubernetes (AKS) 的应用程序，其中混合了 gRPC 和 http 服务。我最初将新 gRPC 服务的路由添加到以前仅提供 http 服务的现有入口。这不起作用并深入研究，我读到我们需要添加nginx.ingress.kubernetes.io/backend-protocol: GRPC注释，并且它适用于所有路由，因此我们需要两个单独的入口。我目前在io.grpc.internal.ManagedChannelImpl$NameResolverListener尝试使用 message 连接到 gRPC 服务时遇到异常错误nodename nor servname provided, or not known。我猜想，虽然当 Ingress 中的多个路径与请求匹配时，最长匹配路径优先，这不适用于两个入口。所以我需要使用不同的主机，或者更改/*路径以使其不匹配/results? 还是我需要在配置中更改其他内容？

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /graphql
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /graphql/*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
      - path: /*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-grpc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /results
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.externalResults.host }}
            port:
              number: 9000

标签： nginxkubernetesgrpckubernetes-ingressnginx-ingress

Answer 1

这最终通过创建指向我们的 k8s 集群的第二个主机名来解决。我将 grpc 服务的路由更改为 ImplementationSpecific 的根路径和 pathType。

      - path: /
        pathType: ImplementationSpecific

两个主机名都需要包含在两个入口的 tls 部分中。更改路由但未更新每个入口的 tls 部分中的主机后，我遇到了 SSL 异常。

Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
    at io.grpc.Status.asRuntimeException(Status.java:533)
    at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
  | => cat io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:721)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
stderr: 
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1771)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:776)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1079)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:970)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1443)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
stderr: 
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
stderr: 
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching grpc.aks.dev.app.cycleautomation.com found.
stderr: 
    at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
    at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
    at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:462)
    at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:428)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:261)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSslTlsv13X509ExtendedTrustManager.checkServerTrusted(OpenSslTlsv13X509ExtendedTrustManager.java:223)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:261)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:700)
    at io.grpc.netty.shaded.io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:595)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1202)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1324)
    at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1367)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
    at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1380)
    ... 21 more
    Suppressed: javax.net.ssl.SSLHandshakeException: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1287)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1248)
        ... 25 more

最终的 yaml 看起来像这样：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    - {{ .Values.ingress.grpc.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /graphql
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /graphql/*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
      - path: /*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-grpc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    - {{ .Values.ingress.grpc.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.externalResults.host }}
            port:
              number: 9000

然后我能够通过启用 tls 的端口 443 连接到 grpc 服务，并且只使用我的连接中没有路径的主机名。