时间戳

首页 > 解决方案 > Apache Spark 2.2.1 中的 Google PubSub

apache-spark - Apache Spark 2.2.1 中的 Google PubSub

问题描述

我正在尝试在 Spark 应用程序中使用 Google Cloud PubSub。为简单起见，我们只说这个应用程序是 Spark 的外壳。尝试实例化 aPublisher会抛出 a NoClassDefFoundError，这很可能是依赖版本冲突的结果。但是，通过这样的简单设置（只是 Spark 和 Google Cloud PubSub 依赖项），我无法弄清楚如何解决这个问题。

bash-4.4# spark-shell --packages com.google.cloud:google-cloud-pubsub:1.105.0
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/opt/spark-2.2.1-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.google.cloud#google-cloud-pubsub added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
    confs: [default]
    found com.google.cloud#google-cloud-pubsub;1.105.0 in central
    found io.grpc#grpc-api;1.28.1 in central
    ...
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

scala> com.google.cloud.pubsub.v1.Publisher.newBuilder("topic").build
java.lang.NoClassDefFoundError: com/google/api/gax/grpc/InstantiatingGrpcChannelProvider
  at com.google.cloud.pubsub.v1.stub.PublisherStubSettings.defaultGrpcTransportProviderBuilder(PublisherStubSettings.java:225)
  at com.google.cloud.pubsub.v1.TopicAdminSettings.defaultGrpcTransportProviderBuilder(TopicAdminSettings.java:169)
  at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:674)
  at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:625)
  at com.google.cloud.pubsub.v1.Publisher.newBuilder(Publisher.java:621)
  ... 48 elided
Caused by: java.lang.ClassNotFoundException: com.google.api.gax.grpc.InstantiatingGrpcChannelProvider
  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 53 more

有什么办法可以让它工作吗？我可以更改 pubsub 依赖版本，但不能更改 Spark 版本。

标签： apache-sparkgoogle-cloud-pubsub

解决方案

这是由于 Google 的 Guava 依赖冲突，已知在使用 Spark + Google Libraries 时都会存在这种依赖冲突。解决方法（使用 Maven）是使用 maven-shade-plugin。

推荐阅读