apache-spark - 无法为 shc-core 导入 sbt 依赖项
问题描述
我正在尝试使用 spark 连接到 hbase,并且我想使用 hortonworks shc,但是我无法为 shc-core 导入 sbt 依赖项(尽管我可以导入 shc)。以下是built.sbt 快照-
name := "untitled"
version := "0.1"
scalaVersion := "2.11.12"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.2"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.2"
resolvers += "Hortonworks Repository" at "http://repo.hortonworks.com/content/repositories/releases/"
// https://mvnrepository.com/artifact/com.hortonworks.shc/shc
libraryDependencies += "com.hortonworks.shc" % "shc" % "1.1.0.3.0.2.0-50" pomOnly()
// https://mvnrepository.com/artifact/com.hortonworks.shc/shc-core
libraryDependencies += "com.hortonworks.shc" % "shc-core" % "1.1.0.3.0.2.0-50"
我收到以下错误(不包括开头几行)-
[info] Resolving org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook ...
[info] Resolving org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook ...
[warn] module not found: org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook
[warn] ==== local: tried
[warn] /home/ashish/.ivy2/local/org.apache.hadoop/hadoop-aws/working@ashish-HP-Notebook/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/working@ashish-HP-Notebook/hadoop-aws-working@ashish-HP-Notebook.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/ashish/.sbt/preloaded/org.apache.hadoop/hadoop-aws/working@ashish-HP-Notebook/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/ashish/.sbt/preloaded/org/apache/hadoop/hadoop-aws/working@ashish-HP-Notebook/hadoop-aws-working@ashish-HP-Notebook.pom
[warn] ==== Hortonworks Repository: tried
[warn] http://repo.hortonworks.com/content/repositories/releases/org/apache/hadoop/hadoop-aws/working@ashish-HP-Notebook/hadoop-aws-working@ashish-HP-Notebook.pom
[info] Resolving jline#jline;2.14.3 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.github.stephenc.jcip#jcip-annotations;1.0-1: com.github.stephenc.jcip#jcip-annotations;1.0-1!jcip-annotations.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/github/stephenc/jcip/jcip-annotations/1.0-1/jcip-annotations-1.0-1.pom
[warn] :: com.google.guava#guava;13.0.1: com.google.guava#guava;13.0.1!guava.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.pom
[warn] :: com.google.inject.extensions#guice-assistedinject;3.0: com.google.inject.extensions#guice-assistedinject;3.0!guice-assistedinject.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.pom
[warn] :: ch.qos.logback#logback-classic;1.0.9: ch.qos.logback#logback-classic;1.0.9!logback-classic.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/ch/qos/logback/logback-classic/1.0.9/logback-classic-1.0.9.pom
[warn] :: ch.qos.logback#logback-core;1.0.9: ch.qos.logback#logback-core;1.0.9!logback-core.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/ch/qos/logback/logback-core/1.0.9/logback-core-1.0.9.pom
[warn] :: org.apache.thrift#libthrift;0.9.0: org.apache.thrift#libthrift;0.9.0!libthrift.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/apache/thrift/libthrift/0.9.0/libthrift-0.9.0.pom
[warn] :: it.unimi.dsi#fastutil;6.5.6: it.unimi.dsi#fastutil;6.5.6!fastutil.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/it/unimi/dsi/fastutil/6.5.6/fastutil-6.5.6.pom
[warn] :: org.antlr#antlr-runtime;3.5.2: org.antlr#antlr-runtime;3.5.2!antlr-runtime.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.pom
[warn] :: sqlline#sqlline;1.2.0: sqlline#sqlline;1.2.0!sqlline.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/sqlline/sqlline/1.2.0/sqlline-1.2.0.pom
[warn] :: org.iq80.snappy#snappy;0.3: org.iq80.snappy#snappy;0.3!snappy.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/iq80/snappy/snappy/0.3/snappy-0.3.pom
[warn] :: org.apache.commons#commons-csv;1.0: org.apache.commons#commons-csv;1.0!commons-csv.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/apache/commons/commons-csv/1.0/commons-csv-1.0.pom
[warn] :: org.ow2.asm#asm;5.0.4: org.ow2#ow2;1.3!ow2.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/org/ow2/ow2/1.3/ow2-1.3.pom
[warn] :: org.apache.hadoop#hadoop-aws;working@ashish-HP-Notebook: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.apache.hadoop:hadoop-aws:working@ashish-HP-Notebook
[warn] +- org.apache.hbase:hbase-client:2.0.0.3.0.2.0-50
[warn] +- org.apache.hbase:hbase-replication:2.0.0.3.0.2.0-50
[warn] +- org.apache.hbase:hbase-mapreduce:2.0.0.3.0.2.0-50
[warn] +- org.apache.phoenix:phoenix-core:5.0.0.3.0.2.0-50
[warn] +- com.hortonworks.shc:shc-core:1.1.0.3.0.2.0-50 (/home/ashish/sparkprojects/untitled/build.sbt#L20-21)
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: com.github.stephenc.jcip#jcip-annotations;1.0-1: com.github.stephenc.jcip#jcip-annotations;1.0-1!jcip-annotations.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/github/stephenc/jcip/jcip-annotations/1.0-1/jcip-annotations-1.0-1.pom
[error] unresolved dependency: com.google.guava#guava;13.0.1: com.google.guava#guava;13.0.1!guava.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/guava/guava/13.0.1/guava-13.0.1.pom
[error] unresolved dependency: com.google.inject.extensions#guice-assistedinject;3.0: com.google.inject.extensions#guice-assistedinject;3.0!guice-assistedinject.pom(pom.original) origin location must be absolute: file:/home/ashish/.m2/repository/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.pom
我正在使用 spark 2.2.2 和 sbt .13 并使用 ubuntu。
此外,当我在 Windows 中执行此操作时,我没有收到任何错误。
解决方案
似乎错误地指定了库导入。根据文档,仅导入shc-core
就足够了,如下所示:
libraryDependencies += “com.hortonworks/shc-core:1.1.1-2.1-s_2.11”
另外,请注意,添加pomOnly()
到您的 build.sbt 将使处理程序跳过此依赖项的任何 jar 库或工件。添加它只会从聚合 pom 中获取元数据。
希望这可以帮助。
推荐阅读
- electron - 构建和移动后的 Electron JS 路径问题
- javascript - 如何在 iframe 中获取“href”属性?jQuery
- python-3.x - 根据路径的扩展名检索文件名
- javascript - 如何使用 puppeteer 在警报框中获取整个文本?
- python - 用python替换字符串,但前提是它尊重条件
- unit-testing - 如何在 Flutter 中模拟 VideoPlayerController.network()
- excel - 如何在不扭曲结果的情况下复制excel中的计算结果,然后将其从列转置为原始?
- r - 通过按 R 中的值过滤列来应用 glm()
- reactjs - 反应 - 通过特定数组映射取决于 url
- python - Tkinter 条目小部件验证