时间戳

首页 > 解决方案 > 从 Presto DB 连接 Azure 数据湖存储：方案没有文件系统：adl

azure - 从 Presto DB 连接 Azure 数据湖存储：方案没有文件系统：adl

问题描述

我正在尝试在本地计算机上运行 Presto DB 以连接来自 Azure Datalake 存储的数据。即使为 Azure Data Lake Store 添加了 JAR 文件，也无法从 Azure DataLake Store 获取数据。

我收到以下错误：

Query 20181005_191247_00000_wcgur failed: No FileSystem for scheme: adl

    com.facebook.presto.spi.PrestoException: No FileSystem for scheme: adl
    at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:189)
    at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
    at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
    at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
    at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: No FileSystem for scheme: adl
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.PrestoFileSystemCache.createFileSystem(PrestoFileSystemCache.java:114)
    at org.apache.hadoop.fs.PrestoFileSystemCache.getInternal(PrestoFileSystemCache.java:89)
    at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:62)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at com.facebook.presto.hive.HdfsEnvironment.lambda$getFileSystem$0(HdfsEnvironment.java:71)
    at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
    at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:70)
    at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:64)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:282)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:256)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:91)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:185)
    ... 7 more

参考：

Persto 服务器：presto-server-0.212.tar.gz Presto CLI：presto-cli-0.212-executable.jar

添加到 plugin/hive-hadoop2 的 JAR：hadoop-azure-datalake-3.1.1.jar azure-data-lake-store-sdk-2.3.2.jar hadoop-azure-3.1.1.jar

hive.properties

connector.name=hive-hadoop2
hive.metastore.uri=thrift://hive:9083
hive.config.resources=presto/server/etc/catalog/adls-site.xml

adls-site.xml

<configuration>
        <property>
                <name>fs.adl.impl</name>
                <value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
        </property>

        <property>
                <name>fs.AbstractFileSystem.adl.impl</name>
                <value>org.apache.hadoop.fs.adl.Adl</value>
        </property>
        <property>
        <name>fs.adl.oauth2.access.token.provider.type</name>
        <value>ClientCredential</value>
        </property>

        <property>
        <name>fs.adl.oauth2.refresh.url</name>
        <value>my_url</value>
        </property>

        <property>
        <name>fs.adl.oauth2.client.id</name>
        <value>my_id</value>
        </property>

        <property>
        <name>fs.adl.oauth2.credential</name>
        <value>my_cred</value>
        </property>

  </configuration>

对此的任何评论都会很有帮助。提前致谢！

标签： azurejarazure-data-lakepresto

解决方案

推荐阅读