时间戳

首页 > 解决方案 > webHDFS curl --negotiate 在 Windows 上

windows - webHDFS curl --negotiate 在 Windows 上

问题描述

以下命令适用于 Linux，但在 Windows 上失败。在运行命令之前，我使用 kinit 获取有效的 Kerberos 票证。

curl -v -i --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt "http://<HDP-Cluster>:50070/webhdfs/v1/?op=LISTSTATUS"

预期结果（Linux 输出）：

> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":17,"fileId":18894,"group":"hadoop","length":0,"modificationTime":1589364305719,"owner":"yarn","pathSuffix":"app-logs","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
> {"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16432,"group":"hdfs","length":0,"modificationTime":1566323580684,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
> {"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16419,"group":"hdfs","length":0,"modificationTime":1566323394432,"owner":"hdfs","pathSuffix":"atsv2","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":77127,"group":"hdfs","length":0,"modificationTime":1566983700653,"owner":"hdfs","pathSuffix":"benchmarks","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}

从 Windows 输出：

* Connected to <SERVER> (<IP>) port 50070 (#0)
* Server auth using Negotiate with user ''
> GET /webhdfs/v1/?op=LISTSTATUS HTTP/1.1
> Host: <SERVER>:50070
> Authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAGNFAAAADw==
> User-Agent: curl/7.70.0
> Accept: */*
> Cookie: hadoop.auth=
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 java.lang.IllegalArgumentException
< Date: Wed, 15 Jul 2020 09:32:34 GMT
< Date: Wed, 15 Jul 2020 09:32:34 GMT
< Pragma: no-cache
< X-FRAME-OPTIONS: SAMEORIGIN
* Replaced cookie hadoop.auth="" for domain <SERVER.Full.Domain>, path /, expire 0
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 293
<
{ [293 bytes data]
100   293  100   293    0     0   3367      0 --:--:-- --:--:-- --:--:--  3367HTTP/1.1 403 java.lang.IllegalArgumentException
Date: Wed, 15 Jul 2020 09:32:34 GMT
Date: Wed, 15 Jul 2020 09:32:34 GMT
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 293

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 403 java.lang.IllegalArgumentException</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/. Reason:
<pre>    java.lang.IllegalArgumentException</pre></p>
</body>
</html>

* Connection #0 to host <server> left intact

curl 版本窗口：

curl -V
curl 7.70.0 (x86_64-w64-mingw32) libcurl/7.70.0 OpenSSL/1.1.1g (Schannel) zlib/1.2.11 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP

卷曲版本 Linux：

curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

我在当使用 --negotiate with curl 时，找到了关于如何在 Linux 上安装的详细说明，是否需要 keytab 文件？，但我找不到适用于 Windows 的 GSS-API，适用于 Windows 的 SSPI 是否与适用于 Linux 的 GSS-API 相同？

标签： windowshadoopcurlwebhdfsnegotiate

解决方案

通过以下两个方面我找到了答案：
当使用 --negotiate 和 curl 时，是否需要 keytab 文件？

警告：Windows 版本的 curl 被编译为支持 Microsoft SSPI 库，而不是 Hadoop REST 服务所需的 GSSAPI 库

:(

在没有 SPnego 的情况下访问 kerberos 保护的 WebHDFS

显示了一个 Java 代码，您在其中获得了一个委托令牌并将其传递给 WebHDFS RestAPI。这样你就不需要使用--negotiate。

推荐阅读