hadoop - osx 获取 pyarrow.lib.ArrowIOError：无法加载 libhdfs

Question

import pyarrow as pa

client = pa.hdfs.connect('localhost', 9000)

错误

Traceback (most recent call last):
  File "/Users/wyx/project/py3.7aio/hdfs/list_dir.py", line 13, in <module>
    client = pa.hdfs.connect('localhost', 9000)
  File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 207, in connect
    extra_conf=extra_conf)
  File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 38, in __init__
    self._connect(host, port, user, kerb_ticket, driver, extra_conf)
  File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
  File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unable to load libhdfs

我通过安装 hadoopbrew并获取任何本机库，因此我通过本机库指南构建 hadoop3.1.1，但我无法获取 pyarrow 需要的任何libhdfs.so我只获取libhdfs.dylib

➜  native git:(branch-3.1.1) ✗ hadoop checknative -a
2019-02-24 22:05:31,686 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
2019-02-24 22:05:31,689 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2019-02-24 22:05:31,695 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Native library checking:
hadoop:  true /usr/local/Cellar/hadoop/3.1.1/libexec/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
zstd  :  false
snappy:  true /usr/local/lib/libsnappy.1.dylib
lz4:     true revision:10301
bzip2:   false
openssl: false build does not support openssl.
ISA-L:   false libhadoop was built without ISA-L support
2019-02-24 22:05:31,723 INFO util.ExitUtil: Exiting with status 1: ExitException

标签： hadooppyarrow