python - 将 numpy recarray 转换为 pyarrow.Table

Question

我想将 numpy recarray 转换为 pyarrow.Table。有推荐的方法吗？

通过 pandas DataFrame 进行转换是最简单的：

ra = ... # some recarray
T1 = pa.Table.from_pandas(pd.DataFrame(ra))

但似乎它应该增加不必要的开销。我已经尝试过from_pydict，它似乎工作，虽然有点hacky：

ra = ... # some recarray
T2 = pa.Table.from_pydict({k:ra[k] for k in ra.dtype.fields.keys()})

如果我尝试在一个有点现实的例子中对这两个时间进行计时，则该from_pydict方法要快得多：

c:\>python
Python 3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> import pyarrow as pa
>>> import numpy as np
>>>
>>> np.random.seed(123)
>>> rectype = np.dtype([('timestamp', np.int64),
...                     ('category', np.int32),
...                    ])
>>> nrows = 20000
>>> rawvals = np.random.randint(1000,size=(nrows,2))
>>> ra = np.array([tuple(row) for row in rawvals], dtype=rectype)
>>> T1 = pa.Table.from_pandas(pd.DataFrame(ra))
>>> T2 = pa.Table.from_pydict({k:ra[k] for k in ra.dtype.fields.keys()})
>>> (T1.to_pandas() == T2.to_pandas()).all()
timestamp    True
category     True
dtype: bool
>>>
>>> import timeit
>>> def f1():
...     return pa.Table.from_pandas(pd.DataFrame(ra))
...
>>> def f2():
...     return pa.Table.from_pydict({k:ra[k] for k in ra.dtype.fields.keys()})
...
>>> timeit.timeit(f1,number=1000)
1.4761637000000007
>>> timeit.timeit(f2,number=1000)
0.05712700000000126

from_pydict在这种情况下是更好的方法吗？它有什么缺点吗？

标签： pythonnumpypyarrow