时间戳

首页 > 解决方案 > 如何使用 dask libray 将镶木地板表（pyarrow）附加到现有的输出镶木地板文件？

pandas - 如何使用 dask libray 将镶木地板表（pyarrow）附加到现有的输出镶木地板文件？

问题描述

我正在尝试实现将数据存储到 parquet-structure 的功能，将其输出到 patquet-file，如果存在输出 parquet-file - 使用 dask 附加 parquet-structure

self.flag = True
    self.columns = ['original_ids', 'masked_ids', 'masked_labels', 'masked_positions']
    fields = [
          pa.field('original_ids',pa.list_(pa.int32())),
          pa.field('masked_ids',pa.list_(pa.int32())),
          pa.field('masked_labels', pa.list_(pa.int32())),
          pa.field('masked_positions', pa.list_(pa.int32())),
        ]
    
    self.myschema = pa.schema(fields)
    
    
    
    df_input = pd.DataFrame({'original_ids' : [original_ids], 'masked_ids' : [masked_ids],
                                               'masked_labels' : [masked_lm_ids],                              'masked_positions': [masked_lm_positions]})
    df_input = dd.from_pandas(df_input, npartitions=1)
    if self.flag:
       dd.to_parquet(df_input, self.output_f, engine='pyarrow', compression='gzip', write_index=False,
                                compute=True, append=False, ignore_divisions=True, schema=self.myschema)
       self.flag = False
    else:
       dd.to_parquet(df_input, self.output_f, engine='pyarrow', compression='gzip', write_index=False,
                                compute=True, append=True,  ignore_divisions=True, schema=self.myschema)

但是我得到了下一个错误：

  File "/usr/local/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py", line 424, in to_parquet
    **kwargs_pass
  File "/usr/local/lib/python3.7/site-packages/dask/dataframe/io/parquet/arrow.py", line 583, in initialize_write
    "Previous: {} | New: {}".format(names, list(df.columns))
ValueError: Appended columns not the same.
Previous: ['item', 'item', 'item', 'item'] | New: ['original_ids', 'masked_ids', 'masked_labels', 'masked_positions']

如何更改现有代码以修复此错误？

标签： pandasdataframedaskparquetpyarrow

解决方案

推荐阅读