python - pd.read_csv 一个 750 MB 的文本文件

Question

我使用 读取了一个 750 MB 的文本文件，文本pd.read_csv中前 5 行的字段少于其余行。我所做的是：

ICP_CTCT_TRADER_MEP=pd.read_csv('EDA_CTCT_TRADER_MEP.txt', sep=',', header=None, skiprows=5, encoding='ISO-8859-1',engine='python')

前 5 行：

0   1   2   3   4   5   6   7   8   9   ... 12  13  14  15  16  17  18  19  20  21
DET 0000000001CP630 METERSUMMARY    MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN NGCM    1   ... N   Y   2   N   NaN None    None    None    None    None
METERINSTALL    MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   1   ... HHR F   21/10/2011  21/10/2026  N   X   03/05/2023  N145361 90  NaN
METERCOMP   MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   211313791   ... Y   1   1   NGCS    NaN None    None    None    None    None
METERCHANNEL    MET-10748517    17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   211313791   ... UN  24  kWh X   C   Y   NaN None    None    None
17/08/2013  20/08/2013 17:11:28 NGCM    NGCM20130817MM010100000.txt Active  NaN NaN NaN NaN 1   211313791   ... CN  17  kWh X   C   Y   NaN None    None    None

自动标题来自第六列，前 5 列没有标题。我的解决方案是将其导出到 .csv 并将其导入回来以获取每个列的正确标题，如下所示

ICP_CTCT_TRADER_MEP=ICP_CTCT_TRADER_MEP.dropna(thresh=200000, axis='columns') # to remove the NaN Columns
ICP_CTCT_TRADER_MEP.to_csv('ICP_CTCT_TRADER_MEP.csv') # to export it to a .csv file
ICP_CTCT_TRADER_MEP_1=pd.read_csv('ICP_CTCT_TRADER_MEP.csv') #to import it back again

我的问题是：有更好的方法吗？我上面的方法需要很长时间，并且有时无法正常工作，并因内存错误而失败。

标签： pythonpandascsvdataframe