python - 在 Dask 中使用 read_csv 进行列名移位
问题描述
我正在尝试使用Intake对 csv 数据集进行编目。它使用 Dask 实现,read_csv
而后者又使用 pandas 实现。
我看到的问题是我正在加载的 csv 文件没有索引列,因此 Dask 将第一列解释为索引,然后将列名向右移动。
日期时间 (dt) 列应该是第一列,但是当读取 csv 时,它被解释为索引并且列名被移动,因此从它们的正确位置偏移。我将列names
列表和dtypes
字典提供给 read_csv 调用。
据我所知,如果我使用的是 pandas,我会提供index_col=False
kwarg 来修复,如图所示,但 Dask 返回一个故意错误,说明:Keywords 'index' and 'index_col' not supported. Use dd.read_csv(...).set_index('my-index') instead
. 这似乎是由于并行化限制。
建议的修复(使用set_index('my-index)
)在这种情况下无效,因为它希望读取整个文件,同时还具有列名来设置索引。主要问题是,如果名称偏移,我无法准确设置索引列。
在 Dask 中,加载没有明确具有索引列的 csv 以使解释的索引列至少保留指定的列名的最佳方法是什么?
更多信息:
我正在使用的游戏数据集:https ://www.kaggle.com/NUFORC/ufo-sightings?select=scrubbed.csv
我使用的 Intake catalog.yml 文件如下:
name:
intake-explore-catalog
metadata:
version: 1
sources:
ufo_sightings:
description: data around ufo sightings
driver: csv
args:
urlpath: "{{CATALOG_DIR}}/data/ufo_scrubbed.csv"
csv_kwargs:
header: 0
names: ['dt', 'city', 'state', 'country', 'shape', 'duration_s', 'duration_hm', 'comments', 'date_posted', 'latitude']
dtype: {'dt': 'str', 'city': 'str', 'state': 'str', 'country': 'str', 'shape': 'str', 'duration_s': 'str', 'duration_hm': 'str', 'comments': 'str', 'date_posted': 'str', 'latitude': 'str'}
infer_datetime_format: true
metadata:
version: 1
custom_field: blah
我正在使用以下内容加载目录和相应的数据集:
cat = intake.open_catalog("catalog.yml")
ufo_ds = cat.ufo_sightings.read()
这将导致读取上面显示的数据帧和该数据的 csv 副本:
,dt,city,state,country,shape,duration_s,duration_hm,comments,date_posted,latitude
10/10/1949 20:30,san marcos,tx,us,cylinder,2700,45 minutes,This event took place in early fall around 1949-50. It occurred after a Boy Scout meeting in the Baptist Church. The Baptist Church sit,4/27/2004,29.8830556,-97.9411111
10/10/1949 21:00,lackland afb,tx,,light,7200,1-2 hrs,1949 Lackland AFB, TX. Lights racing across the sky & making 90 degree turns on a dime.,12/16/2005,29.38421,-98.581082
10/10/1955 17:00,chester (uk/england),,gb,circle,20,20 seconds,Green/Orange circular disc over Chester, England,1/21/2008,53.2,-2.916667
10/10/1956 21:00,edna,tx,us,circle,20,1/2 hour,My older brother and twin sister were leaving the only Edna theater at about 9 PM,...we had our bikes and I took a different route home,1/17/2004,28.9783333,-96.6458333
10/10/1960 20:00,kaneohe,hi,us,light,900,15 minutes,AS a Marine 1st Lt. flying an FJ4B fighter/attack aircraft on a solo night exercise, I was at 50ꯠ' in a "clean" aircraft (no ordinan,1/22/2004,21.4180556,-157.8036111
与原始/原始数据 csv(无前导逗号)相比:
datetime,city,state,country,shape,duration (seconds),duration (hours/min),comments,date posted,latitude,longitude
10/10/1949 20:30,san marcos,tx,us,cylinder,2700,45 minutes,"This event took place in early fall around 1949-50. It occurred after a Boy Scout meeting in the Baptist Church. The Baptist Church sit",4/27/2004,29.8830556,-97.9411111
10/10/1949 21:00,lackland afb,tx,,light,7200,1-2 hrs,"1949 Lackland AFB, TX. Lights racing across the sky & making 90 degree turns on a dime.",12/16/2005,29.38421,-98.581082
10/10/1955 17:00,chester (uk/england),,gb,circle,20,20 seconds,"Green/Orange circular disc over Chester, England",1/21/2008,53.2,-2.916667
10/10/1956 21:00,edna,tx,us,circle,20,1/2 hour,"My older brother and twin sister were leaving the only Edna theater at about 9 PM,...we had our bikes and I took a different route home",1/17/2004,28.9783333,-96.6458333
10/10/1960 20:00,kaneohe,hi,us,light,900,15 minutes,"AS a Marine 1st Lt. flying an FJ4B fighter/attack aircraft on a solo night exercise, I was at 50ꯠ' in a "clean" aircraft (no ordinan",1/22/2004,21.4180556,-157.8036111
10/10/1961 19:00,bristol,tn,us,sphere,300,5 minutes,"My father is now 89 my brother 52 the girl with us now 51 myself 49 and the other fellow which worked with my father if he's still livi",4/27/2007,36.5950000,-82.1888889
Dask 调用:
df = dask.dataframe.read_csv('data/ufo_scrubbed.csv',
names=['dt',
'city',
'state',
'country',
'shape',
'duration_s',
'duration_hm',
'comments',
'date_posted',
'latitude'],
dtype = {'dt': 'str',
'city': 'str',
'state': 'str',
'country': 'str',
'shape': 'str',
'duration_s': 'str',
'duration_hm': 'str',
'comments': 'str',
'date_posted': 'str',
'latitude': 'str'}
)
解决方案
不幸的是,标题行以逗号开头,这就是为什么您的列名会减一的原因。你最好解决这个问题,而不是解决它。
但是,如果您不提供列名,则不会自动获得索引:
df = dask.dataframe.read_csv('file.csv', header=0)
这里的索引只是一个范围(从每个分区中的 0 开始计数)。然后,您可以在事后分配列名
df2 = df.rename(columns=dict(zip(df.columns, df.columns[1:]), latitude='longitude'))
您无法仅通过 Intake 处方来实现此目的,您必须通过 to_dask() 或 read() 获取数据帧(分别用于 dask 或 pandas 输出)。
推荐阅读
- glib - 我们是否应该总是使用 g_object_ref_sink 而不是 g_object_ref?
- java - 在 C# 中使用与在 Java 中导入
- angular - Jasmine 单元测试,仅覆盖服务中的属性
- pandas - 在堆叠的 Seaborn 条形图中选择适当的颜色
- css - CSS 不适用于字体真棒图标下的 SVG 元素
- discord - 在触发 DiscordJS 之前获取上一条消息
- excel - 在 VBA 中为操作系统创建 IF 语句
- python - 发生异常:TypeError rmdir:路径应该是字符串、字节或 os.PathLike,而不是 TemporaryDirectory
- multithreading - 相同版本但具有不同配置选项的两个 perls 之间的兼容性?
- reactjs - 打字稿声明合并原因是指一个值,但在反应中被用作类型