python - PySpark - 如何在 csv 输出中删除科学记数法
问题描述
我有一个火花聚合,我想将结果输出到 csv,但我发现火花总是以科学计数法输出大量小数。我已经尝试过这个问题中提到的解决方案,但这也没有奏效。
预期输出:
foo,avg(bar)
a,0.0000002
b,0.0000001
实际输出:
foo,avg(bar)
a,2.0E-7
b,1.0E-7
请参阅下面的示例:
from os import path
import shutil
import glob
from pyspark.sql import SQLContext, functions as F, types
def test(sc):
sq = SQLContext(sc)
data = [("a", 1e-7), ("b", 1e-7), ("a", 3e-7)]
df = sq.createDataFrame(data, ['foo', 'bar'])
# 12 digits with 9 decimal places
decType = types.DecimalType(precision=12, scale=9)
# Cast both the column input and column output to Decimal
aggs = [F.mean(F.col("bar").cast(decType)).cast(decType)]
groups = [F.col("foo")]
result = df.groupBy(*groups).agg(*aggs)
write(result)
return df, aggs, groups, result
def write(result):
tmpDir = path.join("res", "tmp")
config = {"sep": ","}
result.write.format("csv")\
.options(**config)\
.save(tmpDir)
# Once the distributed portion is done, write out to a single a file
allFiles = glob.glob(path.join(tmpDir,"*.csv"))
fullOut = path.join("res", "final.csv")
with open(fullOut, 'wb') as wfd:
# First write out the header row
header = config.get("sep", ',').join(result.columns)
wfd.write(header + "\n")
for f in allFiles:
with open(f, 'rb') as fd:
shutil.copyfileobj(fd, wfd)
pass
pass
shutil.rmtree(tmpDir)
return
在 pyspark 外壳中:
import spark_test as t
t.test(sc)
解决方案
>>> df1 = spark.createDataFrame([('a','2.0e-7'),('b','1e-5'),('c','1.0e-7')],['foo','avg'])
>>> df1.show()
+---+------+
|foo| avg|
+---+------+
| a|2.0e-7|
| b| 1e-5|
| c|1.0e-7|
+---+------+
>>> df1.select('foo','avg',format_string('%.7f',df1.avg.cast('float')).alias('converted')).show()
+---+------+---------+
|foo| avg|converted|
+---+------+---------+
| a|2.0e-7|0.0000002|
| b| 1e-5|0.0000100|
| c|1.0e-7|0.0000001|
+---+------+---------+
推荐阅读
- django - 修改基于类的视图对象保存
- sql-server - 为什么我的 WHERE 子句不能将字段识别为日期,即使在将此字段从 int 转换为日期之后?
- python - python sklearn 特征选择mutual_info_regression
- c++ - Sublime Text 3 编译 C++ 程序的问题
- blazor - Blazor 和浏览器页面刷新
- reactjs - 为胜利图表栏添加自定义样式
- terraform - 无法使用 Terraform destroy 销毁一个模块
- java - 没有 volatile 线程无法工作,并从 RAM 中读取值而不是缓存
- javascript - 在javascript中生成动态id
- python - 使用 Python Selenium 选择下拉值