azure - 使用 azure databricks 将数据帧写入 blob

Question

是否有任何链接或示例代码，我们可以使用 python（不使用 pyspark 模块）将数据帧写入 azure blob 存储。

标签： azureazure-storageazure-blob-storageazure-databricks

Answer 1

下面是用于将（数据帧）CSV 数据直接写入 Azure Databricks Notebook 中的 Azure Blob 存储容器的代码片段。

# Configure blob storage account access key globally
spark.conf.set(
  "fs.azure.account.key.%s.blob.core.windows.net" % storage_name,
  sas_key)

output_container_path = "wasbs://%s@%s.blob.core.windows.net" % (output_container_name, storage_name)
output_blob_folder = "%s/wrangled_data_folder" % output_container_path

# write the dataframe as a single file to blob storage
(dataframe
 .coalesce(1)
 .write
 .mode("overwrite")
 .option("header", "true")
 .format("com.databricks.spark.csv")
 .save(output_blob_folder))

# Get the name of the wrangled-data CSV file that was just saved to Azure blob storage (it starts with 'part-')
files = dbutils.fs.ls(output_blob_folder)
output_file = [x for x in files if x.name.startswith("part-")]

# Move the wrangled-data CSV file from a sub-folder (wrangled_data_folder) to the root of the blob container
# While simultaneously changing the file name
dbutils.fs.mv(output_file[0].path, "%s/predict-transform-output.csv" % output_container_path)

示例：笔记本

输出：使用 Azure Databricks 写入 Blob 存储的数据帧