python - 下载文件时如何在每个块之后提交/写入数据，以便它不会使用所有内存？

Question

当我下载文件时，我可以看到内存正在被用于存储。有没有办法在下载每块 1GB 数据后提交数据？

submission_path = r'D:\Users\Jonathan\Desktop\Reddit Data\ETL-Python\\'
for download_file in clean_matching_list_of_href:
    save_file_w_submission_path = submission_path + download_file
    constructured_url = url_to_download + download_file
    request = urllib.request.Request(constructured_url)
    response = urllib.request.urlopen(request)
    data_content = response.read()
    shutil.copyfileobj(save_file_w_submission_path,data_content,length = 10000)

更新： 所以我改用了 write，如下所示：

for download_file in matching_list_of_href:
    filename = download_file[download_file.rfind("/")+1:]
    save_file_w_submission_path = path_to_save_document + filename
    request = urllib.request.Request(download_file)
    response = urllib.request.urlopen(request)
    data_content = response.read()
    with open(save_file_w_submission_path, 'wb') as wf:    
        wf.write(data_content)
    print(save_file_w_submission_path)

标签： pythonwindows