scrapy - 如何使用 requests 或 scrapy 从 opensubtitle.org 下载 zip 文件

Question

我有 zip 文件下载网址，链接如下： https ://dl.opensubtitles.org/en/download/sub/7790586/vrf-f5760bc3 。我使用所需的标头请求此 URL，但仍然无法下载 zip 文件。它被重定向到旧电影页面，否则下载不相关的 zip 文件内容。我知道有可用的 API，但我需要通过脚本来完成，而不是通过 API 或 selenium。

我尝试了 request 和 scrapy get 方法仍然无法下载正确的 zip 文件。

headers = { 
"authority": "dl.opensubtitles.org","Connection": "keep-alive", 
"user-agent":user_agent, 
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding": "gzip,deflate,br","accept-language": "en-IN,en-US;q=0.9,en;q=0.8","referer": movie_url,'upgrade-insecure-requests':'1'}

requests.get(url,headers=headers)

标签： scrapypython-requestspython-3.6

Answer 1

with requests.get(url, stream=True) as res:
    with open('test.zip', 'wb') as f:
        for chunk in res.iter_content(chunk_size=1024): #you can also change the chunk size
             if chunk: # filter out 
                 f.write(chunk)