时间戳

首页 > 解决方案 > 如何通过 scrapy Request 发送 cookie 数据？

python - 如何通过 scrapy Request 发送 cookie 数据？

问题描述

我一直在尝试将标头中的 cookie 发送到这个 url并抓取 json 响应。要发送的cookie是cookie："tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"

在使用 python 的请求库时，我得到了所需的 json 数据。这是我的 Python 代码。

import requests
url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
headers = {'cookie':cookie}
requests.get(url, headers=headers).json()

这是我的蜘蛛代码：

def parse(self, response):
        cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
        url = "https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
        request = scrapy.Request(url=url, cookies ={'cookie': cookie},callback=self.parsejson)
        yield request

    def parsejson(self, response):
        jsonresponse = json.loads(response.text)
        print(jsonresponse)

但是蜘蛛给出了这个错误。

2019-07-16 15:59:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://amazon.zappos.com/c/404> from <GET https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3>
2019-07-16 15:59:59 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://amazon.zappos.com/c/404>
Cookie: cookie=tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg

2019-07-16 16:00:00 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3)
2019-07-16 16:00:00 [scrapy.core.scraper] ERROR: Spider error processing <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3)
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/mac/scrapy/projects/zappos/zappos/spiders/zappo.py", line 18, in parsejson
    jsonresponse = json.loads(response.text)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2019-07-16 16:00:00 [scrapy.core.engine] INFO: Closing spider (finished)

标签： pythoncookiesscrapypython-requests

解决方案

您需要使用key: value格式cookies：

def parse(self, response):
        # cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
        url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
        request = scrapy.Request(url=url, cookies ={'tid': "fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg"},callback=self.parsejson)
        yield request

更新看起来你在url（mobileapi f vs mobileapi）中有错字。检查更新的代码。

推荐阅读