时间戳

首页 > 解决方案 > Why do I get scrapy response empty?

xpath - Why do I get scrapy response empty?

问题描述

I started 

scrapy shell -s USER_AGENT='Mozilla/5.0' https://www.gumtree.com/p/property-to-rent/brand-new-modern-studio-flat-%C2%A31056pcm-all-bills-included-in-willesden-green-area/1303463798

Next step

In [5]: response                                                                                                                                                                                            
Out[5]: <405 https://www.gumtree.com/p/property-to-rent/brand-new-modern-studio-flat-%C2%A31056pcm-all-bills-included-in-willesden-green-area/1303463798>

After inspected page element,and copied XPath

In [6]: response.xpath('//*[@id="ad-title"]').extract()                                                                                                                                                     
Out[6]: []

Copy outerHTML

<h1 itemprop="name" id="ad-title">Brand New Modern Studio Flat £1056pcm | All Bills Included | In Willesden Green area</h1>

Image view response enter image description here

Why?

标签： xpathscrapy

解决方案

尝试将用户代理设置为更真实的东西，例如：Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0.

一些网站在用户代理上做一些基本的验证，如果他们检测到一些奇怪的东西，就会把你重定向到一些特殊的页面。

scrapy shell -s USER_AGENT='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0' https://www.gumtree.com/p/property-to-rent/brand-new-modern-studio-flat-%C2%A31056pcm-all-bills-included-in-willesden-green-area/1303463798
>>> response.xpath('//*[@id="ad-title"]').extract()
['<h1 itemprop="name" id="ad-title">Brand New Modern Studio Flat £1056pcm | All Bills Included | In Willesden Green area</h1>']
>>>

推荐阅读