python - 为什么scrapy spider在抓取amazon时没有返回输出？
问题描述
我在亚马逊畅销书页面上测试了scrapy spider（见下面的URL），但它返回奇怪的价格数字或根本没有输出，正如你在最后的输出中看到的那样（我只分享了1页的输出） . css 选择器可能有问题，但我不确定。我希望蜘蛛将输出保存在 JSON 文件中，以便我可以快速将其转换为 pandas 数据框以进行一些分析。这是我在终端中编写的运行蜘蛛的代码：scrapy crawl amazon_booksUK -o somefilename.json
我知道这需要仔细阅读，但如果你有时间，它真的会帮助我！:)
1.蜘蛛代码：
import scrapy
from ..items import AmazonscrapeItem
class AmazonSpiderSpider(scrapy.Spider):
page_number = 2
name = 'amazon_booksUK'
start_urls = [
'https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_1'
]
def parse(self, response):
items = AmazonscrapeItem()
# if multiple classes --> .css("::text").extract()
product_name = response.css('.a-color-base.a-text-normal::text').extract()
product_author = response.css('.a-color-secondary .a-size-base.a-link-normal').css('::text').extract()
product_nbr_reviews = response.css('.a-size-small .a-link-normal .a-size-base').css('::text').extract()
product_type = response.css('.a-spacing-top-small .a-link-normal.a-text-bold').css('::text').extract()
product_price = response.css('.a-spacing-top-small .a-price-whole').css('::text').extract()
product_more_choice = response.css('.a-spacing-top-mini .a-color-secondary .a-link-normal').css('::text').extract()
# this only selects the element that has the image --> need stuff inside src (source attr)
product_imagelink = response.css('.s-image::attr(src)').extract() # want attr of src
items['product_name'] = product_name
items['product_author'] = product_author
items['product_nbr_reviews'] = product_nbr_reviews
items['product_type'] = product_type
items['product_price'] = product_price
items['product_more_choice'] = product_more_choice
items['product_imagelink'] = product_imagelink
yield items
next_page = 'https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page='+ str(AmazonSpiderSpider.page_number)+'&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_'+ str(AmazonSpiderSpider.page_number)
if AmazonSpiderSpider.page_number <3:
AmazonSpiderSpider.page_number += 1
yield response.follow(next_page, callback=self.parse)
2.项目.py：
import scrapy
class AmazonscrapeItem(scrapy.Item):
# define the fields for your item here like:
product_name = scrapy.Field()
product_author = scrapy.Field()
product_nbr_reviews = scrapy.Field()
product_type = scrapy.Field()
product_price = scrapy.Field()
product_more_choice = scrapy.Field()
product_imagelink = scrapy.Field()
3.settings.py： 为了避免在测试爬虫时被封禁，我使用了google bot用户代理。
USER_AGENT = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
4.管道.py：
class AmazonscrapePipeline:
def process_item(self, item, spider):
return item
输出
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
(venv) (base) Eriks-MBP:scrapyTutorial erikhren$ cd amazonscrape
(venv) (base) Eriks-MBP:amazonscrape erikhren$ scrapy crawl amazon_booksUK -o mm.json
2020-09-01 15:01:43 [scrapy.utils.log] INFO: Scrapy 2.3.0 started (bot: amazonscrape)
2020-09-01 15:01:43 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.3 (default, Mar 6 2020, 22:34:30) - [Clang 11.0.3 (clang-1103.0.32.29)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g 21 Apr 2020), cryptography 3.1, Platform Darwin-19.6.0-x86_64-i386-64bit
2020-09-01 15:01:43 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-09-01 15:01:43 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'amazonscrape',
'NEWSPIDER_MODULE': 'amazonscrape.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['amazonscrape.spiders'],
'USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; '
'+http://www.google.com/bot.html)'}
2020-09-01 15:01:43 [scrapy.extensions.telnet] INFO: Telnet Password: e57aa4639df179b5
2020-09-01 15:01:43 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2020-09-01 15:01:44 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-09-01 15:01:44 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-09-01 15:01:44 [scrapy.middleware] INFO: Enabled item pipelines:
['amazonscrape.pipelines.AmazonscrapePipeline']
2020-09-01 15:01:44 [scrapy.core.engine] INFO: Spider opened
2020-09-01 15:01:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-09-01 15:01:44 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-09-01 15:01:44 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.co.uk/robots.txt> (referer: None)
2020-09-01 15:01:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.amazon.co.uk/b?ie=UTF8&node=266239> from <GET https://www.amazon.co.uk/s?i=stripbooks&bbn=266239&rh=n%3A266239%2Cp_72%3A184315031%2Cp_36%3A389028011&dc&page=1&fst=as%3Aoff&qid=1598942460&rnid=389022011&ref=sr_pg_1>
2020-09-01 15:01:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.amazon.co.uk/books-used-books-textbooks/b?ie=UTF8&node=266239> from <GET https://www.amazon.co.uk/b?ie=UTF8&node=266239>
2020-09-01 15:01:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.co.uk/books-used-books-textbooks/b?ie=UTF8&node=266239> (referer: None)
2020-09-01 15:01:45 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.amazon.co.uk/books-used-books-textbooks/b?ie=UTF8&node=266239>
{'product_author': [],
'product_imagelink': [],
'product_more_choice': [],
'product_name': [],
'product_nbr_reviews': [],
'product_price': ['11',
'.',
'12',
'.',
'9',
'.',
'6',
'.',
'9',
'.',
'9',
'.',
'6',
'.',
'8',
'.',
'30',
'.',
'13',
'.',
'11',
'.',
'10',
'.',
'7',
'.',
'6',
'.',
'24',
'.',
'39',
'.',
'20',
'.',
'34',
'.',
'6',
'.',
'48',
'.',
'52',
'.',
'47',
'.',
'14',
'.',
'20',
'.',
'17',
'.',
'14',
'.',
'71',
'.',
'34',
'.',
'34',
'.',
'44',
'.',
'44',
'.',
'89',
'.',
'40',
'.',
'31',
'.',
'39',
'.',
'12',
'.',
'24',
'.',
'66',
'.',
'68',
'.',
'17',
'.',
'289',
'.',
'14',
'.',
'20',
'.',
'44',
'.',
'26',
'.',
'13',
'.',
'100',
'.',
'60',
'.',
'20',
'.',
'107',
'.',
'222',
'.',
'21',
'.',
'39',
'.',
'20',
'.',
'60',
'.',
'61',
'.',
'60',
'.',
'22',
'.',
'40',
'.',
'18',
'.'],
'product_type': []}
解决方案
你只需要像这样使用用户代理：
USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'
像真正的浏览器使用的东西
推荐阅读
- vba - Excel 宏需要多次执行才能正常运行
- flutter - 颤振网；Riverpod Firebase 身份验证；热重启/重新加载 - 用户会话丢失？
- python - 如果找不到值，则在 Panda 中插入行
- delphi - Steema VCL - 试图旋转 TPolarSeries.Pointer.Picture
- docker - Docker默认fs大小？
- ios - 在swiftui中保存照片？
- java - 从 SchedulerExecutorService 抛出异常
- twitter-bootstrap - 默认情况下，引导模式在 HTML 中可见
- python - 如何覆盖python对象中的方法？
- rust - Solana Anchor 如何使用结构数组定义自定义类型？