python - scrapy - 抓取多个字段
问题描述
非常新的和缺乏经验的程序员在这里！
我正在构建一个scrapy 项目，它可以从这个网站上抓取公司名称和位置，并输出一个JSON 文件。
https://www.f6s.com/programs?type[]=accelerator&page=93&page_alt=1&sort=open
目前，我的爬虫正在提取公司名称，但也在提取日期。此外，JSON 输出分几个部分出现，首先是公司名称列表，然后是位置列表（包含我不想要的其他信息）。
如何提取公司名称/位置并对其进行格式化，以便将每个公司名称与特定位置相关联？
我认为我的问题是这些位置没有定义为特定的类。
此外，非常感谢有关如何设置 JSON 输出格式的建议！
我的项目目录：
`myproject`/
scrapy.cfg
__init__.py
items.py
pipelines.py
settings.py
spiders/
__init__.py
byub.py
F6sSpider.py
我的蜘蛛文件：
import scrapy
class CompanySpider(scrapy.Spider):
name = "Company"
def start_requests(self):
urls = [
'https://www.f6s.com/programs?type[]=accelerator&sort=open',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
for title in response.css('div.result-description'):
yield {
'program': title.css('div.title a.action.main.noline::text').extract(),
}
for subtitle in response.css('div.result-description'):
yield {
'location': subtitle.css('div.subtitle span::text').extract(),
}
我的终端
2018-05-16 13:10:30 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: myproject)
2018-05-16 13:10:30 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 3.6.5 (default, Apr 25 2018, 14:23:58) - [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Darwin-17.5.0-x86_64-i386-64bit
2018-05-16 13:10:30 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'myproject', 'FEED_FORMAT': 'json', 'FEED_URI': 'companies.json', 'NEWSPIDER_MODULE': 'myproject.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myproject.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 5.2; rv:10.0.1) Gecko/20100101 Firefox/10.0.1 SeaMonkey/2.7.1'}
2018-05-16 13:10:30 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2018-05-16 13:10:30 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-05-16 13:10:30 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-05-16 13:10:30 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-05-16 13:10:30 [scrapy.core.engine] INFO: Spider opened
2018-05-16 13:10:30 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-05-16 13:10:30 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-05-16 13:10:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.f6s.com/robots.txt> (referer: None)
2018-05-16 13:10:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.f6s.com/programs?type[]=accelerator&sort=open> (referer: None)
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['K - LAUNCHPAD 2018']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['Z Nation Lab Real Estate Cohort']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['C-mint-International']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['StartOut Growth Lab - 2018 Fall Cohort']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['IBA Application']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['WATT Factory Accelerator Programme 2018']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ["AdvantEdge Founder's Adda"]}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['SpinLab - The HHL Accelerator']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['Shell LiveWIRE Accelerator']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['Shell France Accelerator ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['ELEVATE by TheVentury']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'program': ['F6S R&D Money Back']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Jun 1-Jul 20 •\n Berlin, Germany \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Mumbai, India \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Atlanta, United States \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Jul 8-Dec 31 •\n San Francisco, United States \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Mar 19-May 16 •\n Los Angeles, United States \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Jun 3-Nov 30 •\n Gent, Belgium \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Delhi, India \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n Leipzig, Germany \n ']}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ["\n Jun 20 '18-Jun 21 '19 •\n Paris, France \n "]}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ["\n Jun 20 '18-Jun 1 '19 •\n Paris, France \n "]}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ["\n Sep 5 '18-Feb 14 '19 •\n Vienna, Austria \n "]}
2018-05-16 13:10:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.f6s.com/programs?type[]=accelerator&sort=open>
{'location': ['\n London, United Kingdom \n ']}
2018-05-16 13:10:32 [scrapy.core.engine] INFO: Closing spider (finished)
2018-05-16 13:10:32 [scrapy.extensions.feedexport] INFO: Stored json feed (24 items) in: companies.json
2018-05-16 13:10:32 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 569,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 16600,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 5, 16, 12, 10, 32, 32121),
'item_scraped_count': 24,
'log_count/DEBUG': 27,
'log_count/INFO': 8,
'memusage/max': 49434624,
'memusage/startup': 49434624,
'response_received_count': 2,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2018, 5, 16, 12, 10, 30, 500700)}
2018-05-16 13:10:32 [scrapy.core.engine] INFO: Spider closed (finished)
我的 JSON 输出：
[
{"program": ["K - LAUNCHPAD 2018"]},
{"program": ["Z Nation Lab Real Estate Cohort"]},
{"program": ["C-mint-International"]},
{"program": ["StartOut Growth Lab - 2018 Fall Cohort"]},
{"program": ["IBA Application"]},
{"program": ["WATT Factory Accelerator Programme 2018"]},
{"program": ["AdvantEdge Founder's Adda"]},
{"program": ["SpinLab - The HHL Accelerator"]},
{"program": ["Shell LiveWIRE Accelerator"]},
{"program": ["Shell France Accelerator "]},
{"program": ["ELEVATE by TheVentury"]},
{"program": ["F6S R&D Money Back"]},
{"location": ["\n Jun 1-Jul 20 \u2022\n Berlin, Germany \n "]},
{"location": ["\n Mumbai, India \n "]},
{"location": ["\n Atlanta, United States \n "]},
{"location": ["\n Jul 8-Dec 31 \u2022\n San Francisco, United States \n "]},
{"location": ["\n Mar 19-May 16 \u2022\n Los Angeles, United States \n "]},
{"location": ["\n Jun 3-Nov 30 \u2022\n Gent, Belgium \n "]},
{"location": ["\n Delhi, India \n "]},
{"location": ["\n Leipzig, Germany \n "]},
{"location": ["\n Jun 20 '18-Jun 21 '19 \u2022\n Paris, France \n "]},
{"location": ["\n Jun 20 '18-Jun 1 '19 \u2022\n Paris, France \n "]},
{"location": ["\n Sep 5 '18-Feb 14 '19 \u2022\n Vienna, Austria \n "]},
{"location": ["\n London, United Kingdom \n "]}
]
谢谢！
解决方案
在日期即将到来的某些情况下，您需要格式化位置，如下所示
import scrapy
class CompanySpider(scrapy.Spider):
name = "Company"
def start_requests(self):
urls = [
'https://www.f6s.com/programs?type[]=accelerator&sort=open',
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
for title in response.css('div.result-description'):
location = title.css('div.subtitle span::text').extract_first()
# remove "." from some locations and split by lines
location = location.encode("ascii","ignore").splitlines()
# get second last element and remove whitespaces
location = location[-2].strip()
yield {
'program': title.css('div.title a.action.main.noline::text').extract_first(),
'location': location
}
输出：
[{"program": "Z Nation Lab Real Estate Cohort", "location": "Mumbai, India"},
{"program": "C-mint-International", "location": "Atlanta, United States"},
{"program": "StartOut Growth Lab - 2018 Fall Cohort", "location": "San Francisco, United States"},
{"program": "IBA Application", "location": "Los Angeles, United States"},
{"program": "WATT Factory Accelerator Programme 2018", "location": "Gent, Belgium"},
{"program": "AdvantEdge Founder's Adda", "location": "Delhi, India"},
{"program": "SpinLab - The HHL Accelerator", "location": "Leipzig, Germany"},
{"program": "Shell LiveWIRE Accelerator", "location": "Paris, France"},
{"program": "Shell France Accelerator ", "location": "Paris, France"},
{"program": "ELEVATE by TheVentury", "location": "Vienna, Austria"},
{"program": "F6S R&D Money Back", "location": "London, United Kingdom"},
{"program": "2018 StarterStudio Seed Fund Accelerator", "location": "Orlando, United States"}]
推荐阅读
- jq - 从 JSON 数组中选择版本字符串并使用 jq 将其加一
- scala - 这是避免 OOP 类的 Scala / 函数式编程模式，官方设计模式吗？它有名字吗？
- linux - Linux终端测量命令的cpu使用情况
- python - 要检查 Pandas Dataframe 列的 TRUE/FALSE，如果 TRUE 检查另一列是否满足条件并生成具有值 PASS/FAIL 的新列
- python - 使用 Amazon Textract 从多页文档 PDF 中同步检测和分析文本
- c# - 如何停止在 5 秒内多次调用方法？
- symfony - 如何正确实现与 Doctrine 的“朋友关系”
- .net - .Net Core Avalonia 中如何支持汉字？
- php - 如何从 Ramsey\Uuid\Codec\StringCodec 中提取 uuid？
- javascript - 在 NodeJS 中等待数据返回而不是休眠