今天用scrapy框架爬取一下所有知乎用户的信息。道理很简单，找一个知乎大V（就是粉丝和关注量都很多的那种），找到他的粉丝和他关注的人的信息，然后分别再找这些人的粉丝和关注的人的信息，层层递进，这样下来，只要有关注的人或者有粉丝的账号，几乎都能被爬下来。话不多说，进入正题。
1、首先按照上篇博客的介绍，先建立项目，然后建一个spider文件，scrapy genspider zhihu www.zhihu.com.
进入settings.py，修改内容 ROBOTSTXT_OBEY = False，意思是知乎网页中禁止爬取的内容也可以爬的到。
再添加一个User_agent，因为知乎是通过浏览器识别的，否则知乎会禁止爬取
DEFAULT_REQUEST_HEADERS = { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en', 'User-agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36' }
2、进入知乎网页页面，搜索一个大V，这里我用的是vczh，这个账号的关注量和粉丝都特别多。
改写zhihu.py的内容如下：
from scrapy import Request,Spider class ZhihuSpider(Spider): name = 'zhihu' allowed_domains = ['www.zhihu.com'] start_urls = ['http://www.zhihu.com/'] def start_requests(self): # zhihu.py 会先调用 start_requests 函数 url = 'https://www.zhihu.com/api/v4/members/ji-he-61-7?include=allow_message%2Cis_followed%2Cis_following%2Cis_org%2Cis_blocking%2Cemployments%2Canswer_count%2Cfollower_count%2Carticles_count%2Cgender%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics' yield Request(url,callback=self.parse) def parse(self, response): print(response.text)
随便找一个vczh关注的人，把他的Request URL拿过来请求一下，发现正常输出了结果，说明是能够获取 vczh 自己的信息的。
将URL换成 followees（vczh关注的人） 的Request URL，同样输出了正确的结果，说明 vczh 关注的人的信息也能正常输出。
经过观察发现，每个用户详细信息的 URL 只有 url_token 不同，因此分别构造 user_url 和 follows_url：
class ZhihuSpider(Spider): name = 'zhihu' allowed_domains = ['www.zhihu.com'] start_urls = ['http://www.zhihu.com/'] start_user = 'excited-vczh' user_url = 'https://www.zhihu.com/api/v4/members/{user}?include={include}' user_query = 'allow_message,is_followed,is_following,is_org,is_blocking,employments,answer_count,follower_count,articles_count,gender,badge[?(type=best_answerer)].topics' follows_url = 'https://www.zhihu.com/api/v4/members/{user}/followees?include={include}&offset={offset}&limit={limit}' follows_query = 'data[*].answer_count,articles_count,gender,follower_count,is_followed,is_following,badge[?(type=best_answerer)].topics' def start_requests(self): yield Request(self.user_url.format(user=self.start_user, include=self.user_query),callback=self.parse_user) yield Request(self.follows_url.format(user=self.start_user, include=self.follows_query, offset=0, limit=20),callback=self.parse_follows) def parse_user(self, response): result = json.loads(response.text) item = Zhihu3Item() for field in item.fields: if field in result.keys(): item[field] = result.get(field) yield item def parse_follows(self, response): results = json.loads(response.text) if 'data' in results.keys(): for result in results.get('data'): yield Request(self.user_url.format(user=result.get('url_token'), include=self.user_query), self.parse_user) if 'paging' in results.keys() and results.get('paging').get('is_end') == False: # 如果当前页不是最后一页 next_page_str = results.get('paging').get('next') next_page = next_page_str.replace('https://www.zhihu.com/', 'https://www.zhihu.com/api/v4/') # next_page = next_page_str[0:22] + 'api/v4/' + next_page_str[22:len(next_page_str)] # 这种写法也行 yield Request(next_page, self.parse_follows)
结果正确输出了 vczh 和 他所关注的用户的详细信息。
3、接下来，把 vczh 关注的用户 所 关注的用户的信息输出来：
在 def parse_user(self, response): 函数后添加一句
yield Request(self.follows_url.format(user=result.get('url_token'), include=self.follows_query, offset=0, limit=20),self.parse_follows) 就好了。
4、除了获取用户关注的人以外，还要获取用户的粉丝信息。经观察，发现粉丝信息的 URL 与关注的人的 URL 类似，改写以后的最终版本如下：
class ZhihuSpider(Spider): name = 'zhihu' allowed_domains = ['www.zhihu.com'] start_urls = ['http://www.zhihu.com/'] start_user = 'excited-vczh' user_url = 'https://www.zhihu.com/api/v4/members/{user}?include={include}' user_query = 'allow_message,is_followed,is_following,is_org,is_blocking,employments,answer_count,follower_count,articles_count,gender,badge[?(type=best_answerer)].topics' follows_url = 'https://www.zhihu.com/api/v4/members/{user}/followees?include={include}&offset={offset}&limit={limit}' follows_query = 'data[*].answer_count,articles_count,gender,follower_count,is_followed,is_following,badge[?(type=best_answerer)].topics' followers_url = 'https://www.zhihu.com/api/v4/members/{user}/followers?include={include}&offset={offset}&limit={limit}' followers_query = 'data[*].answer_count,articles_count,gender,follower_count,is_followed,is_following,badge[?(type=best_answerer)].topics' def start_requests(self): yield Request(self.user_url.format(user=self.start_user, include=self.user_query),callback=self.parse_user) yield Request(self.follows_url.format(user=self.start_user, include=self.follows_query, offset=0, limit=20),callback=self.parse_follows) yield Request(self.followers_url.format(user=self.start_user, include=self.followers_query, offset=0, limit=20),callback=self.parse_followers) def parse_user(self, response): result = json.loads(response.text) item = Zhihu3Item() for field in item.fields: if field in result.keys(): item[field] = result.get(field) yield item yield Request(self.follows_url.format(user=result.get('url_token'), include=self.follows_query, offset=0, limit=20),self.parse_follows) yield Request(self.followers_url.format(user=result.get('url_token'), include=self.followers_query, offset=0, limit=20),self.parse_followers) def parse_follows(self, response): results = json.loads(response.text) if 'data' in results.keys(): for result in results.get('data'): yield Request(self.user_url.format(user=result.get('url_token'), include=self.user_query), self.parse_user) if 'paging' in results.keys() and results.get('paging').get('is_end') == False: # 如果当前页不是最后一页 next_page_str = results.get('paging').get('next') next_page = next_page_str.replace('https://www.zhihu.com/', 'https://www.zhihu.com/api/v4/') # next_page = next_page_str[0:22] + 'api/v4/' + next_page_str[22:len(next_page_str)] # 这种写法也行 yield Request(next_page, self.parse_follows) def parse_followers(self, response): results = json.loads(response.text) if 'data' in results.keys(): for result in results.get('data'): yield Request(self.user_url.format(user=result.get('url_token'), include=self.user_query), self.parse_user) if 'paging' in results.keys() and results.get('paging').get('is_end') == False: # 如果当前页不是最后一页 next_page_str = results.get('paging').get('next') next_page = next_page_str.replace('https://www.zhihu.com/', 'https://www.zhihu.com/api/v4/') # next_page = next_page_str[0:22] + 'api/v4/' + next_page_str[22:len(next_page_str)] # 这种写法也行 yield Request(next_page, self.parse_followers)
5、将输出的内容保存到MongoDb中
跟上一篇博文类似，在pipeline.py中写入一下内容，直接上代码：
import pymongo class MongoPipeline(object): def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri self.mongo_db = mongo_db @classmethod def from_crawler(cls, crawler): return cls( mongo_uri=crawler.settings.get('MONGO_URI'), mongo_db=crawler.settings.get('MONGO_DB') ) def open_spider(self, spider): self.client = pymongo.MongoClient(self.mongo_uri) self.db = self.client[self.mongo_db] def process_item(self, item, spider): self.db['information'].update({'url_token':item['url_token']},{'$set':item},True) return item # 根据'url_token'去重，True 表示如果重复就执行刷新操作，如果不重复就执行插入操作 def close_spider(self, spider): self.client.close()
在settings.py中写入以下代码：
ITEM_PIPELINES = { 'zhihu_3.pipelines.MongoPipeline': 300 } MONGO_URI='localhost' MONGO_DB = 'zhihu_3'
运行即可，成功保存到MongoDB中