爬虫大作业

1.选一个自己感兴趣的主题（所有人不能雷同）。

我选择了虎扑nba的体育新闻页面，与校园新闻版面类似，爬去50页

2.用python 编写爬虫程序，从网络上爬取相关主题的数据。

利用所学知识，导入要用的类

import requests
from bs4 import BeautifulSoup
import jieba

审查元素，获取网页内容

3.对爬了的数据进行文本分析，生成词云。

for i in range(2294450,2294500):
        pages = i;
        nexturl = 'https://voice.hupu.com/nba/%s.html' % (pages)
        reslist = requests.get(nexturl)
        reslist.encoding = 'utf-8'
        soup_list = BeautifulSoup(reslist.text, 'html.parser')
        for news in soup_list.find_all('div',class_='artical-main-content'):
            print(news.text)
            f = open('hpnba.txt', 'a', encoding='utf-8')
            f.write(news.text)
            f.close()
def changeTitleToDict():
    f = open("hpnba.txt", "r", encoding='utf-8')
    str = f.read()
    stringList = list(jieba.cut(str))
    delWord = {"+", "/", "（", "）", "【", "】", " ", "；", "！", "、"}
    stringSet = set(stringList) - delWord
    title_dict = {}
    for i in stringSet:
        title_dict[i] = stringList.count(i)
    print(title_dict)
    return title_dict

4.对文本分析结果进行解释说明。

这是爬取的新闻文字内容，存放在hpnba.txt中

这是我的词云的背景图片，生成的大致样式。

词云如上

5.写一篇完整的博客，描述上述实现过程、遇到的问题及解决办法、数据分析思想及结论。

遇到的问题就是，我原本想生成五角星的词云，找了两张图片，都不行，后来换了一张小猪，就可以了，原因目前不知道。

6.最后提交爬取的全部数据、爬虫及数据分析源代码。

import requests
from bs4 import BeautifulSoup
import jieba
from PIL import Image,ImageSequence
import numpy as np
import matplotlib.pyplot as plt
from wordcloud import WordCloud,ImageColorGenerator
for i in range(2294450,2294500):
        pages = i;
        nexturl = 'https://voice.hupu.com/nba/%s.html' % (pages)
        reslist = requests.get(nexturl)
        reslist.encoding = 'utf-8'
        soup_list = BeautifulSoup(reslist.text, 'html.parser')
        for news in soup_list.find_all('div',class_='artical-main-content'):
            print(news.text)
            f = open('hpnba.txt', 'a', encoding='utf-8')
            f.write(news.text)
            f.close()
def changeTitleToDict():
    f = open("hpnba.txt", "r", encoding='utf-8')
    str = f.read()
    stringList = list(jieba.cut(str))
    delWord = {"+", "/", "（", "）", "【", "】", " ", "；", "！", "、"}
    stringSet = set(stringList) - delWord
    title_dict = {}
    for i in stringSet:
        title_dict[i] = stringList.count(i)
    print(title_dict)
    return title_dict


# 获取上面保存的字典
title_dict = changeTitleToDict()
graph = np.array(title_dict)
font = r'C:\Windows\Fonts\simhei.ttf'


image= Image.open('./3.jpg')
graph = np.array(image)
font=r'C:\Windows\Fonts\simhei.TTF'
wc = WordCloud(font_path=font,background_color='White',max_words=50,mask=graph)
wc.generate_from_frequencies(title_dict)
image_color = ImageColorGenerator(graph)
plt.imshow(wc)
plt.axis("off")
plt.show()

爬虫大作业

推荐阅读