原文链接:http://www.limerence2017.com/2019/10/22/python29/
抓取准备
今天是10月24日，祝所有程序员节日快乐。今天打算写个爬虫抓取3DMGAME论坛美女cosplay壁纸。
论坛首页网址为https://www.3dmgame.com/tu_53_1/
我们点击其中一个图集，然后网页跳转，看下源码
<div class="dg-wrapper"> <a data-src = "/uploads/images/thumbpicfirst/20190730/1564452665_126346.jpg"> <div class="img"><img src="https://img.3dmgame.com/uploads/images/thumbpicfirst/20190730/1564452665_126346.jpg"></div> <div class="miaoshu"> <p></p> <div class="num"><i></i> /<u></u></div> </div> </a> <a data-src = "/uploads/images/thumbpicfirst/20190730/1564452665_242197.jpg"> <div class="img"><img src="https://img.3dmgame.com/uploads/images/thumbpicfirst/20190730/1564452665_242197.jpg"></div> <div class="miaoshu"> <p></p> <div class="num"><i></i> /<u></u></div> </div> </a>
网址是静态的，我们直接提取其中的图片链接再下载即可。
抓取网页采用的是python的requests库，直接发送http请求即可。收到回包后，通过BeautifulSoup提炼其中图片地址再次下载即可。
另外我们的界面用的是python自带的Tinker编写的。
代码实现
实现线程装饰器
def thread_run(func): def wraper(*args, **kwargs): t = threading.Thread(target=func, args=args, kwargs=kwargs) t.daemon = True t.start() return wraper
我们实现了DownloadFrame类封装了一个装饰器，启动线程并调用传入的函数。
类里实现如下方法
def prepare(self, downloadlinks): self.flag = True self.downloadlinks = downloadlinks self.base_url = self.downloadlinks fail = 0 try: url = self.base_url result = requests.get(url, headers=HEADERS,timeout=10) restxt = result.content.decode('UTF-8') soup = BeautifulSoup(restxt,'lxml') titles = soup.select('div .bt') if titles is None or len(titles)==0: print("html page res not found ! \n") return title = re.split(r'[;,\s]',titles[0].text)[0] curdir = os.path.dirname(os.path.abspath(__file__)) picpath = os.path.join(curdir,title) if not os.path.exists(picpath): os.mkdir(picpath) print(picpath) imglist = soup.select('.dg-wrapper img') if imglist is None or len(imglist)==0: print("html page res not found ! \n") return self.downloadPic(imglist,picpath) except Exception as e: print(e) time.sleep(3)
prpare函数实现了请求指定网页，并用BeautifulSoup处理回包的功能。
@thread_run def download(self, url, path): try: if lock.acquire(): self.name += 1 imgname = str(self.name)+'.'+url.split('.')[-1] filename = os.path.join(path,imgname) lock.release() print(url) print(filename) # res = requests.get(url, headers=header ) res = requests.get(url, headers=HEADERS,timeout=10 ) with open(filename, 'wb') as f: f.write(res.content) # 下载完后检查是否完成下载 if lock.acquire(): if self.flag: self.flag = False messagebox.showinfo("提示", "下载完成") lock.release() except Exception as e: print(e)
效果展示download传给了我们之前封装的装饰器thread_fun, download实现了下载指定图片的功能。
下载图片
感谢关注我的公众号