时间戳

首页 > 解决方案 > 如何限制在for循环中填充数据框的行数

python - 如何限制在for循环中填充数据框的行数

问题描述

我编写了以下从网站上抓取多个页面的函数。我只想得到前 20 页左右。如何限制填写数据框的行数：

def scrape_page(poi,page_name):
    base_url="https://www.fake_website.org/"
    report_url=(base_url+poi)
    page=urlopen(report_url)
    experiences=BeautifulSoup(page,"html.parser")
    empty_list=[]
    for link in experiences.findAll('a', attrs={'href': re.compile(page_name+".shtml$")}):
        url=urljoin(base_url, link.get("href"))
        subpage=urlopen(url)
        expages=BeautifulSoup(subpage, "html.parser")
        for report in expages.findAll('a', attrs={'href': re.compile("^/experiences/exp")}):
            url=urljoin(base_url, report.get("href"))
            reporturlopen=urlopen(url)
            reporturl=BeautifulSoup(reporturlopen, "html.parser")
            book_title= reporturl.findAll("div",attrs={'class':'title'})
            for i in book_title:
                title=i.get_text()
            book_genre= reporturl.findAll("div",attrs={'class':'genre'})
            for i in book_genre:
                genre=i.get_text()
            book_author= reporturl.findAll("div",attrs={'class':'author'})
            for i in book_author:
                author=i.get_text()
                author = re.sub("by", "",author)
     empty_list.append({'title':title,'genre':genre,'author':author})
     setattr(sys.modules[__name__], '{}_df'.format(poi+"_"+page_name), empty_list)

标签： pythondataframeweb-scraping

解决方案

例如，您可以添加一个 while 循环：

i = 0
while i < 20:
    < insert your code >
    i += 1

推荐阅读