python - Python程序随机挂断

Question

我正在使用BeautifulSoup在 Python 中进行一些网页抓取，这涉及访问大约 500 个类似的网页以将数据放入 .txt 文件中。

但是，我遇到了一些问题：

• 我的程序的 CPU 使用率（通过任务管理器检查）随机下降到 0% 并保持这种状态。
• 我的命令提示符本身变得无响应，因为 Ctrl + C 不会杀死程序。

• 这发生在看似随机的时间，介于第 8 个网页和第 480 个网页之间。

  def getAnime():
  
    for index in range(2, 502):
  
        # gets anime statistics from HTML
        container = containers[index]
        ranking = container.td.text
        name = container.findAll('td', {'class', 't'})
        link = 'https://www.animenewsnetwork.com' + name[0].a['href']
        name = name[0].text
        statistics = container.findAll('td', {'class', 'r'})
        rating = statistics[0].text
        numVotes = statistics[1].text
  
        # prints out anime stats to file
        currentAnime = Anime(name, ranking, rating, numVotes, link)
        animeFile.write('\nname: ' + name)
        animeFile.write('\nlink: ' + link)
        animeFile.write('\nranking: ' + ranking)
        animeFile.write('\nrating: ' + rating)
        animeFile.write('\nvotes: ' + numVotes)
  
        # Goes to the webpage for the current anime
        animeClient = uReq(link)
        animeHTML = animeClient.read()
        animeClient.close()
        pageSoup = soup(animeHTML, 'html.parser')
  
        # Genres of the current anime
        try:
            genreDiv = pageSoup.find(id='infotype-30')
            genres = genreDiv.findAll('span')
            genreList = []
            for genre in genres:
                genreList.append(genre.a.text)
            currentAnime.genres = genreList
        except:
            currentAnime.genres = 'unknown'
  
        # Themes of the current anime
        try:
            themes = pageSoup.find(id='infotype-31').findAll('span')
            themeList = []
            for theme in themes:
                themeList.append(theme.a.text)
            currentAnime.themes = themeList
        except:
            currentAnime.themes = 'unknown'
  
        # Premiere date of the current anime
        try:
            date = pageSoup.find(id='infotype-9').div.text
            currentAnime.premiereDate = date
        except:
            currentAnime.premiereDate = 'unknown'
  
        # Director of the current anime
        try:
            director = pageSoup.find('b', text='Director').parent.a.text
            currentAnime.director = director
        except:
            currentAnime.director = 'unknown'
  
        # Production Studio of the current anime
        try:
            productionStudio = pageSoup.find('b', text='Production').parent.a.text
            currentAnime.studio = productionStudio
        except:
            currentAnime.studio = 'unknown'
  
        # Prints the genres
        animeFile.write('\ngenres: ')
        for genre in currentAnime.genres:
            animeFile.write(genre + ', ')
        # Prints the themes
        animeFile.write('\nthemes: ')
        for theme in currentAnime.themes:
            animeFile.write(theme + ', ')
        # Prints the premiere date, director, and studio
        animeFile.write('\npremiere date: ' + currentAnime.premiereDate)
        animeFile.write('\ndirector: ' + currentAnime.director)
        animeFile.write('\nproduction studio: ' + currentAnime.studio)
  
        animeFile.write('\n')
  

标签： pythonpython-3.xweb-scrapingbeautifulsoup

Answer 1

CTRL-C 只是向 Python 发送一个键盘中断命令。这意味着当在 BeautifulSoup 中发出 HTML 请求时，您可能只是提升了一个异常级别。CtrlBreak将完全停止程序。

您的脚本很可能正在运行到无响应的网页中。您的 CPU将处于 0%，因为它正在等待 Web 服务。link我建议在每次调用之前打印代码中的值uReq以跟踪发生的位置。