时间戳

首页 > 解决方案 > Python 3 使用 BeautifulSoup 抓取网页导致 UnicodeDecodeError

python - Python 3 使用 BeautifulSoup 抓取网页导致 UnicodeDecodeError

问题描述

我正在做一个个人项目。我正在尝试解析网页。不幸的是，我没有意识到抓取网站可能会暂停您对网页的访问。我使用 hide.me 制作了网站的本地副本，但显然它添加了使 BeautifulSoup 难以阅读的信息。这是我的代码：

def pull_safe(location):
url = (os.getcwd())+'/HTML_SOURCES/'+location
page = open(url,encoding="ascii")
soup = BeautifulSoup(page, "html.parser", exclude_encodings=["ascii"])
hospital = list()
templist = list()
tempcount = 0
for td in soup.find('div', {'class':'report'}).parent.find_all('td'):
    if tempcount !=5:
        templist.append(td.text)
        tempcount+=1
    else:
        templist.append(td.text)
        hospital.append(templist)
        templist = list()
        tempcount = 0
return hospital

这是我得到的例外：

Traceback (most recent call last):


File "/home/memeputer/Documents/Projects/NYC Hospital Bed count/main.py", line 51, in <module>
    g = pull_safe(item)
  File "/home/memeputer/Documents/Projects/NYC Hospital Bed count/main.py", line 17, in pull_safe
    soup = BeautifulSoup(page, "html.parser", exclude_encodings=["utf-8"])
  File "/home/memeputer/Documents/Projects/NYC Hospital Bed count/venv/lib/python3.8/site-packages/bs4/__init__.py", line 286, in __init__
    markup = markup.read()
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 59903: invalid start byte

任何帮助表示赞赏。

标签： pythonhtmlutf-8beautifulsoup

解决方案

推荐阅读