python - 使用 url 通过 urllib python 请求时出现编码问题

Question

我正在尝试使用 urllib 从 URL 中获取响应，然后使用 bs4 来提取数据，但是在某些 URL 上我遇到了这个异常并且

from urllib.request import urlopen 

处理请求失败。鉴于以下错误：

编辑：错误：（回溯完成）

Traceback (most recent call last):
  File "G:/Internships/Botnostic Solutions/AllScrapers/careerz360/test_bs4.py", line 74, in <module>
    html = urlopen(url_link)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\urllib\request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1240, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Users\muhammadhamaadlatif\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1107, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 18: ordinal not in range(128)

代码片段：

from bs4 import BeautifulSoup
from urllib.request import urlopen
urls_list = []
with open('links_file.txt', 'r', encoding="utf-8") as linksFile:
    for l in linksFile:
        # url_parsed = urllib.parse.quote(l.strip())
        urls_list.append(l.strip())
linksFile.close()

将所有链接附加到列表中，以便可以使用以下方法进一步处理它们：urllib.urlopen()

这是在此处创建问题的链接：

urls_list = ['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']

重复代码：

for url in urls_list:
    html = urlopen(url)
    if html.getcode() == 200:
        response = BeautifulSoup(html.read(), "lxml")
        company_name = response.select('h2#business-name')[0].text.strip()

使用不同的编码：

我也尝试过不同的编码，如：latin-1、utf-8 和 iso 等，但得到相同的错误。如果可能的话，请给我一个解决方案。谢谢并恭祝安康

标签： pythonurllib

Answer 1

这段代码对我有用。在file.txt我有一个网址https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853：

import requests
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

soup = BeautifulSoup(requests.get(urls_list[0]).text, 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

印刷：

['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
Protègè Global

我正在使用 Python 3.6.8

编辑：urlopen你需要quote()的 URL：

from urllib.request import urlopen
from urllib.parse import quote, urlparse
from bs4 import BeautifulSoup

urls_list = []
with open('file.txt', 'r') as linksFile:
    for l in linksFile:
        urls_list.append(l.strip())

print(urls_list)

p = urlparse(urls_list[0])
url = p.scheme + '://' + p.netloc + quote(p.path)

print(url)

html = urlopen(url)
soup = BeautifulSoup(html.read(), 'lxml')

company_name = soup.select('h2#business-name')[0].text.strip()

print(company_name)

印刷：

['https://www.careerz360.com/pakistan/protègè-global-android-developer-karachi-jobs-108853']
https://www.careerz360.com/pakistan/prot%C3%A8g%C3%A8-global-android-developer-karachi-jobs-108853
Protègè Global