时间戳

首页 > 解决方案 > 通过创建单独的函数来解决 try-except 以应用于单行中的许多语句

python-3.x - 通过创建单独的函数来解决 try-except 以应用于单行中的许多语句

问题描述

我正在从https://www.dictionary.com/网站上抓取字典数据。目的是从字典页面中删除不需要的元素并将它们离线保存以供进一步处理。由于网页有些非结构化，因此可能存在也可能不存在以下代码中提到的要删除的元素；缺少元素会产生异常（在片段 2 中）。而且由于在实际代码中，有许多元素要删除，它们可能存在或不存在，如果我们将 应用于每个这样的语句，代码行数将急剧增加。try - except

因此，我正在通过为try - except（在代码片段 3 中）创建一个单独的函数来解决这个问题，这是我从这里得到的想法。但是我无法让代码片段 3 中的代码像命令一样工作，例如soup.find_all('style')返回None应该返回style类似于片段 2 的所有标签列表的位置。我无法直接应用引用的解决方案，因为有时我必须达到预期通过引用其parent或sibling等间接删除元素soup.find('h2',{'class':'css-1iltn77 e17deyx90'}).parent

Snippet 1 用于设置代码执行的环境。

如果您能提供一些建议以使代码段 3 正常工作，那就太好了。

片段1（设置执行代码的环境）：

import urllib.request
import requests
from bs4 import BeautifulSoup
import re

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
           'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',}

folder = "dictionary_com"

片段2（工作）：

def makedefinition(url):
    success = False
    while success==False:
        try:
            request=urllib.request.Request(url,headers=headers)
            final_url = urllib.request.urlopen(request, timeout=5).geturl()
            r = requests.get(final_url, headers=headers, timeout=5)
            success=True
        except:
            success=False
    
    soup = BeautifulSoup(r.text, 'lxml')
    
    soup = soup.find("section",{'class':'css-1f2po4u e1hj943x0'})
    
    # there are many more elements to remove. mentioned only 2 for shortness
    remove = soup.find_all("style") # style tags
    remove.extend(safe_execute(soup.find('h2',{'class':'css-1iltn77 e17deyx90'}).parent)) # related content in the page
    
    for x in remove: x.decompose()
    
    return(soup)

# testing code on multiple urls
#url = "https://www.dictionary.com/browse/a"
#url = "https://www.dictionary.com/browse/a--christmas--carol"
#url = "https://www.dictionary.com/brdivowse/affection"
#url = "https://www.dictionary.com/browse/hot"
#url = "https://www.dictionary.com/browse/move--on"
url = "https://www.dictionary.com/browse/cuckold"
#url = "https://www.dictionary.com/browse/fear"
maggi = makedefinition(url)

with open(folder+"/demo.html", "w") as file:
    file.write(str(maggi))

片段3（不工作）：

soup = None

def safe_execute(command):
    global soup
    try:
        print(soup) # correct soup is printed
        print(exec(command)) # this should print the list of style tags but printing None, and for related content this should throw some exception
        return exec(command) # None is being returned for style
    except Exception:
        print(Exception.with_traceback())
        return []

def makedefinition(url):
    global soup
    success = False
    while success==False:
        try:
            request=urllib.request.Request(url,headers=headers)
            final_url = urllib.request.urlopen(request, timeout=5).geturl()
            r = requests.get(final_url, headers=headers, timeout=5)
            success=True
        except:
            success=False

    soup = BeautifulSoup(r.text, 'lxml')
    
    soup = soup.find("section",{'class':'css-1f2po4u e1hj943x0'})

    # there are many more elements to remove. mentioned only 2 for shortness
    remove = safe_execute("soup.find_all('style')") # style tags
    remove.extend(safe_execute("soup.find('h2',{'class':'css-1iltn77 e17deyx90'}).parent")) # related content in the page

    for x in remove: x.decompose()

    return(soup)

# testing code on multiple urls
#url = "https://www.dictionary.com/browse/a"
#url = "https://www.dictionary.com/browse/a--christmas--carol"
#url = "https://www.dictionary.com/brdivowse/affection"
#url = "https://www.dictionary.com/browse/hot"
#url = "https://www.dictionary.com/browse/move--on"
url = "https://www.dictionary.com/browse/cuckold"
#url = "https://www.dictionary.com/browse/fear"
maggi = makedefinition(url)

with open(folder+"/demo.html", "w") as file:
    file.write(str(maggi))

标签： python-3.xweb-scrapingerror-handlingbeautifulsouptry-except

解决方案

在代码段 3 中的代码中，您使用exec内置方法，None无论它使用其参数做什么，它都会返回。有关详细信息，请参阅此SO 线程。

补救措施：

用于exec修改变量并返回它，而不是返回其exec自身的输出。

def safe_execute(command):
   d = {}
   try:
       exec(command, d)
       return d['output']
   except Exception:
       print(Exception.with_traceback())
       return []

然后像这样调用它：

remove = safe_execute("output = soup.find_all('style')")

编辑：

执行此代码后，再次None返回。然而，在调试时，try如果我们打印print(soup)了正确的soup值，则在部分内部，但exec(command,d)给出NameError: name 'soup' is not defined.

这种差异已通过使用eval()代替来克服exec()。定义的函数是：

def safe_execute(command):
    global soup
    try:
        output = eval(command)
        return(output)
    except Exception:
        return []

电话看起来像：

remove = safe_execute("soup.find_all('style')")
remove.extend(safe_execute("soup.find('h2',{'class':'css-1iltn77 e17deyx90'}).parent"))

推荐阅读