时间戳

首页 > 解决方案 > 如何使用 python、html.parser 和 regex 提取 url

python - 如何使用 python、html.parser 和 regex 提取 url

问题描述

我需要在 Python 中创建一个程序来解析 .html 文件中的所有 URL 并打印出所有标签和链接，如下所示：

meta: https://someurl.com
a: https://someurl.com
link: css/bootstrap.min.css
script: https://somescript.js

目前，我所拥有的是

from html.parser import HTMLParser
import re

class HeadParser(HTMLParser):

def handle_starttag(self, tag, attrs):
    #use re.findall to get all the links
    links = re.findall("http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\), ]|(?:%[0-9a-fA-F][0-9a-fA-F]))+", website)
    for url in links:
        print("{0}: {1}".format(tag, url))

website = open("./head.html").read()            
HeadParser().feed(website)

它回到我身边

head: https://scooptacular.net
head: https://scooptacular.net/img/uploaded/379d05029c0d84618c70ac037a25fd88.jpg
head: https://scooptacular.net/img/uploaded/4baaa58a1a37fd3da3e4e78caf366b7f.jpg
head: https://fonts.googleapis.com/css?family=Montserrat:400,700
head: https://fonts.googleapis.com/css?family=Kaushan+Script' rel='stylesheet' type='text/css'>
head: https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
head: https://fonts.googleapis.com/css?family=Roboto+Slab:400,100,300,700' rel='stylesheet' type='text/css'>
head: https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js
head: https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js
meta: https://scooptacular.net
meta: https://scooptacular.net/img/uploaded/379d05029c0d84618c70ac037a25fd88.jpg
meta: https://scooptacular.net/img/uploaded/4baaa58a1a37fd3da3e4e78caf366b7f.jpg
meta: https://fonts.googleapis.com/css?family=Montserrat:400,700
meta: https://fonts.googleapis.com/css?family=Kaushan+Script' rel='stylesheet' type='text/css'>
meta: https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
meta: https://fonts.googleapis.com/css?family=Roboto+Slab:400,100,300,700' rel='stylesheet' type='text/css'>
meta: https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js
meta: https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js
meta: https://scooptacular.net
meta: https://scooptacular.net/img/uploaded/379d05029c0d84618c70

如您所见，它为我返回每个标签的链接，甚至是重复的，并且不返回任何本地文件链接。我的代码有什么问题？

编辑：

我正在使用的 html 是：

<head>
<meta property="og:url" content="https://scooptacular.net" />    
<meta property="og:image" content="https://scooptacular.net/img/uploaded/379d05029c0d84618c70ac037a25fd88.jpg" />
<meta property="og:image" content="https://scooptacular.net/img/uploaded/4baaa58a1a37fd3da3e4e78caf366b7f.jpg" />

<link href="css/bootstrap.min.css" rel="stylesheet">

<link href="css/agency.css" rel="stylesheet">

<link href="font-awesome-4.1.0/css/font-awesome.min.css" rel="stylesheet" type="text/css">

        <link href="https://fonts.googleapis.com/css?family=Montserrat:400,700" rel="stylesheet" type="text/css">
    <link href='https://fonts.googleapis.com/css?family=Kaushan+Script' rel='stylesheet' type='text/css'>
    <link href='https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
    <link href='https://fonts.googleapis.com/css?family=Roboto+Slab:400,100,300,700' rel='stylesheet' type='text/css'>


<link href="css/bootstrap-formhelpers.min.css" rel="stylesheet" media="screen">

    <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
    <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
</head>

标签： pythonregex

解决方案

主要问题是handle_starttag每个标签都会调用它，并且每次调用都在整个页面中搜索与您的正则表达式匹配的内容，而不仅仅是您所在的标签（您传递给的第二个参数re.findall是website）。

我不明白为什么你需要在这里使用正则表达式。为什么不只依赖标签是否具有href,src或content属性：

from html.parser import HTMLParser


class HeadParser(HTMLParser):

    def handle_starttag(self, tag, attrs):
        for attr in attrs:
            if attr[0] in ['href', 'src', 'content']:
                print('{0}: {1}'.format(tag, attr[1]))


website = open("./head.html").read()
HeadParser().feed(website)

输出：

meta: https://scooptacular.net
meta: https://scooptacular.net/img/uploaded/379d05029c0d84618c70ac037a25fd88.jpg
meta: https://scooptacular.net/img/uploaded/4baaa58a1a37fd3da3e4e78caf366b7f.jpg
link: css/bootstrap.min.css
link: css/agency.css
link: font-awesome-4.1.0/css/font-awesome.min.css
link: https://fonts.googleapis.com/css?family=Montserrat:400,700
link: https://fonts.googleapis.com/css?family=Kaushan+Script
link: https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic,700italic
link: https://fonts.googleapis.com/css?family=Roboto+Slab:400,100,300,700
link: css/bootstrap-formhelpers.min.css
script: https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js
script: https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js

推荐阅读