python - 用漂亮的汤刮掉 yt 格式的字符串

Question

我试图用 BeautifulSoup 抓取 yt 格式的字符串，但它总是给我一个错误。这是我的代码：

import requests
import bs4
from bs4 import BeautifulSoup

r = requests.get('https://www.youtube.com/channel/UCPyMcv4yIDfETZXoJms1XFA')
soup = bs4.BeautifulSoup(r.text, "html.parser")
def onoroff():
    onoroff = soup.find('yt-formatted-string',{'id','subscriber-count'}).text
    return onoroff


print("Subscribers:  "+str(onoroff().strip()))

这是我得到的错误

AttributeError: 'NoneType' object has no attribute 'text'

还有另一种方法来抓取 yt 格式的字符串吗？

标签： pythonbeautifulsoupscrape

Answer 1

大多数 Youtube 内容是通过 JavaScript 生成的，BeautifulSoup 不具备这种能力，但是您可以通过抓取源代码上的 json 对象而不是直接抓取 HTML 元素来获得运气，即：

import requests, json, re

h = {
    'Host': 'www.youtube.com',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0',
    'Accept': '*/*',
    'Accept-Language': 'en-US,pt;q=0.7,en;q=0.3',
    'Referer': 'https://www.youtube.com/channel/UCPyMcv4yIDfETZXoJms1XFA',
}
u = "https://www.youtube.com/channel/UCPyMcv4yIDfETZXoJms1XFA"
html = requests.get(u, headers=h).text

# lets get the json object that contains all the info we need from the source code and convert it into a python dict that we can use later
matches = re.findall(r'window\["ytInitialData"\] = (.*\}\]\}\}\});', html, re.IGNORECASE | re.DOTALL)
if matches:
    j = json.loads(matches[0])
    # browse the json object and search the info you need : https://jsoneditoronline.org/#left=cloud.123ad9bb8bbe498c95f291c32962aad2
    # We are now ready to get the the number of subscribers (among other info):

    subscribers = j['header']['c4TabbedHeaderRenderer']['subscriberCountText']['runs'][0]["text"]
    print(subscribers)
    # 110 subscribers

演示