时间戳

首页 > 解决方案 > requests.get(url) 没有返回这个特定的 url

python - requests.get(url) 没有返回这个特定的 url

问题描述

我正在尝试使用 requests.get(url).text 从该网站获取 HTML。但是，当使用此特定 url 调用 requests.get(url) 时，无论我等待多长时间，它都不会返回。这适用于其他网址，但这个特别给我带来了麻烦。代码如下

from bs4 import BeautifulSoup
import requests

source = requests.get('https://www.carmax.com/cars/all', allow_redirects=True).text

soup = BeautifulSoup(source, 'lxml')

print(soup.prettify().encode('utf-8'))

谢谢你的帮助！

标签： pythonwebpython-requestsscreen-scraping

解决方案

尝试：

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36', "Upgrade-Insecure-Requests": "1","DNT": "1","Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Accept-Language": "en-US,en;q=0.5","Accept-Encoding": "gzip, deflate"}
html = requests.get("https://www.carmax.com/cars/all",headers=headers)
soup = BeautifulSoup(html.content, 'html.parser')
print(soup.prettify())

推荐阅读