python - 应用过滤器后如何从网页中抓取数据表？
问题描述
我正在尝试构建一个数据抓取工具，但是，在应用必要的过滤器后，表的值会发生变化。我不确定如何使用 selenium 或其他工具应用过滤器。
我的计划是加载基表，然后弄清楚如何应用过滤器并改进我的代码，但即使将基表从网页中删除，我仍然卡住了。我要应用的过滤器位于网站上标有“Slates”的下拉工具栏上' https://rotogrinders.com/projected-stats/nfl '
我相当有信心这段代码得到了正确的表格：
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
url = 'https://rotogrinders.com/projected-stats/nfl-qb?site=fanduel'
driver.get(url)
table = driver.find_element_by_xpath("//*[@id='proj-stats']")
然而，将其转换为 pandas 数据框并不顺利。
results_table = []
for row in table:
temp = []
columns = row.find_element_by_xpath("//*[@id='proj-stats']/div[1]")
for column in columns:
temp.append(column.text)
results_table.append(temp)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-79-bdda19bc35a3> in <module>
1 results_table = []
----> 2 for row in table:
3 temp = []
4 columns = row.find_element_by_xpath("//*[@id='proj-stats']/div[1]")
5 for column in columns:
TypeError: 'WebElement' object is not iterable
解决方案
所有这些都在<script>标签中的 json 格式中。例如，您可以遍历 slate id，并将它们与这些 slate 的球员和薪水相匹配：
import requests
from bs4 import BeautifulSoup
import json
url = 'https://rotogrinders.com/projected-stats/nfl-qb?site=fanduel'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
script = soup.find_all('script')[12].text
jsonStr_slate = script.split('slates:')[-1]
jsonStr_slate = jsonStr_slate.split('schedules:')
jsonStr_slate = jsonStr_slate[0].rsplit(',',1)[0]
slatesData = json.loads(jsonStr_slate)
script = soup.find_all('script')[13].text
jsonStr = script.split('data = ')[-1]
jsonStr = jsonStr.rsplit(';',4)[0]
jsonData = json.loads(jsonStr)
for each in jsonData:
name = each['player_name']
for slate in each['import_data']:
slate_id = slate['slate_id']
salary = slate['salary']
for k, v in slatesData.items():
if v['importId'] == slate_id:
print ('%-20s $%-8s %s' %(name, salary, k))
输出：
Russell Wilson $8600 8:20pm Thu-Mon
Russell Wilson $8600 2:00pm Main
Russell Wilson $8600 2:00pm Sun-Mon
Russell Wilson $9500 2:00pm SuperFlex
Russell Wilson $8600 5:05pm 4pm Only
Lamar Jackson $8000 8:20pm Thu-Mon
Lamar Jackson $8000 2:00pm Sun-Mon
Lamar Jackson $8800 2:00pm SuperFlex
Mitchell Trubisky $6500 8:20pm Thu-Mon
Mitchell Trubisky $6500 2:00pm Main
Mitchell Trubisky $6500 2:00pm 1pm Only
Mitchell Trubisky $6500 2:00pm Sun-Mon
Mitchell Trubisky $6800 2:00pm SuperFlex
Deshaun Watson $8400 8:20pm Thu-Mon
Dak Prescott $7800 8:20pm Thu-Mon
Dak Prescott $7800 2:00pm Sun-Mon
Josh Allen $7700 8:20pm Thu-Mon
Josh Allen $7700 2:00pm Main
Josh Allen $7700 2:00pm 1pm Only
Josh Allen $7700 2:00pm Sun-Mon
Josh Allen $8400 2:00pm SuperFlex
Jameis Winston $7500 8:20pm Thu-Mon
Jameis Winston $7500 2:00pm Main
Jameis Winston $7500 2:00pm Sun-Mon
Jameis Winston $8200 2:00pm SuperFlex
Jameis Winston $7500 5:05pm 4pm Only
Jimmy Garoppolo $15500 8:20pm SF @ ARI
Jimmy Garoppolo $7600 8:20pm Thu-Mon
Jacoby Brissett $7300 8:20pm Thu-Mon
Jacoby Brissett $7300 2:00pm Main
Jacoby Brissett $7300 2:00pm 1pm Only
Jacoby Brissett $7300 2:00pm Sun-Mon
Jacoby Brissett $7900 2:00pm SuperFlex
Patrick Mahomes $8500 8:20pm Thu-Mon
Patrick Mahomes $8500 2:00pm Main
Patrick Mahomes $8500 2:00pm 1pm Only
Patrick Mahomes $8500 2:00pm Sun-Mon
Patrick Mahomes $9400 2:00pm SuperFlex
Carson Wentz $7400 8:20pm Thu-Mon
Carson Wentz $7400 2:00pm Main
Carson Wentz $7400 2:00pm 1pm Only
Carson Wentz $7400 2:00pm Sun-Mon
Carson Wentz $8000 2:00pm SuperFlex
Aaron Rodgers $8100 8:20pm Thu-Mon
Aaron Rodgers $8100 2:00pm Main
Aaron Rodgers $8100 2:00pm Sun-Mon
Aaron Rodgers $9000 2:00pm SuperFlex
Aaron Rodgers $8100 5:05pm 4pm Only
Derek Carr $7300 8:20pm Thu-Mon
Derek Carr $7300 2:00pm Main
Derek Carr $7300 2:00pm Sun-Mon
Derek Carr $7900 2:00pm SuperFlex
Derek Carr $7300 5:05pm 4pm Only
Tom Brady $7900 8:20pm Thu-Mon
Tom Brady $7900 2:00pm Sun-Mon
Tom Brady $8700 2:00pm SuperFlex
Kirk Cousins $7800 8:20pm Thu-Mon
Kirk Cousins $7800 2:00pm Main
Kirk Cousins $7800 2:00pm 1pm Only
Kirk Cousins $7800 2:00pm Sun-Mon
Kirk Cousins $8500 2:00pm SuperFlex
Daniel Jones $7300 8:20pm Thu-Mon
Daniel Jones $7300 2:00pm Sun-Mon
Kyle Allen $6800 8:20pm Thu-Mon
Kyle Allen $6800 2:00pm Main
Kyle Allen $6800 2:00pm 1pm Only
Kyle Allen $6800 2:00pm Sun-Mon
Kyle Allen $7200 2:00pm SuperFlex
Gardner Minshew $7200 8:20pm Thu-Mon
Philip Rivers $7200 8:20pm Thu-Mon
Philip Rivers $7200 2:00pm Main
Philip Rivers $7200 2:00pm Sun-Mon
Philip Rivers $7700 2:00pm SuperFlex
Philip Rivers $7200 5:05pm 4pm Only
Mason Rudolph $6800 8:20pm Thu-Mon
Mason Rudolph $6800 2:00pm Main
Mason Rudolph $6800 2:00pm 1pm Only
Mason Rudolph $6800 2:00pm Sun-Mon
Mason Rudolph $7200 2:00pm SuperFlex
Sam Darnold $7300 8:20pm Thu-Mon
Sam Darnold $7300 2:00pm Main
Sam Darnold $7300 2:00pm 1pm Only
Sam Darnold $7300 2:00pm Sun-Mon
Sam Darnold $7800 2:00pm SuperFlex
Matthew Stafford $7900 8:20pm Thu-Mon
Matthew Stafford $7900 2:00pm Main
Matthew Stafford $7900 2:00pm Sun-Mon
Matthew Stafford $8700 2:00pm SuperFlex
Matthew Stafford $7900 5:05pm 4pm Only
Kyler Murray $15000 8:20pm SF @ ARI
Kyler Murray $7200 8:20pm Thu-Mon
Brandon Allen $6000 8:20pm Thu-Mon
Brandon Allen $6000 2:00pm Main
Brandon Allen $6000 2:00pm Sun-Mon
Brandon Allen $6200 2:00pm SuperFlex
Brandon Allen $6000 5:05pm 4pm Only
Ryan Tannehill $7100 8:20pm Thu-Mon
Ryan Tannehill $7100 2:00pm Main
Ryan Tannehill $7100 2:00pm 1pm Only
Ryan Tannehill $7100 2:00pm Sun-Mon
Ryan Tannehill $7500 2:00pm SuperFlex
Baker Mayfield $7200 8:20pm Thu-Mon
Baker Mayfield $7200 2:00pm Main
Baker Mayfield $7200 2:00pm Sun-Mon
Baker Mayfield $7700 2:00pm SuperFlex
Baker Mayfield $7200 5:05pm 4pm Only
Ryan Fitzpatrick $7000 8:20pm Thu-Mon
Ryan Fitzpatrick $7000 2:00pm Main
Ryan Fitzpatrick $7000 2:00pm 1pm Only
Ryan Fitzpatrick $7000 2:00pm Sun-Mon
Ryan Fitzpatrick $7400 2:00pm SuperFlex
Case Keenum $6300 8:20pm Thu-Mon
Case Keenum $6300 2:00pm Main
Case Keenum $6300 2:00pm 1pm Only
Case Keenum $6300 2:00pm Sun-Mon
Case Keenum $6600 2:00pm SuperFlex
推荐阅读
- angular - 我可以使用 carto.js (v4) 和 google maps api (Javascript) 从 carto 编辑图层吗？
- performance - 如果 CPU 是二进制机器，为什么它在位操作上很慢？
- angular - 找不到可监视的对象以进行 navigateBackToLanding
- python - 计算 x1 ^ (x2 ^ (x3 ^ (... ^ xn))) 的最后一位（十进制）
- angular - HTTP 服务调用绑定到 HTML 元素
- node.js - AWS NodeJS Lambda 函数中的缓存内存
- github - !git push:-- 致命：无法读取“https://github.com”的用户名：没有这样的设备或地址||google colaboratory
- javascript - 是否可以使用摊牌解析 Markdown 文件中的标题？
- android - 离子图标不能与 Android 上的离子日期时间一起点击
- javascript - 如何在画布中创建撤消功能？