css - Selenium 抓取 - 使用 CSS 选择器查找元素
问题描述
我正在尝试从 html 表中检索数据，但我定义的 CSS 选择器找不到整个表的元素。整个代码在底部，但问题如下：
for match in driver.find_elements_by_css_selector("div[id='all_games'] td.right")
这会导致 NoSuchElementException。
如果我将其更改为：
for match in driver.find_elements_by_css_selector("div[id='all_games'] tr[data-row='16']")
我从第 16 行收到数据，反对和数据线必须工作。但是，我无法找到合适的 CSS 选择器来返回所有要匹配的数据。
任何想法将不胜感激！
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
url = 'https://www.pro-football-reference.com/teams/nwe/2020.htm'
data = []
driver = webdriver.Chrome('/Users/zachbeaulieu/Downloads/chromedriver')
driver.implicitly_wait(10)
wait = WebDriverWait(driver, 10)
driver.get(url)
# wait for the page to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[id='all_games'] td.right")))
for match in driver.find_elements_by_css_selector("div[id='all_games'] td.right"):
opposition = match.find_element_by_css_selector("td[data-stat~='opp']").text
date = match.find_element_by_css_selector("td[data-stat~='game_date']").text
data.append({
"opposition": opposition.strip(),
"date": date.strip()
})
driver.close()
df = pd.DataFrame(data)
print(df)
解决方案
您为此使用 Selenium 的任何原因？使用 pandas 解析表格要简单、快捷、高效得多。然后，如果您只需要日期和对手列，则可以对其进行切片。
import pandas as pd
import requests
from bs4 import BeautifulSoup, Comment
url = 'https://www.pro-football-reference.com/teams/nwe/2020.htm'
# This will put all the tables not in the comments into a list of tables
tables = df = pd.read_html(url, header=1)
print('Found %s tables in html.' %(len(tables)))
# Now lets get the other tables, and put those into that list
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
comments = soup.find_all(string=lambda text: isinstance(text, Comment))
for each in comments:
if 'table' in str(each):
try:
tables.append(pd.read_html(str(each))[0])
except:
continue
print('Found %s tables in html with additional in comments.' %(len(tables)))
输出：
您最初可以看到您得到 3 个表，但随后它会拉取在 html 的注释中找到的其他 8 个表。
Found 3 tables in html.
Found 11 tables in html with additional in comments.
所有 11 个表：
for df in tables:
print(df)
Player PF Yds Ply Y/P ... Start Time Plays Yds.4 Pts
0 Team Stats 326 5236 979.0 5.3 ... Own 28.5 2:57 6.41 33.1 1.92
1 Opp. Stats 353 5660 982.0 5.8 ... Own 24.6 3:10 6.40 35.9 2.16
2 Lg Rank Offense 27 27 NaN NaN ... 15 7 5.00 17.0 24.00
3 Lg Rank Defense 7 15 NaN NaN ... 1 32 24.00 25.0 16.00
[4 rows x 31 columns]
Week Day Date Unnamed: 3 ... TO.1 Offense Defense Sp. Tms
0 1 Sun September 13 1:00PM ET ... 3.0 14.11 2.20 -5.64
1 2 Sun September 20 8:20PM ET ... 1.0 19.82 -14.54 -6.98
2 3 Sun September 27 1:00PM ET ... 3.0 11.03 -0.71 5.81
3 4 Mon October 5 7:05PM ET ... 1.0 -10.82 -5.72 3.78
4 5 NaN NaN NaN ... NaN NaN NaN NaN
5 6 Sun October 18 1:00PM ET ... 2.0 -13.30 6.67 1.59
6 7 Sun October 25 4:25PM ET ... 2.0 -9.49 -22.44 6.55
7 8 Sun November 1 1:00PM ET ... 1.0 8.26 -8.85 -0.85
8 9 Mon November 9 8:15PM ET ... 1.0 18.70 -18.71 2.16
9 10 Sun November 15 8:20PM ET ... 1.0 13.95 -3.45 -1.86
10 11 Sun November 22 1:00PM ET ... NaN 13.67 -15.69 -0.27
11 12 Sun November 29 1:00PM ET ... 1.0 -4.68 0.00 9.06
12 13 Sun December 6 4:25PM ET ... 2.0 9.66 16.13 22.19
13 14 Thu December 10 8:20PM ET ... 1.0 -29.77 -1.19 8.58
14 15 Sun December 20 1:00PM ET ... 1.0 -1.00 -16.71 6.16
15 16 Mon December 28 8:15PM ET ... NaN -2.39 -26.17 1.10
16 17 Sun January 3 1:00PM ET ... 2.0 18.13 -3.92 -0.05
[17 rows x 25 columns]
Player 3DAtt 3DConv 3D% ... 4D% RZAtt RZTD RZPct
0 Team Stats 186.0 76.0 40.9 ... 52.9 48.0 26.0 54.2
1 Opp. Stats 186.0 76.0 40.9 ... 56.3 49.0 32.0 65.3
2 Lg Rank Offense NaN NaN 17.0 ... 22.0 NaN NaN 24.0
3 Lg Rank Defense NaN NaN 16.0 ... 16.0 NaN NaN 27.0
[4 rows x 10 columns]
No. Player Age Pos G ... NY/A ANY/A Sk% 4QC GWD
0 1.0 Cam Newton 31.0 QB 15 ... 6.17 5.44 7.8 1.0 3.0
1 4.0 Jarrett Stidham 24.0 NaN 5 ... 4.90 2.92 8.3 NaN NaN
2 2.0 Brian Hoyer 35.0 qb 1 ... 4.31 2.58 7.7 NaN NaN
3 11.0 Julian Edelman 34.0 wr 6 ... 19.00 19.00 0.0 NaN NaN
4 16.0 Jakobi Meyers 24.0 wr 14 ... 21.50 41.50 0.0 NaN NaN
5 NaN Team Total 26.5 NaN 16 ... 6.06 5.24 7.8 1.0 3.0
6 NaN Opp Total NaN NaN 16 ... 6.90 6.20 4.6 NaN NaN
[7 rows x 29 columns]
Unnamed: 0_level_0 ... Unnamed: 27_level_0
No. ... Fmb
0 37.0 ... 1
1 1.0 ... 6
2 26.0 ... 0
3 34.0 ... 0
4 28.0 ... 1
5 42.0 ... 0
6 4.0 ... 0
7 80.0 ... 1
8 16.0 ... 1
9 10.0 ... 3
10 15.0 ... 1
11 11.0 ... 0
12 19.0 ... 0
13 14.0 ... 0
14 2.0 ... 1
15 85.0 ... 1
16 47.0 ... 1
17 44.0 ... 1
18 86.0 ... 0
19 NaN ... 19
20 NaN ... 11
[21 rows x 28 columns]
Unnamed: 0_level_0 Unnamed: 1_level_0 ... Kick Returns Unnamed: 16_level_0
No. Player ... Y/Rt APYd
0 80.0 Gunner Olszewski+ ... 23.2 849.0
1 14.0 Donte Moncrief ... 23.6 184.0
2 42.0 J.J. Taylor ... 21.8 212.0
3 10.0 Damiere Byrd ... NaN 619.0
4 35.0 Kyle Dugger ... 23.5 47.0
5 NaN Team Total ... 23.1 6672.0
6 NaN Opp Total ... 21.3 NaN
[7 rows x 17 columns]
Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0 ... Punting
No. Player Age ... Lng Blck Y/P
0 6.0 Nick Folk 36.0 ... NaN NaN NaN
1 7.0 Jake Bailey*+ 23.0 ... 71.0 0.0 48.7
2 NaN Team Total 26.5 ... 71.0 0.0 48.7
3 NaN Opp Total NaN ... NaN 0.0 46.0
[4 rows x 32 columns]
Unnamed: 0_level_0 Unnamed: 1_level_0 ... Tackles Unnamed: 22_level_0
No. Player ... QBHits Sfty
0 21.0 Adrian Phillips ... 1.0 NaN
1 51.0 Ja'Whaun Bentley ... 3.0 NaN
2 31.0 Jonathan Jones ... 1.0 NaN
3 32.0 Devin McCourty ... 0.0 NaN
4 35.0 Kyle Dugger ... 0.0 NaN
5 93.0 Lawrence Guy ... 7.0 NaN
6 55.0 John Simon ... 4.0 NaN
7 50.0 Chase Winovich ... 12.0 NaN
8 27.0 J.C. Jackson ... 0.0 NaN
9 59.0 Terez Hall ... 0.0 NaN
10 91.0 Deatrich Wise Jr. ... 11.0 NaN
11 30.0 Jason McCourty ... 0.0 NaN
12 24.0 Stephon Gilmore* ... 0.0 NaN
13 70.0 Adam Butler ... 7.0 NaN
14 99.0 Byron Cowart ... 3.0 NaN
15 25.0 Terrence Brooks ... 1.0 NaN
16 90.0 Shilique Calhoun ... 3.0 NaN
17 58.0 Anfernee Jennings ... 1.0 NaN
18 33.0 Joejuan Williams ... 0.0 NaN
19 29.0 Justin Bethel ... 0.0 NaN
20 41.0 Myles Bryant ... 0.0 NaN
21 96.0 Tashawn Bower ... 0.0 NaN
22 52.0 Brandon Copeland ... 0.0 NaN
23 53.0 Josh Uche ... 7.0 NaN
24 92.0 Nick Thurman ... 0.0 NaN
25 22.0 Cody Davis ... 0.0 NaN
26 52.0 Akeem Spence ... 0.0 NaN
27 18.0 Matthew Slater* ... 0.0 NaN
28 1.0 Cam Newton ... NaN NaN
29 NaN Derek Rivers ... 4.0 NaN
30 98.0 Carl Davis ... 0.0 NaN
31 80.0 Gunner Olszewski+ ... 0.0 NaN
32 7.0 Jake Bailey*+ ... 0.0 NaN
33 10.0 Damiere Byrd ... NaN NaN
34 37.0 Damien Harris ... NaN NaN
35 NaN Xavier Williams ... 0.0 NaN
36 43.0 Rashod Berry ... 0.0 NaN
37 28.0 Michael Jackson ... 0.0 NaN
38 71.0 Michael Onwenu ... NaN NaN
39 28.0 James White ... NaN NaN
40 66.0 James Ferentz ... NaN NaN
41 15.0 N'Keal Harry ... NaN NaN
42 2.0 Brian Hoyer ... NaN NaN
43 85.0 Ryan Izzo ... NaN NaN
44 47.0 Jakob Johnson ... NaN NaN
45 44.0 Dalton Keene ... NaN NaN
46 94.0 Isaiah Mack ... 0.0 NaN
47 69.0 Shaq Mason ... NaN NaN
48 16.0 Jakobi Meyers ... NaN NaN
49 76.0 Isaiah Wynn ... NaN NaN
50 NaN Team Total ... 65.0 NaN
51 NaN Opp Total ... NaN NaN
[52 rows x 23 columns]
No. Player Age Pos G ... FGM FGA Sfty Pts Pts/G
0 6.0 Nick Folk 36.0 k 16 ... 26.0 28.0 NaN 108 6.8
1 1.0 Cam Newton 31.0 QB 15 ... NaN NaN NaN 78 5.2
2 34.0 Rex Burkhead 30.0 NaN 10 ... NaN NaN NaN 36 3.6
3 28.0 James White 28.0 NaN 14 ... NaN NaN NaN 18 1.3
4 37.0 Damien Harris 23.0 rb 10 ... NaN NaN NaN 12 1.2
5 15.0 N'Keal Harry 23.0 wr 14 ... NaN NaN NaN 12 0.9
6 32.0 Devin McCourty 33.0 FS 16 ... NaN NaN NaN 12 0.8
7 26.0 Sony Michel 25.0 rb 9 ... NaN NaN NaN 12 1.3
8 80.0 Gunner Olszewski+ 24.0 wr 13 ... NaN NaN NaN 12 0.9
9 86.0 Devin Asiasi 23.0 te 9 ... NaN NaN NaN 6 0.7
10 10.0 Damiere Byrd 27.0 WR 16 ... NaN NaN NaN 6 0.4
11 47.0 Jakob Johnson 26.0 FB 16 ... NaN NaN NaN 6 0.4
12 91.0 Deatrich Wise Jr. 26.0 de 16 ... NaN NaN NaN 6 0.4
13 16.0 Jakobi Meyers 24.0 wr 14 ... NaN NaN NaN 2 0.1
14 NaN Team Total 26.5 NaN 16 ... 26.0 28.0 NaN 326 NaN
15 NaN Opp Total NaN NaN 16 ... 22.0 27.0 NaN 353 NaN
[16 rows x 23 columns]
Rk ... Detail
0 1 ... Cam Newton 4 yard rush (Nick Folk kick)
1 2 ... Cam Newton 11 yard rush (Nick Folk kick)
2 3 ... Sony Michel 1 yard rush (Nick Folk kick)
3 4 ... Devin McCourty 43 yard interception return (Ni...
4 5 ... Cam Newton 1 yard rush (Nick Folk kick)
5 6 ... Jakob Johnson 1 yard pass from Cam Newton (run...
6 7 ... Cam Newton 1 yard rush (Nick Folk kick)
7 8 ... Rex Burkhead 11 yard pass from Cam Newton (Nic...
8 9 ... Rex Burkhead 5 yard rush (Nick Folk kick)
9 10 ... Rex Burkhead 2 yard rush (Nick Folk kick failed)
10 11 ... Deatrich Wise Jr. 0 yard fumble return (Nick F...
11 12 ... N'Keal Harry 4 yard pass from Jarrett Stidham ...
12 13 ... Cam Newton 1 yard rush (run failed)
13 14 ... Damien Harris 22 yard rush (Jakobi Meyers pass...
14 15 ... Cam Newton 2 yard rush (Nick Folk kick)
15 16 ... Cam Newton 5 yard rush (Nick Folk kick)
16 17 ... Rex Burkhead 1 yard rush (Nick Folk kick)
17 18 ... Cam Newton 1 yard rush (Nick Folk kick)
18 19 ... Rex Burkhead 7 yard pass from Cam Newton (Nick...
19 20 ... Rex Burkhead 24 yard pass from Jakobi Meyers (...
20 21 ... Cam Newton 4 yard rush (Nick Folk kick)
21 22 ... Damien Harris 9 yard rush (Nick Folk kick)
22 23 ... Damiere Byrd 42 yard pass from Cam Newton (Nic...
23 24 ... James White 7 yard rush (Nick Folk kick)
24 25 ... James White 1 yard rush (Nick Folk kick)
25 26 ... Cam Newton 1 yard rush (Nick Folk kick)
26 27 ... Gunner Olszewski 70 yard punt return (Nick Fol...
27 28 ... Cam Newton 2 yard rush (Nick Folk kick)
28 29 ... Devin McCourty 44 yard blocked field goal retu...
29 30 ... N'Keal Harry 5 yard pass from Cam Newton (Nick...
30 31 ... Gunner Olszewski 38 yard pass from Jarrett Sti...
31 32 ... Cam Newton 9 yard rush (Nick Folk kick failed)
32 33 ... James White 7 yard pass from Cam Newton (Nick ...
33 34 ... Cam Newton 19 yard pass from Jakobi Meyers (Ni...
34 35 ... Devin Asiasi 26 yard pass from Cam Newton (Nic...
35 36 ... Sony Michel 31 yard pass from Cam Newton (Nick...
[36 rows x 9 columns]
Rk ... Detail
0 1 ... Jordan Howard 1 yard rush (Ryan Fitzpatrick run)
1 2 ... Tyler Lockett 4 yard pass from Russell Wilson ...
2 3 ... D.K. Metcalf 54 yard pass from Russell Wilson ...
3 4 ... David Moore 38 yard pass from Russell Wilson (...
4 5 ... Freddie Swain 21 yard pass from Russell Wilson...
5 6 ... Chris Carson 18 yard pass from Russell Wilson ...
6 7 ... Foster Moreau 1 yard pass from Derek Carr (Dan...
7 8 ... Hunter Renfrow 13 yard pass from Derek Carr (D...
8 9 ... Tyreek Hill 6 yard pass from Patrick Mahomes (...
9 10 ... Mecole Hardman 6 yard pass from Patrick Mahome...
10 11 ... Tyrann Mathieu 25 yard interception return (Ha...
11 12 ... Jeff Wilson 3 yard rush (Robbie Gould kick)
12 13 ... Kyle Juszczyk 4 yard rush (Robbie Gould kick f...
13 14 ... Jeff Wilson 16 yard rush (Robbie Gould kick)
14 15 ... Jeff Wilson 7 yard rush (Robbie Gould kick)
15 16 ... Zack Moss 8 yard rush (Tyler Bass kick)
16 17 ... Zack Moss 4 yard rush (Tyler Bass kick)
17 18 ... Josh Allen 2 yard rush (Tyler Bass kick)
18 19 ... Breshad Perriman 50 yard pass from Joe Flacco ...
19 20 ... Jamison Crowder 20 yard pass from Joe Flacco (...
20 21 ... Breshad Perriman 15 yard pass from Joe Flacco ...
21 22 ... Willie Snead 6 yard pass from Lamar Jackson (J...
22 23 ... Willie Snead 18 yard pass from Lamar Jackson (...
23 24 ... Randall Cobb 3 yard pass from Deshaun Watson (...
24 25 ... Deshaun Watson 4 yard rush (Ka'imi Fairbairn k...
25 26 ... Keke Coutee 6 yard pass from Deshaun Watson (K...
26 27 ... Kenyan Drake 1 yard rush (Zane Gonzalez kick)
27 28 ... Kenyan Drake 1 yard rush (Zane Gonzalez kick)
28 29 ... Jared Goff 1 yard rush (Matt Gay kick)
29 30 ... Kenny Young 79 yard interception return (Matt ...
30 31 ... Cooper Kupp 2 yard pass from Jared Goff (Matt ...
31 32 ... Salvon Ahmed 1 yard rush (Jason Sanders kick)
32 33 ... Tua Tagovailoa 3 yard rush (Salvon Ahmed pass ...
33 34 ... Tua Tagovailoa 1 yard rush (Jason Sanders kick)
34 35 ... Zack Moss 5 yard rush (Tyler Bass kick)
35 36 ... Lee Smith 4 yard pass from Josh Allen (Tyler B...
36 37 ... Stefon Diggs 50 yard pass from Josh Allen (Tyl...
37 38 ... Stefon Diggs 18 yard pass from Josh Allen (Tyl...
38 39 ... Stefon Diggs 8 yard pass from Josh Allen (Tyle...
39 40 ... Chris Herndon 21 yard pass from Sam Darnold (C...
40 41 ... Josh Adams 1 yard rush (Chase McLaughlin kick)
[41 rows x 9 columns]
推荐阅读
- php - 在 PHP Mysqli 中使用 WHERE 子句时，有时结果不显示
- django - 如何在覆盖的 save() 方法中从 ManyToManyField 写入和检索数据？
- sql - 如何在MS Access中检索尚未放入交叉表的数据？
- php - PHP中的按钮编程
- kotlin - 如何为文件添加顶级 KDoc？
- oracle-apex - 我如何通过动态操作将文件插入到顶点集合表中的 blob001 列中
- c# - 是否可以在正则表达式中放置一个字符串变量？
- php - PHPUnit 错误 - 无法使用 MAMP 服务器从命令行运行 Laravel 测试
- python - 在 python 项目上一起工作的最佳地点？
- php - 在准备好的语句中传递一个文字变量和占位符？