时间戳

首页 > 解决方案 > 从网站抓取表格（无法找到表格或属性）

python - 从网站抓取表格（无法找到表格或属性）

问题描述

我正在尝试在http://tickertrak.com/上抓取表格，但我无法做到。代码在 table 标记之后没有读取任何内容，我什至看不到它，所以我非常困惑。我是网络抓取的新手，到目前为止只能做维基百科表格。

import time
!pip install selenium
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get("http://tickertrak.com/")
time.sleep(2)

df = pd.read_html(driver.page_source, flavor="bs4")
df = pd.concat(df)
df.drop(index=0, axis=0, inplace=True)
df.to_csv("your_table.csv", index=False)`

我收到以下错误：

driver = webdriver.Chrome(options=options)

对于 webdriver 异常。我是不是忘记了某处的路径？

标签： pythonweb-scrapingbeautifulsoup

解决方案

这是使用请求模块从该网页获取数据的更快方法之一，因为数据已经在脚本标记内的页面源中。您现在要做的就是在将数据存储到数据框之前清理数据。

import re
import requests

URL = 'http://tickertrak.com/'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
    r = s.get(URL)
    items = re.findall(r"var arrayFromPHP = \[(.*?)\];",r.text)[0]
    trs = re.findall(r"\[(.*?)\]",items)
    for tds in trs:
        print(tds)

输出如下：

"Options","gamestop corp","gme","1","58662","131","-80","-85","1"
"Options","amc entertainment holdings inc","AMC","1","16290","36","-79","-66","2"
"Options","nokia corp","nok","1","3568","14","-86","-88","3"
"Options","regal-beloit corp","RBC","1","3254","11","-56","-89","4"
"Options","blackberry ltd","BB","1","3002","10","-91","-92","5"

推荐阅读