时间戳

首页 > 解决方案 > 网页抓取新手

python - 网页抓取新手

问题描述

试图自学一些网络抓取，只是为了好玩。决定用它来查看网站上发布的工作列表。我被卡住了。我希望能够提取此页面上列出的所有作业，但似乎无法让它识别我制作的容器中更深层次的任何内容。任何建议都值得赞赏。

当前代码：

import bs4
from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

myURL = 'https://jobs.collinsaerospace.com/search-jobs/'

uClient = uReq(myURL)
page_html = uClient.read()
uClient.close()

page_soup = soup(page_html, "html.parser")
container = page_soup.findAll("section", {"id":"search-results-list"})
container

容器样本：

<section id="search-results-list">
 <ul>
 <li>
 <a data-job-id="12394447" href="/job/melbourne/test-technician/1738/12394447">
 <h2>Test Technician</h2>
 <span class="job-location">Melbourne, Florida</span>
 <span class="job-date-posted">06/27/2019</span>
 </a>
 </li>
 <li>
 <a data-job-id="12394445" href="/job/cedar-rapids/associate-systems-engineer/1738/12394445">
 <h2>Associate Systems Engineer</h2>
 <span class="job-location">Cedar Rapids, Iowa</span>
 <span class="job-date-posted">06/27/2019</span>
 </a>
 </li>
 <li>

我正在尝试了解如何实际提取 h2 级别信息（或实际上是我当前创建的容器中的任何信息）

标签： pythonweb-scrapingbeautifulsoup

解决方案

我尝试使用 lxml 复制相同的内容。

import requests
from lxml import html
resp = requests.get('https://jobs.collinsaerospace.com/search-jobs/')
data_root = html.fromstring(resp.content)

data = []
for node in data_root.xpath('//section[@id="search-results-list"]/ul/li'):
    data.append({"url":node.xpath('a/@href')[0],"name":node.xpath('a/h2/text()')[0],"location":node.xpath('a/span[@class="job-location"]/text()')[0],"posted":node.xpath('a/span[@class="job-date-posted"]/text()')[0]})
print(data)

推荐阅读