时间戳

首页 > 解决方案 > 如何在没有评论的情况下返回 BeautifulSoup find_all 结果

python-3.x - 如何在没有评论的情况下返回 BeautifulSoup find_all 结果

问题描述

我正在尝试根据 div id 匹配返回 BeautifulSoup 搜索结果，但第一个（也是唯一一个）结果是给我没有任何数据的 HTML 结构。我想返回结构和数据，所以我可以继续我的教程/解析并最终提取单个数据元素（例如，搜索结果的名称、搜索结果的描述等）。

背景：这是我正在努力学习 BeautifulSoup 的练习的一部分，非常新。看过 BS 文档，但鉴于我的知识水平，没有发现任何对我有帮助的东西。

使用本教程（https://www.dataquest.io/blog/web-scraping-tutorial-python/），我尝试编写一段相当简单的 Python 代码来捕获此页面上的搜索结果：http:// /www.ncsl.org/searchresults/issearch/false/kwdid/463.aspx。我正在尝试使用 BeautifulSoup 来收集每个搜索结果的各种元素。

代码片段应该不言自明，但基本上我试图：

相反，我得到：

（例如，有一个“text:title”片段，但后面没有标题的实际文本（应该是“2019 State Legislative Races and Ballot Measures”）。）

我一直在使用的代码包含在下面。

ncsl_statevote = requests.get("http://www.ncsl.org/searchresults/issearch/false/kwdid/463.aspx")
soup = BeautifulSoup(ncsl_statevote.content, 'html.parser')
table_id = soup.find(id="search-results")

在这一点上，我期望 HTML 页面上所有搜索结果元素的“列表”。相反，我得到了我之前在问题中引用的代码片段。

假设上面的代码实际上提取了正确的内容，我希望以下内容可以进一步将我的目标搜索结果解析为必要的元素：

table_element1 = table_id.find_all(class_="article")
first_searchresult = table_element1[0]
result_excerpt = firstsearchresult.find(class_="result-    excerpt").get_text()
result_content_type = firstsearchresult.find(class_="result-contenttype grey").get_text()

我还应该补充一点，理想情况下，我也可以提取搜索结果的名称（即当前在<a data-bind="..........">标签中捕获的信息，但是这部分 HTML 的结构不允许我遵循上面的方法。我认为这是一个单独的问题，所以我可以尝试在这个问题之外解决这个问题。

我希望得到一个搜索结果列表，之后table_id = soup.find(id="search-results")我可以进一步解析为result_excerpt,result_content_type等元素。

相反，我得到了一段没有数据的代码，看起来就像这样：

    <div class="article">
     <h2>
      <a data-bind="attr: {href: link }">
       <span data-bind="    text:title">
       </span>
      </a>
     </h2>
     <p class="result-excerpt">
      <span data-bind="text:excerpt">
      </span>
     </p>
     <p class="result-contenttype grey">
      <span data-bind="text: portalName +': ' +  contentType + '  | ' +         displayDate" style="font-style: italic">
      </span>
     </p>
    </div>

标签： python-3.xweb-scrapingbeautifulsoup

解决方案

如果您转到网络选项卡，您将获得以下 URL，该 URL 以 json 格式返回输出。

http://www.ncsl.org/DesktopModules/NCSLWebApi/API/ASR/GetASRList?SortBy=date&QueryString=&TopicId=463&IsSearch=False&SpecialCol=Standard&NCSLPortalId=724&_=1571685649582

代码在这里获取瓷砖和dseciption。

import requests
import json
res=requests.get("http://www.ncsl.org/DesktopModules/NCSLWebApi/API/ASR/GetASRList?SortBy=date&QueryString=&TopicId=463&IsSearch=False&SpecialCol=Standard&NCSLPortalId=724&_=1571685649582").text
sjson=json.loads(res)
for item in sjson:
    print(item['Title'])
    print(item['Excerpt'])
    print(item['DisplayDate'])
    print(item['PortalName'])
    print(item['ContentType'])

输出：

2019 State Legislative Races and Ballot Measures
This page includes information about 2019 state legislative raises and ballot measures, along with information about state control. 
September 03, 2019
Elections & Campaigns
Reports
State Election Results | State Election Analysis Overview
Summaries and compilations of election results for each state. Analysis and information about statewide elections, legislative party composition.
May 01, 2019
Elections & Campaigns
Articles/Briefs/Newsletters
Election 2018: Women Ran, Women Won
The 2018 election was a big one for female candidates in state legislative races. Approximately 3,564 female candidates ran for state legislative seats in the midterm elections (Democrats, Republicans and third-party members), which is a whopping 28 percent increase compared with the 2,781 women who ran two years ago.
November 20, 2018
Elections & Campaigns
State Legislatures Magazine
StateVote 2018 | State Legislative Races and Ballot Measures
This page contains key information about the 2018 state legislative elections and statewide ballot measures. 
November 09, 2018
Elections & Campaigns
Articles/Briefs/Newsletters
A Wave or a Wash? Midterm Outcomes Will Depend on How Voters Feel About Trump
The strong economy could help Trump's party, but his low poll numbers could boost the Democrats in hundreds of legislative races.
October 31, 2018
Elections & Campaigns
State Legislatures Magazine
Something for Everyone: The Ballot Measures Awaiting Voters in November
In the November midterm elections, voters will weigh in on more than 150 ballot measures, including some big issues like school funding, taxes, redistricting and Medicaid.
October 31, 2018
Elections & Campaigns
State Legislatures Magazine
NCSL StateVote 2018 Post-Election Briefing
This page includes information about an NCSL post-election briefing on Nov. 9, 2018, at the National Press Club, Washington, D.C. 
July 23, 2018
Elections & Campaigns
Meetings
2018 Legislative Races by State and Legislative Chamber
In November 2018, legislative races in 46 states (88 legislative chambers) will be held. The table below indicates which states and chambers are holding legislative elections, and how many seats are up for re-election this year. A total of 6,066 seats are up for regularly scheduled elections.
February 23, 2018
Elections & Campaigns
Bill Summaries/Databases
StateVote: 2017 Elections
This page contains post-election analysis of the 2017 legislative races in New Jersey and Virginia, and analyses of statewide ballot measures planned for seven states. 
November 08, 2017
Elections & Campaigns
Articles/Briefs/Newsletters
2016 District-by-District State Legislative Control
This page shows district-by-district control of state Senates and state lower houses. These maps are a joint project between NCSL and Quorum. 
December 07, 2016
Elections & Campaigns
Articles/Briefs/Newsletters
[Interactive Maps] StateVote 2016 | State Legislative Races and Statewide Ballot Measures
This NCSL page contains key information about 2016 elections for state legislatures, statewide ballot measures, gubernatorial races and state government control.
December 06, 2016
Elections & Campaigns
Multimedia
2016 Pre- and Post-Election State Legislative Control
This page contains maps showing 2016 pre- and post-election control of legislative chambers.
November 09, 2016
Elections & Campaigns
Articles/Briefs/Newsletters
What Does the Election Mean for the States?
- Join the National Conference of State Legislatures (NCSL) StateVote elections in cooperation with The Hill to hear from analysts and national political experts for a post-election discussion, Monday, Nov. 14, 1-5 p.m., The National Press Club, 529 14th St. NW, 13th Floor, Washington, D.C., about what the outcome will mean for states, and what to expect in the upcoming legislative sessions.
November 02, 2016
Elections & Campaigns
News Release
NCSL StateVote 2016 Post-Election Briefing | Nov. 14, 2016
Find out more about the NCSL 2016 post-election briefing on Nov. 14 at the National Press Club in Washington, D.C., on state legislative elections, statewide ballot measures and the forecast for state-federal relations in a new administration. 
September 08, 2016
Elections & Campaigns
Meetings
Helping Constituents Vote
Answers to common questions about how, where and when to vote.
May 17, 2016
Elections & Campaigns
Articles/Briefs/Newsletters
2016 Legislative Races by State and Legislative Chamber
This page contains information about 2016 Legislative races by state and legislative chamber.
March 11, 2016
Elections & Campaigns
Reports
StateVote: 2015 State Elections
This year legislative seats are up in four states—Louisiana, Mississippi, New Jersey and Virginia. There are governors races in three states—Kentucky, Louisiana and Mississippi—and just 26 statewide ballot measures in seven states. Pre- and post-election analysis for 2015 legislative elections and ballot  measures are found here
November 04, 2015
Elections & Campaigns
Articles/Briefs/Newsletters
StateVote 2010: Initiatives and Referenda
Ballot measure news from 2010
April 06, 2015
Elections & Campaigns
Articles/Briefs/Newsletters
Publications from NCSL’s Elections Team
Publications from NCSL's Elections Team from 2008-2014, including links to documents on voter ID, voter registration, Election Day registration, elections technology and more.
February 23, 2015
Elections & Campaigns
Articles/Briefs/Newsletters
Scaling New Heights
Republicans reached historic highs following the 2014 state legislative elections. In this article, NCSL elections expert Tim Storey breaks down the results. .
December 01, 2014
Elections & Campaigns
State Legislatures Magazine
Measure by Measure
Voters gave an enthusiastic thumbs-up to many ballot measures, from marijuana legalization to minimum wage hikes. In this article, NCSL's Wendy Underhill analyzes the results of the 2014 election on statewide ballot measures. 
December 01, 2014
Elections & Campaigns
State Legislatures Magazine
2014 Post-Election State Legislative Seat Turnover
This page has a table showing the turnover (1,325 seats) to newly elected members after the 2014 election. The table displays the number and percentage of seats that were filled by new legislators.
December 01, 2014
Elections & Campaigns
Articles/Briefs/Newsletters
StateVote 2014: Election Results
Post-election analysis of state legislative races, including partisan control of legislative chambers and turnover, and statewide ballot measures from the 2014 General Election.
November 19, 2014
Elections & Campaigns
Reports
Blog Posts | 2014 Statewide Ballot Measures
Links to blogs on various state ballot measures.
November 06, 2014
Elections & Campaigns
Articles/Briefs/Newsletters
StateVote 2014: Pre-Election Analysis
This page contains information about the 2014 state legislative elections and statewide ballot measures, including partisan control of legislatures, turnover and -re-election analysis. 
November 06, 2014
Elections & Campaigns
Articles/Briefs/Newsletters
[Interactive Map] StateVote 2014 | Interactive Map 
This page contains an interactive map that shows 2014 partisan control in state legislatures.
November 06, 2014
Elections & Campaigns
Multimedia
[Interactive Map] 2014 Election Governors Party Control
This page contains an interactive map showing gubernatorial control of states in the 2014 election.
November 05, 2014
Elections & Campaigns
Multimedia
[Interactive map] 2014 Governors Party Pre-Election
Governors Party Prior to the 2014 Election
October 31, 2014
Elections & Campaigns
Multimedia
2014 Election  Legislatures Party Control Switch
Legislatures: Party Control Switch Map
October 24, 2014
Elections & Campaigns
Reports
2014 State Legislative Races by State and Legislative Chamber
In 2014, state legislative elections take place in 46 states.  This table gives the number of seats up for re-election and in which chambers. The results of the 2014 legislative races will determine party control for legislatures.
October 09, 2014
Elections & Campaigns
Articles/Briefs/Newsletters
2014 Calendar | State Primary Dates and Runoff Dates 
Most states hold primaries between June and August in even years, such as 2014. The list of dates for primaries, and in some states for primary runoffs, is presented here, along with background information and links to related sources.
February 03, 2014
Elections & Campaigns
Bill Summaries/Databases
STATES HOLD STEADY
Read this State Legislatures Magazine story to learn about a quiet off-year election for state legislative seats with no surprises and only a couple of ballot measures worth noting.
December 01, 2013
Elections & Campaigns
State Legislatures Magazine
StateVote: 2013 Elections
Pre- and post-election analysis for 2013 legislative elections and ballot  measures are found here. Two states have legislative races, and six states have statewide ballot measures.
November 06, 2013
Elections & Campaigns
Articles/Briefs/Newsletters
Democrats Bounce Back 1
Read this story to find out how state legislative control fared in the 2012 election. After getting shellacked by Republicans two years ago, Democrats rebounded in the 2012 election, gaining about 150 seats and taking back eight chambers they lost in 2010.
December 04, 2012
Elections & Campaigns
State Legislatures Magazine
Social Issues Score Big: December 2012
Read this story to learn how voters responded to several social issues on state ballots in 2012. Same-sex marriage and the legalization of recreational use of marijuana were among the topics.
December 01, 2012
Elections & Campaigns
State Legislatures Magazine
Statevote 2012 
Visit NCSL's StateVote articles to learn about the 2012 legislative elections in which the seats of more than 6,000 state legislators were up for elections. 
November 06, 2012
Elections & Campaigns
Articles/Briefs/Newsletters
Everything You Need to Know About 2012 State Elections 
Read this page for NCSL's guide to resources related to the 2012 election for state legislative seats and ballot measures. 
November 01, 2012
Elections & Campaigns
Articles/Briefs/Newsletters
Statevote: Pre-Election Governors Map
This interactive, 50-state map shows gubernatorial control following the November 2012 election. 
October 25, 2012
Elections & Campaigns
Multimedia
Statevote: Live 2012 Election Night Results for Governors
This interactive 50-statemap shows which party controls the governor's office before the November 2012 election. 
October 25, 2012
Elections & Campaigns
Multimedia
2012 Ballot Measures | Election Results 
Read this NCSL StateVote article to learn more about the 37 states that are considering 172 statewide ballot issues on the Nov. 6, 2012, ballot. There are 42 citizen initiatives, 12 popular referenda 113 measures referred to the ballot by state legislatures, and five measures that fall into unique categories. You can also learn about election results after the Nov. 6, 2012 election. 
October 25, 2012
Elections & Campaigns
Articles/Briefs/Newsletters
Statevote Party Comp Pre Election Maps
This interactive map shows the partisan control of state legislatures before the 2012 election. 
October 24, 2012
Elections & Campaigns
Multimedia
Statevote 2012 Election Night Results Map
This map was updated on Election Day 2012 to show party control of state legislatures.
October 24, 2012
Elections & Campaigns
Multimedia
Partisan Control Switch in the 2012 Election
This interactive 50-state map will show legislatures that have switched control following the 2012 election. 
October 24, 2012
Elections & Campaigns
Multimedia
Statevote State Control 2012
This 50-state map will be updated on and after Election Day 2012 to indicate partisan control of state legislatures and governor's offices.
October 24, 2012
Elections & Campaigns
Multimedia
StateVote Legislative Elections
Read this NCSL StateVote article to learn more about the 2012 legislative elections in which more than 6,000 state legislative seats are up for elections that could change the balance of power between Republicans and Democrats in as many as 20 chambers on Election Day, Nov. 6, 2012. 
October 18, 2012
Elections & Campaigns
Articles/Briefs/Newsletters
StateVote 2010 Archive
StateVote 2010 Archive
October 09, 2012
Elections & Campaigns
Articles/Briefs/Newsletters
2012 State and Legislative Partisan Composition 
2012 State and Legislative Partisan Composition 
June 06, 2012
Elections & Campaigns
Reports
StateVote 2011
StateVote 2011
November 09, 2011
Elections & Campaigns
Multimedia
StateVote 2011 Pre-Election Map
Pre-Election 2011 partisan composition map
November 04, 2011
Elections & Campaigns
Multimedia
Red Tide
A GOP wave washed over state legislatures on Election Day.
November 12, 2010
Elections & Campaigns
State Legislatures Magazine
Voters Take Stands on Budget Issues Across America
Results could complicate efforts to balance budgets in 2011
November 03, 2010
Elections & Campaigns
News Release
Republicans Exceed Expectations in 2010 State Legislative Elections
The 2010 election will shape the national political landscape for at least the next 10 years.  
November 03, 2010
Elections & Campaigns
News Release
StateVote 2010
This is the main landing page for NCSL's StateVote.
October 28, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
Ballot Measures: News
News on 2010 ballot measures
October 27, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
Pre-Election Legislature Party Control
Chart of pre-election partisan composition of state legislatures and post-election results from the Nov. 2, 2010, election. 
October 26, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
StateVote: General Information
This page includes general information about NCSL's StateVote project and links to a variety of resources.
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
Pre-Election Party Control of Legislatures
This map shows pre-election party control of state Legislatures before the Nov. 2, 2010 election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 Pre-Election Government Control
This map shows control of the legislature and governor's office prior to the Nov. 2, 2010 election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 Pre-Election State Governor
This map shows gubernatorial control before the Nov. 2, 2010, election.  
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 Post-Election Control of Legislatures
This map shows control of state legislatures following the Nov. 2, 2010, election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 Legislature Party Control Switch
This map shows legislatures that changed party control as a result of the Nov. 2, 2010, election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 Post-Election State Government Control
This map shows the control of state government--the legislature and the governor's office--following the Nov. 2, 2010, election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 Post-Election State Governor
This map shows the party controlling the governor's office following the Nov. 2, 2010, election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
Legislatures Partisan Composition Chart
A chart showing party control of state legislatures following the Nov. 2, 2010, election. 
September 14, 2010
Elections & Campaigns
Articles/Briefs/Newsletters
2010 State Legislative Party Composition
2010 State Legislative Party Composition
January 06, 2010
Elections & Campaigns
Reports
SL Magazine: Trading Places
Obama Leads State Lawmakers Headed to Washington
December 01, 2009
Elections & Campaigns
State Legislatures Magazine
SL Magazine: Election 2008 - Making History
Democrats pump up their control of state legislatures, but Republicans also make gains.
December 01, 2008
Elections & Campaigns
State Legislatures Magazine

推荐阅读