python - 使用 bs4 更改属性值后更新网站

Question

我目前正在使用 bs4 来抓取以下网站。我想将视图从“Top Reactions”更改为“Latest Reactions”，但唯一的方法是添加“selected”作为“Latest Reactions”周围的标签类之一。但是，在这样做之后，html 代码不会改变，并且仍然显示“热门反应”。有没有办法使用bs4解决这个问题？

url = "https://finance.yahoo.com/quote/{0}/community?p={0}".format(ticker)
page = urlopen(url)
html = page.read().decode("utf-8")
soup = BeautifulSoup(html, "html.parser")

latest_button = soup.find('li', {'class' : 'H(44px)'})
latest_button['class'] = latest_button['class'] + 'selected'

comments = soup.findAll("li", {"class": "comment"})


for comment in comments: # Goes through the latest n messages
    post = {
        
    }
    
    post["content"] = comment.find("div", {"class": "C($c-fuji-grey-l)"}).text

    print(post)

标签： pythonweb-scrapingbeautifulsoup

Answer 1

不需要硒。可以通过api获取数据。只需将 url 中的参数从 sort bypopular更改createdAt为即可。

import requests
import time
import re
import statistics


ticker = 'TSLA'

# Get prid
url = 'https://uk.finance.yahoo.com/quote/{ticker}/community?p={ticker}'.format(ticker=ticker)
response = requests.get(url).text
prid_search = re.search('{\"prid\":\"([a-zA-Z0-9]*)', response)
prid = prid_search.group(1)

finmb_search = re.findall('finmb_([\d]*)', response)
finmb = statistics.mode(finmb_search)

api_url = 'https://uk.finance.yahoo.com/_finance_doubledown/api/resource/canvass.getMessageList;apiVersion=v1;context=finmb_{finmb};count=20;index=null;lang=en-GB;namespace=yahoo_finance;oauthConsumerKey=finance.oauth.client.canvass.prod.consumerKey;oauthConsumerSecret=finance.oauth.client.canvass.prod.consumerSecret;query=namespace%20%3D%20%22yahoo_finance%22%20and%20(contextId%3D%22finmb_{finmb}%22%20or%20tag%3D%22{ticker}%22);rankingProfile=canvassPCRDProfile;region=GB;sortBy=createdAt;spaceId=1186050005;type=null;userActivity=true?bkt=finance-GB-en-GB-def&device=desktop&ecma=modern&feature=canvassOffnet%2CccOnMute%2CdisableCommentsMessage%2Cdebouncesearch100%2CdeferDarla%2CecmaModern%2CemptyServiceWorker%2Cenable3pConsent%2CenableCCPAFooter%2CenableCMP%2CenableConsentData%2CenableGuceJs%2CenableGuceJsOverlay%2CenableNavFeatureCue%2CenablePrivacyUpdate%2CenableStreamDebounce%2CenableTheming%2CenableUpgradeLeafPage%2CenableVideoDocking%2CenableVideoURL%2CenableXrayNcp%2CenableXrayNcpInModal%2CenableXrayTickerEntities%2CenableYahooSans%2CenableYodleeErrorMsgCriOS%2CncpListStream%2CncpPortfolioStream%2CncpQspStream%2CncpStream%2CncpStreamIntl%2CncpTopicStream%2CnewContentAttribution%2CnewLogo%2CoathPlayer%2CrelatedVideoFeature%2CuseNextGenHistory%2CvideoNativePlaylist%2CsunsetMotif2%2CenableUserPrefAPI%2CenableSingleRail%2CenhanceAddToWL%2Carticle2_csn%2CenableStageAds%2CsponsoredAds&intl=uk&lang=en-GB&partner=none&prid={prid}&region=GB&site=finance&tz=Europe%2FLondon&ver=0.102.4664&returnMeta=true'.format(ticker=ticker,finmb=finmb,prid=prid)
jsonData = requests.get(api_url).json()

messages = jsonData['data']['canvassMessages']
for message in messages:
    author = message['meta']['author']['nickname']
    createdAt = message['meta']['createdAt']
    createdAt = time.strftime('%Y-%m-%d %I:%M:%S %p', time.localtime(createdAt))
    text = message['details']['userText']
    print('%s posted at: %s \n%s\n\n' %(author, createdAt, text)) 
    

输出：

trippett posted at: 2021-04-19 02:51:05 PM 
I sat on the fence for 3 weeks about joining (http://Stocks-tracker.com) for their weekly picks. Big mistake. AAPL exploded and I completely missed out. Don’t make the same mistake I made. Join them today.


Maqrsqrt posted at: 2021-04-19 02:51:05 PM 
Gonna makes me lotsa money


festus posted at: 2021-04-19 02:50:27 PM 
165 into earnings


Maqrsqrt posted at: 2021-04-19 02:49:58 PM 
Flush those daytraders!!


matthew posted at: 2021-04-19 02:48:54 PM 
Bought 1 apple share! Woo


steve posted at: 2021-04-19 02:48:36 PM 
go baby, go  enjoy the ride to 150+ this week 


Aleem posted at: 2021-04-19 02:48:03 PM 
If only this hits 145 this week 


felix posted at: 2021-04-19 02:47:55 PM 
I think by July it should be in the $200


Akshay posted at: 2021-04-19 02:47:41 PM 
What's the news?


Maqrsqrt posted at: 2021-04-19 02:46:32 PM 
look out below!!


michael posted at: 2021-04-19 02:45:17 PM 
Where’s Giuseppe ? Lol


Kafil posted at: 2021-04-19 02:43:38 PM 
Lets stay above $ 135 for today


Tina posted at: 2021-04-19 02:42:58 PM 
yesss! $135 crossed!


Kafil posted at: 2021-04-19 02:42:57 PM 
BINGO


Jack posted at: 2021-04-19 02:37:59 PM 
And the climb into earnings continues


Trader posted at: 2021-04-19 02:37:43 PM 
Wants to run to 141??


ray posted at: 2021-04-19 02:36:35 PM 
Turning green


Tina posted at: 2021-04-19 02:36:34 PM 
Way to go! This will be $138 tomorrow at 3.30 pm


Kafil posted at: 2021-04-19 02:36:16 PM 
Watch out if we brake $ 135 next stop will be $ 140


Richie Rich posted at: 2021-04-19 02:34:57 PM 
134.98 then down