python - 为什么标题的一部分是空的？

Question

我想抓取https://www.reddit.com/search?q=expiration&type=link&sort=new上所有问题的链接、标题和时间戳。由于有很多问题，我们需要向下滚动以触发加载。这是代码：

from selenium import webdriver
import time

urlMethod = "https://www.reddit.com/search?q=expiration&type=link&sort=new"
driver = webdriver.Chrome()
driver.get(urlMethod)

SCROLL_PAUSE_TIME = 3
i = 0

last_height = driver.execute_script("return document.body.scrollHeight")

while (i <= 3):
    i = i + 1
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(SCROLL_PAUSE_TIME)    
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

questions = driver.find_elements_by_xpath('//div[@data-click-id="body"]')
print(len(questions))

for q in questions:
    url = q.find_elements_by_xpath('.//a[@data-click-id="body"]')[0].get_attribute('href')
    title = q.find_element_by_xpath('.//a[@data-click-id="body"]').find_element_by_xpath('./h2/span').text
    print("title : " + title)
    datePostRel = q.find_elements_by_xpath('.//a[@data-click-id="timestamp"]')[0].get_attribute('innerHTML')

但是，我的测试显示部分标题是空的。有谁知道为什么？

PS：我尝试使用WebDriverWait，但它给了我一个TimeoutException错误。

标签： pythonselenium

Answer 1

试试下面的代码。它应该可以工作。而不是你必须使用的文本innerHTML。

  title = q.find_element_by_xpath('.//a[@data-click-id="body"]').find_element_by_xpath('./h2/span').get_attribute("innerHTML")

输出：

title : (Possibly Unpopular) Opinion: MTG Arena's Substandard Free-to-Play Economy Creates an Inferior Meta
title : Should I bring up the topic of a possible learning disability with my psychiatrist before I go to college or am I making a big deal out of nothing?
title : MeasureUp vouchers for Microsoft technical practice exams
title : should i break up with him or not?
title : Update on the blown up intervention. Things got worse. He went to jail. I am leaving to get treatment for trauma, PTSD, and codependency.
title : AITA for taking two days to consider a employment offer?
title : Hi, I am trying to find the most current application with which to renew DACA, but the only one I can find on the USCIS has 1/31/19 as the <em style="font-weight:700">expiration</em> date. Is this still a valid form, or will it be rejected? Current permit is to expire in September.
title : Just like food, humans have <em style="font-weight:700">expiration</em> dates too..
title : Drug <em style="font-weight:700">Expiration</em> Dates — Do They Mean Anything? - Harvard Health (This should be sticky'd btw)
title : Here is a scientific study made on how long prescription drugs last after their expiry date. I thought it would be relevant here.
title : If poison is past its <em style="font-weight:700">expiration</em> date, is it more poisonous or less poisonous?
title : Is there any coming back from deep-seated resentment?
title : 29 Domains for sale! | All Priced between $19-$79 | BIN via Efty/Paypal
title : This Pringles can with a Leap Day <em style="font-weight:700">expiration</em> date
title : Is it wrong for a relationship to have an <em style="font-weight:700">expiration</em> date?
title : Buy Valtrex From a Usa Pharmacy Without a Prescription, How To Mail Order Valtrex Canada
title : Fragment Bullets
title : Calculating <em style="font-weight: 700;">Expiration</em> Dates - Previous Solution No Longer Works
title : My current dilemma with excess backstory. TLDR at the bottom
title : Worst (and not-so-bad) things for metric-born US resident
title : Weird Question about sorting your papers?
title : Hot Cash Mega Thread
title : [40k] What would a modern earth's tithe consist of?
title : TIL an FDA study requested by the military found 90% of more than 100 drugs, both prescription and over-the-counter, were still safe &amp; effective even 15 years after the <em style="font-weight: 700;">expiration</em> date. <em style="font-weight: 700;">Expiration</em> dates don’t really indicate a point at which the medication is no longer effective or unsafe to use.
title : Do I still have stock options?
title : CVS Coupon <em style="font-weight: 700;">Expiration</em> Policy
title : CVS Coupon <em style="font-weight: 700;">Expiration</em> Date
title : Over 100 OTC and prescription drugs 90% effective 15 years after <em style="font-weight: 700;">expiration</em> dates
title : We need all hands on deck
title : My option grant contract up to scratch (concerned about repurchase/claw back)
title : I'm really not happy anymore
title : How to have a healthy non-committed relationship?
title : TIL an FDA study requested by the military found 90% of more than 100 drugs, both prescription and over-the-counter, were still safe &amp; effective even 15 years after the <em style="font-weight: 700;">expiration</em> date. <em style="font-weight: 700;">Expiration</em> dates don’t really indicate a point at which the medication is no longer effective or unsafe to use.
title : Marriage license question
title : Just plain ole happy....
title : Need a favor! Swap Menopur in the DC area
title : Understanding options
title : <em style="font-weight: 700;">Expiration</em>?
title : Need answers about respiratory symptoms to calm my nerves
title : [HIRING] Looking for PHP developer for Pydio
title : Top-DOCTORS.com make offer
title : PendingIntent.isTargetedToPackage() returns false for IntentService. Why, and how to fix it?
title : Reconnecting with my [28f] ex [28m]?
title : PSA (possibly): Colourpop is technically not allowed to ship to the EU
title : Each time I start getting to know someone new, I can’t help think about the relationship having an <em style="font-weight: 700;">expiration</em> date.
title : Can we get an <em style="font-weight: 700;">expiration</em> timer on all electronics like the sensor dart or a proximity killswitch?
title : Do dates just live forever? No <em style="font-weight: 700;">expiration</em> date on the package. Am I going to die?
title : WTW for opposite of a "limited time offer"
title : S.102 - Committee Vote
title : But it's expired...
title : this <em style="font-weight: 700;">expiration</em> stamp (2017 on the trademark though)
title : /u/Nadieestaaqui on Do you think NAFTA is a horrible deal but USMCA is a great trade agreement?
title : Classic teenager: forgets to check the pills <em style="font-weight: 700;">expiration</em> date and just chucks em down the chute
title : Exit on vertical debit vs credit spreads
title : TEFL Contrct Review China. Am I safe Or Could I Get In Trouble?
title : Expired Betaine HCL w/ Pepsin still good?
title : IKE phase 1 has expired, but PA doesn't start re-key for 45mins to an hour later
title : ConnectWise workflow for Expiring Configurations
title : Your portal (and all the data) on ITBoost is kept online even if you have cancelled months ago.
title : Document Authentication Without Job Offer
title : The Deli Meat Saga
title : Getting errors when trying to import a module
title : I am the dead bedroom.
title : Does rank <em style="font-weight: 700;">expiration</em> affect Grand Champion rewards?
title : How do I tell my (24f) boyfriend (25m) that his words hurt?
title : Freaking out right now
title : Same doctor that said "no longer coronary risk" 2 wks prior to death now refuses to get us death certificates. Do we have legal recourse?
title : A Footstep Echo is FREE on Kindle this week - Sci-fi / Time Travel - Mystery
title : What makes milk/chocolate milk taste/smell bad if it's not expired?
title : The <em style="font-weight: 700;">expiration</em> date on this bag of potato sticks
title : What are the major issues driving the potential lockout?
title : Attention all epic conservatives!
title : Anyone Else Considering Puts for the NIO Lock-up <em style="font-weight: 700;">Expiration</em>?
title : Suggestion Chems/Food/Drinks <em style="font-weight: 700;">expiration</em> time
title : Unknown <em style="font-weight: 700;">Expiration</em> Fuji 160S - Mamiya 645 65mm 2.8 Central Park, my lovely girlfriend with her Minolta SRT 101.
title : What happens when someone sells a put a few days before <em style="font-weight: 700;">expiration</em> and $0.20 over the strike price.
title : Are IUDs still effective near the end of their <em style="font-weight: 700;">expiration</em>?
title : Topical ibuprofen gel out of date, should I be using it?
title : Prestige Points wouldnt feel so awful of a system if we got them from spending RP on skins/emotes/etc too
title : 70% off mens beach shorts
title : Begging for... Sushi?
title : CAE vs IELTS Comparison
title : The double strike <em style="font-weight: 700;">expiration</em> cancel bug is back
title : ISO 13485:2016 Certification in Singapore
title : questions about philly connections tickets
title : [USA] [H][USA] [H] Persona 5, Nier, Spiderman, Yakuza 6, Monster Hunter World, Fallout 76, DBZ Fighterz list [PS4] Mario Odyssey, lets go evee list [SWITCH] Ever Oasis, List [3DS] [W] Ys VIII, list [PS4]Ys VIII: Lacrimosa of Dana, list [SWITCH]
title : A few final questions for a new Meat DM.
title : <em style="font-weight: 700;">Expiration</em> dating hand-made whip cream
title : My chips have an <em style="font-weight: 700;">expiration</em> time
title : Boss forgets 3 year old food container
title : H.R.171: Military Federalism Act of 2018 AMENDMENT PERIOD
title : [Product Question] <em style="font-weight: 700;">Expiration</em> Date On Scar Creams, Do They Matter?
title : DEA claim Chapter 35
title : I am afraid of falling in love with a girl in my university
title : I'm Paige Feldman, a development and acquisitions exec and writer/director/producer - Ask Me Anything!
title : Bungie Rewards <em style="font-weight: 700;">expiration</em> dates have been updated
title : ELI5: How are <em style="font-weight: 700;">expiration</em> dates on food/other products with <em style="font-weight: 700;">expiration</em> dates actually determined? Like how do they figure out when the product is going to expire?
title : [H] VPN Accounts starting at just $1! [W] BTC/ETH/LTC/Paypal
title : $MEET Earnings Play
title : People should be able to activate 2 week free trial n word passes
title : Needle Teeth - <em style="font-weight: 700;">Expiration</em> Date [Cassette out next month on Solidarity Club Records]
title : Issues with trying to add a new Mastercard to SPay. Anyone else having similar issues?
title : Third Party Tool to Monitor AD Cert Authority?
title : MSU ID Card <em style="font-weight: 700;">expiration</em>
title : A law mandating that all best by/<em style="font-weight: 700;">expiration</em> dates be in the same spot and in the same format on all products for easy deciphering by consumers.
title : met her after no contact for bit over 1mo and so confusing
title : When u have the same <em style="font-weight: 700;">expiration</em> date as sour cream
title : What character stayed past their <em style="font-weight: 700;">expiration</em> date and make the show/book/whatever worse because they were popular?
title : Should I take the internship offer or go for the interview for a full-time position? Or both?
title : Attention all Yasuo mains!!
title : RMA without proof of Purchase?
title : I was curious about whois of jw.(b)org
title : [Sell][US]Pick 4 freebies with your order! Selling Makeup &amp; Skincare: eSpoir, Cosrx, Innisfree, Acwell, Neogen, Peripera, Klairs, Etude House, Benton, Too cool for school, Glossier, The Ordinary, Annie's Way, DewyTree, Holika Holika, A'Pieu + more!
title : Lucky day promotions: free wheel spin!! Enter code: RO8AKFZ Good Luck!!
title : Virtually clueless on budget friendly but long lasting Acer Chromebook (uk)
title : A Footstep Echo is Available FREE on Kindle this week!! Sci-fi / Mystery / Time Travel
title : How long/often can I travel in the USA with an ESTA?
title : How long can I travel in the USA with an ESTA?
title : 3-15-19 (no new info so don't freak out)
title : When should I receive SRB payout?
title : [LPT] Everytime you close a contract, set yourself a reminder a couple of weeks/months (depending on your notice period) before <em style="font-weight: 700;">expiration</em> so that you get reminded about it before the contract extends itself.
title : Dumb question of the day.
title : Asp web api Access_token and Refresh token
title : Do you eat food that's past its <em style="font-weight: 700;">expiration</em> date? What is your reason?
title : Facilitated Trading - EARN CREDIT THIS WEEK: T-Mobile Tuesdays Trading/Discussion thread March 05, 2019