时间戳

首页 > 解决方案 > 使用 python selenium 向下滚动以实现新元素的动态出现

python - 使用 python selenium 向下滚动以实现新元素的动态出现

问题描述

网址是https://donstroy.com/full-search. 在那个页面上，里面有 parent<div class="content">和 21 <div class="item">。当人类将鼠标悬停<div class="content">并向下滚动时，<div class="item">会动态出现多个新元素。

我试图通过 selenium 实现这个结果（出现新元素），但我能得到的只是前 21 <div class="item">。

我尝试了不同的方法在 selenium 中向下滚动，但它们都没有导致出现新<div class="item">元素。下面是我的代码。当我在 Element Inspector 的控制台中以纯 javascript 运行它时，使用 scrollIntoView 的第一种方法工作正常（它会导致出现新元素），但仍然无法通过 selenium 工作。

import selenium.webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.action_chains import ActionChains

def parse_objs():
    return driver.find_elements_by_xpath('//div[@class="item"]')

options = Options()
options.headless = True
driver = selenium.webdriver.Firefox(options=options)
driver.set_page_load_timeout(300) # yes, it may take up to multiple minutes to load the page so may need to wait long time, it's normal
driver.get('https://donstroy.com/full-search')

objs = parse_objs()

# Amount of object on initial page load
print('Initial load : %s' % len(objs)) # prints 21 as expected

# Method 1
driver.execute_script('arguments[0].scrollIntoView();', objs[-1])
objs = parse_objs()
print('Method 1 : %s' % len(objs)) # expected to increase but still prints 21
#The thing is that when running the same method in pure javascript in Element Inspector's console, it's working fine:
#var objs=document.getElementsByClassName('item');
#objs[20].scrollIntoView();
#after that, amount of objs (objs.length) increases to 41 so it should work in selenium but it does not!

# Method 2
actions = ActionChains(driver)
actions.move_to_element(objs[-1]).perform()
objs = parse_objs()
print('Method 2 : %s' % len(objs)) # expected to increase but still prints 21

# Method 3
objs_container=driver.find_element_by_xpath('//div[@class="content"]')
driver.execute_script('arguments[0].scrollTop = 300;', objs_container)
print('ScrollTop=%s' % driver.execute_script('return arguments[0].scrollTop', objs_container)) # always returns 0 no matter what value i try to apply for scrollTop in above command
objs = parse_objs()
print('Method 3 : %s' % len(objs)) # expected to increase but still prints 21

driver.quit()

标签： pythonselenium

解决方案

Looking at the website https://donstroy.com/full-search the next set of elements load only after reaching the end of scroll. Now with all your method it seems like webdriver is reading only the first 21 elements which has been loaded initially along with the page. You may want to see if the scroll is working so that the next set of elements appear dynamically. Add an expected conditions wait (EC), print the data and then scroll down again until the next set of data appear. Repeat this step till you get the desired length of list

推荐阅读