时间戳

首页 > 解决方案 > 无法在python中打印产品名称列表

python - 无法在python中打印产品名称列表

问题描述

当我抓取数据 skechers 网站时，它只打印单个产品，但我想提取所有产品名称，这里是网站：https ://www.skechers.com/women/shoes/athletic-sneakers/?start=0&sz= 168

import time
import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np

from selenium import webdriver
driver = webdriver.Chrome('D:/chromedriver.exe')

url= "https://www.skechers.com/women/shoes/athletic-sneakers/?start=0&sz=168"
driver.get(url)
SCROLL_PAUSE_TIME = 1
time.sleep(1)

# Get scroll height
"""last_height = driver.execute_script("return document.body.scrollHeight")

this doesn't work due to floating web elements on youtube
"""

last_height = driver.execute_script("return document.documentElement.scrollHeight")
conte = None

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0,document.documentElement.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.documentElement.scrollHeight")
    
    if new_height == last_height and conte:
       print("break")
       break
       
    last_height = new_height
    time.sleep(5)
    page_Source = driver.page_source
    page_Source = driver.page_source
    soup = BeautifulSoup(page_Source, 'html.parser')
    conte = soup.find_all('div',class_='row product-grid') 

wshoes = []

for items in conte:
    title= items.find('a',class_ = 'link c-product-tile__title').text
    print(title)

标签： pythonseleniumbeautifulsouppython-requestswebdriver

解决方案

import requests
import re
import json


def main(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.5",
        "Upgrade-Insecure-Requests": "1",
        "Cache-Control": "max-age=0"
    }
    r = requests.get(url, headers=headers)
    match = re.findall(r'push\(({\"e\".+})', r.text)
    mydic = {i: json.loads(match[i]) for i in range(0, len(match))}
    for i in mydic.values():
        print(i['d']['name'])


main('https://www.skechers.com/women/shoes/athletic-sneakers/?start=0&sz=168')

输出：

Skechers Luxe Collection: GOwalk Arch Fit - Golden Era
Skechers Luxe Collection: D'Lites - Natural Wave
Skechers x JGoldcrown: Uno - Spread the Love
Skechers x JGoldcrown: Uno - Loving Love
Skechers Max Cushioning Elite
Skechers Max Cushioning Elite - Even Stride
Skechers Max Cushioning Elite - Sunburst
Skechers Max Cushioning Elite - Brilliant
Skechers Max Cushioning Elite - Promised Day
Skechers Max Cushioning Elite - Destination Point
Skechers Max Cushioning Elite - Wind Chill
Skechers Max Cushioning Elite - Step Up
Skechers Max Cushioning Premier
Skechers Max Cushioning Premier - Majestic Road
Glide-Step Sport - Lively Glow
Glide-Step Sport - Daily Spark
Glide-Step Sport - Sweeter Days
Glide-Step Sport - New Appeal
Glide-Step Sport - Next Level
Glide-Step Sport - Fun Stride
Glide-Step Sport - Stay Wild
Skechers Arch Fit - Big Appeal
Skechers Arch Fit - Comfy Wave
Skechers Arch Fit - She's Effortless
Skechers Arch Fit - Gentle Stride
Skechers GOwalk Arch Fit - Motion Breeze
Skechers GOwalk Arch Fit - Glee
Skechers GOwalk Arch Fit - True Vision
Skechers GOwalk Arch Fit - Rose Parade
Skechers Arch Fit - Lucky Thoughts
Skechers Arch Fit - Vivid Memory
Relaxed Fit: D'Lux Walker - Quick Upgrade
Relaxed Fit: D'Lux Walker - Infinite Motion
Relaxed Fit: D'Lux Walker - Glorious Motion
Relaxed Fit: D'Lux Walker - Timeless Path
Relaxed Fit: D'Lux Walker - Cool Groove
Relaxed Fit: D'Lux Walker - Floral Habits
Relaxed Fit: D'Lux Walker - Refreshing Mood
Uno - Stand on Air
Uno - Frosty Kicks
Uno - Solid Air
Uno 2 - 2nd Best
Uno - Viper Queen
Uno - Hyped Hippie
Uno - Spectrum
Uno - Eye Catching
Ultra Groove
Skechers GOrun Razor +
Skechers GOrun Ride 8 Hyper
Skechers GOrun Elevate - Quick Stride
Skechers GOrun Elevate - Hot Streak
Skechers GOrun Pulse - Ultimate Best
Skechers GOrun Pulse - Bold Venture
Skechers GOrun Consistent - Fearsome
Skechers GOrun Consistent - Fleet Rush
Skechers GOrun Consistent
Skechers GOrun 400 V2 - Tropic Burn
Skechers GOrun 400 V.2 - High Road
Skechers GOrun Fast - Glimmer
Skechers GOrun Smart - Lucid Night
Ultra Flex Prime - Step Out
Ultra Flex Prime - Wondrous Gain
Ultra Flex 2.0 - Wild Voyage
Ultra Flex 2.0 - Lite-Groove
Ultra Flex 2.0 - Flash Illusion
Ultra Flex - Twilight Twinkle
Ultra Flex - Statements
Ultra Flex
Skechers GOwalk Hyper Burst - Extreme Outlook
Skechers GOwalk Hyperburst - Lunar Mars
Skechers GOwalk Hyper Burst
Skechers GOwalk Stability - Coco Jazz
Skechers GOwalk Stability - Magnificent Glow
Skechers GOwalk Joy - Magnetic
Skechers GOwalk Joy
Skechers GOwalk Joy - Paradise
Skech-Air Element 2.0 - Vivid Blush
Skech Air Element 2.0 - Kaleidoscope Splash
Skech-Air Element 2.0
Skech-Air Element 2.0 - Rising Gaze
Skechers GOwalk Air - Twirl
GOwalk Air - Whirl
Skechers Arch Fit Refine - Her Best
Skechers Arch Fit Refine
Glide-Step - Stepping Up
Relaxed Fit: Empire D'Lux - Sharp Witted
Relaxed Fit: Empire D'Lux - Paradise Sky
Skechers GOrun Smart
Skechers GOwalk Outdoors - River Path
Skechers GOwalk Stretch Fit - Leopard Zone
Skechers GOwalk Stretch Fit
Skechers GOwalk 5 - Lake Views
Skechers GOwalk 5 - Uprise
Skechers GOwalk 5 - Alive
Summits - Fast Attraction
Summits
Summits - Quick Getaway
Summits - True Glee
Skech-Air Dynamight - Radiant Choice
Skech-Air Dynamight - Top Prize
Dynamight 2.0 - Eye to Eye
Skech-Air Dynamight - Big Step
Skech-Air Dynamight - Easy Call
Dynamight
Skechers Eclipse - She's Breezy
Fashion Fit - Up A Level
Flex Appeal 4.0 - Brilliant View
Flex Appeal 4.0 - Vivid Spirit
Flex Appeal 4.0
Flex Appeal 4.0 - Simple Joy
Flex Appeal 4.0 - True Clarity
Flex Appeal 4.0 - Coated Fidelity
Flex Appeal 4.0 - Floral Grounds
Flex Appeal 3.0 - Steady Energy
Flex Appeal 3.0 - She's Iconic
Flex Appeal 3.0 - First Insight
Flex Appeal 3.0 - Steady
Flex Appeal 3.0 - Fan Craze
D'Lites - Soft Blossom
D'Lites - Biggest Fan
D'lites - Fresh Start
D'Lites - Play On
D'Lites - Sparkling Rain
D'Lites - Me Time
D'Lites - Luscious Chic
D'Lites - New Journey
Evero
Energy Racer - Embrace Her
Energy Racer - She's Iconic
Energy Racer - Oh So Cool
Dr. Seuss The Grinch: Uno Rugged - You're A Mean One
Skechers GOrun Air - Wavy Path
Skechers GOrun Air - Silver Sparkle
Skech-Air Element - Brisk Motion
Ultra Flex 2.0 - Delightful Spot
Skechers GOrun Pure 2
Skechers GOrun Fast - Retro Insight
Skechers GOrun Hyper Pillars - Fuse
Skechers GOwalk 5 - Faith
Flex Appeal 3.0
Skechers GOmeb Speed 6 Cloak Hyper
Skechers GOrun Razor 3 Cloak Hyper
Skechers GOrun 7+
Skechers GOrun Forza 4 Hyper
Skechers GOrun OG Hyper
Skechers GOrun Power
Skechers GOrun MaxRoad 4+
Skechers GOrun Power - Fleetz

如果您访问字典，您还可以获得其他信息，例如：

{
  'e': 'impressions',
  'd': {
    'name': 'Skechers Luxe Collection: GOwalk Arch Fit - Golden Era',
    'id': '124488',
    'brand': [
      'Skechers Arch Fit'
    ],
    'price': 100,
    'originalPrice': 100,
    'discountPrice': 100,
    'categories': [
      'Women/Shoes',
      'Women',
      'Women/Shoes/Athletic Sneakers',
      'Women/Shoes/Lace Up',
      'Women/Women-shop-all',
      'Collections/Arch Fit',
      'Collections/GOwalk',
      'Collections'
    ],
    'categoryTypes': {
      'cat0': 'Shoes',
      'cat1': 'Women',
      'cat2': 'Athletic Sneakers',
      'cat3': 'Lace  Up',
      'cat4': 'Women-shop-all',
      'cat5': 'Arch Fit',
      'cat6': 'GOwalk',
      'cat7': 'Website Premieres'
    },
    'divisionName': [
      'GO WALK WOMENS'
    ],
    'category': 'Women/Shoes',
    'list': 'Category L2 - Shoes',
    'variant': 'TAN',
    'gender': 'W',
    'division': 'GOW',
    'position': '1',
    'isShoes': True,
    'width': 'N/A',
    'upcCode': 'N/A'
  },
  'ID': '7dbaffccc93c8bf0bd4cfa1992e002a2'
}

推荐阅读