python - 从下拉列表中选择后抓取网页
问题描述
转到https://www.comparis.ch/carfinder/default时,“Modell”下拉菜单处于非活动状态,直到您从“Marke”下拉菜单中选择某些内容。我的目标是通过浏览所有“Marke”值的列表来抓取所有“Modell”值。由于从“标记”下拉列表中选择某些内容时 URL 不会更改,因此我正在努力实现这一目标。我可以使用以下代码轻松获得“Marke”值列表:
from bs4 import BeautifulSoup
newest_secondhand_cars = 'https://www.comparis.ch/carfinder/default'
newest_secondhand_cars = requests.get(newest_secondhand_cars)
newest_secondhand_cars = newest_secondhand_cars.content
soup = BeautifulSoup(newest_secondhand_cars, "lxml")
marken = [str(el).split('>')[1].split('<')[0] for el in list(soup.find('select', {'id': 'ddlMake'}).find_all('option'))][1:]
print(marken)
我现在如何动态插入“Marke”值,然后刮掉所有“Modell”值?我认为这个答案非常接近,但我没有设法完成它:从下拉列表中的选定选项中抓取响应。这将是代码尝试使用“Marke”值执行 POST 请求,然后从响应中抓取“Modell”值(这不起作用):
from bs4 import BeautifulSoup
import requests
url = 'https://www.comparis.ch/carfinder/default'
with requests.Session() as session:
session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'}
# parsing parameters
response = session.get(url)
soup = BeautifulSoup(response.content, 'lxml')
data = {
'MakeList': 'BMW',
#'ctl00$ctl00$txtSearchWord': '',
#'__EVENTTARGET': soup.find('input', {'name': '__EVENTTARGET'}).get('value', ''),
#'__EVENTARGUMENT': soup.find('input', {'name': '__EVENTARGUMENT'}).get('value', ''),
#'__LASTFOCUS': soup.find('input', {'name': '__LASTFOCUS'}).get('value', ''),
#'__VIEWSTATE': soup.find('input', {'name': '__VIEWSTATE'}).get('value', ''),
#'__VIEWSTATEGENERATOR': soup.find('input', {'name': '__VIEWSTATEGENERATOR'}).get('value', ''),
#'__EVENTVALIDATION': soup.find('input', {'name': '__EVENTVALIDATION'}).get('value', ''),
}
# parsing data
response = session.post(url, data=data)
soup = BeautifulSoup(response.content, 'lxml')
print(soup.find('select', {'id': 'ddlModelGroup'}))
解决方案
import requests
from bs4 import BeautifulSoup
from itertools import zip_longest
r = requests.get("https://www.comparis.ch/carfinder/default")
soup = BeautifulSoup(r.text, 'html.parser')
marke = []
tag = []
model = []
for item in soup.findAll("select", {'id': 'ddlMake'}):
for name in item.findAll("option")[1:]:
marke.append(name.text)
for item in soup.findAll("select", {'id': 'ddlMake'}):
for num in item.findAll("option")[1:]:
tag.append(num.get("value"))
for a, b in zip_longest(marke, tag):
print(f"{'*' * 10}Extracting: {a}{'*' * 10}\n")
r = requests.get(
f"https://www.comparis.ch/carfinder/searchservice/GetModelGroupsForMake?makeID={b}").json()
data = list(item['Value'].strip() for item in r)
print(f"{data}\n")
model.append(data)
输出样本:
**********Extracting: Aixam**********
['300', '325', '400', '500', '600', '721', '751', 'Crossline', 'MAC', 'Scouty']
等等....
或使用以下内容:
import requests
from bs4 import BeautifulSoup
from itertools import zip_longest
r = requests.get("https://www.comparis.ch/carfinder/default")
soup = BeautifulSoup(r.text, 'html.parser')
marke = []
tag = []
model = []
for item in soup.findAll("select", {'id': 'ddlMake'}):
for name in item.findAll("option")[1:]:
marke.append(name.text)
for item in soup.findAll("select", {'id': 'ddlMake'}):
for num in item.findAll("option")[1:]:
tag.append(num.get("value"))
for a, b in zip_longest(marke, tag):
print(f"Extracting: {a}")
r = requests.get(
f"https://www.comparis.ch/carfinder/searchservice/GetModelGroupsForMake?makeID={b}").json()
data = list(item['Value'] for item in r)
model.append(data)
for a, b in zip_longest(marke, model):
print("Marke: {:<20}, Model: {:<100}".format(
a, ", ".join(str(i).strip() for i in b)))
Marke: Aixam , Model: 300, 325, 400, 500, 600, 721, 751, Crossline, MAC,Scouty
Marke: Alfa Romeo , Model: Stelvio, 145, 146, 147, 155, 156, 159, 164, 166, 1750, 33, 4C, 6, 75, 8C, 90, Alfasud, Alfetta, Arna, Brera, Giulia, Giulietta, GT, GTV, Mito, Montreal, Spider, Zagato
Marke: American Motors , Model: Eagle
Marke: Artega , Model: GT
Marke: Aston Martin , Model: Cygnet, DB 11, DB 7, DB 9, DBS, Lagonda, Rapide, V12 Vanquish, V12 Vantage, V8, V8 Vantage, Virage
Marke: Audi , Model: 100, 200, 80, 90, A1, A2, A3, A4, A4 Allroad, A5, A6, A6 Allroad, A7, A8, Cabriolet, Coupé, e-Tron, Q2, Q3, Q5, Q7, Q8, Quattro, R8, RS Q3, RS3, RS4, RS5, RS6, S1, S3, S4, S5, S6, S7, S8, SQ5, TT, V8
Marke: Austin , Model: Healey, Maestro, Metro, Montego
Marke: Bentley , Model: Arnage, Azure, Bentayga, Brooklands, Continental, Flying Spur, Mulsanne, Turbo
Marke: Bertone , Model: Freeclimber
Marke: BMW , Model: 1 Series, 2 Series, 2002, 3 Series, 4 Series, 5 Series, 6 Series, 7 Series, 8 Series, i3, i8, M-Series, X-Series, Z-Series
Marke: Bmw-Alpina , Model: B10, B12, B3, B4, B5, B6, B7, B8, D10, D3, D4, D5,Roadster, XD3, XD4, Z8
Marke: Bugatti , Model: EB, Veyron
Marke: Buick , Model: Century, Electra, Park Avenue, Regal, Riviera, Skylark
Marke: Cadillac , Model: Allanté, ATS, BLS, CT, CTS, Deville, Eldorado, Escalade, Fleetwood, Seville, SRX, STS, XLR, XT
Marke: Caterham , Model: Seven, Super Seven
Marke: Chevrolet , Model: Alero, Astro, Aveo, Beretta, Blazer, Camaro, Caprice, Captiva, Cavalier, Celebrity, Citation, Corsica, Corvair, Corvette, Cruze, Epica, Evanda, HHR, Impala, Kalos, Lacetti, Malibu, Matiz, Nubira, Orlando, Spark, Tacuma, Tahoe, TrailBlazer, Trans-Sport, Trax, Volt
Marke: Chrysler , Model: 300C, 300M, Crossfire, Daytona, ES, Grand Voyager,GS, GTS, Le Baron, Neon, New Yorker, PT Cruiser, Saratoga, Sebring, Stratus, Valiant, Viper, Vision, Voyager
Marke: Citroen , Model: 2CV, Acadiane, AX, Berlingo, Berlingo Electric, BX, C 15, C 25, C 35, C1, C2, C3, C3 Picasso, C3 Pluriel, C4, C4 Aircross, C4 Cactus, C4 Grand Picasso, C4 Picasso, C5, C6, C8, C-Crosser, CX, C-Zero, DS3, DS4, DS5, E-Mehari, Evasion, GSA, Jumper, Jumpy, LNA, Nemo, Saxo, Spacetourer, Visa, Xantia, XM, Xsara, ZX
Marke: Cupra , Model: Ateca
Marke: Dacia , Model: Dokker, Duster, Lodgy, Logan, Sandero
Marke: Daewoo , Model: Espero, Evanda, Kalos, Korando, Lacetti, Lanos, Leganza, Matiz, Musso, Nexia, Nubira, Rexton, Tacuma
Marke: Daihatsu , Model: Applause, CAB, Charade, Charmant, Copen, Cuore, Feroza, Gran Move, Hijet, Materia, Move, Pick-up, Rocky, Sirion, Taft, Terios, Trevis, YRV
Marke: Daimler , Model: X300, X308, XJ 40, XJ Serie III, XJ Serie V
Marke: De Tomaso , Model: Guarà, Pantera
Marke: DFM , Model: Mini
Marke: Dfsk , Model: C-Serie, K-Serie, V-Serie
Marke: Dodge , Model: Avenger, Caliber, Challenger, Dart, Durango, Journey, Magnum, Nitro, Ram, Viper
Marke: Donkervoort , Model: D8, S8
Marke: Ds Automobiles , Model: DS3, DS4, DS5, DS7
Marke: Ferrari , Model: 308, 328, 348, 412, 456, 512, 575, 599, 612, 812, California, Dino, Enzo, F12, F355, F360, F40, F430, F458, F488, F50, F550, FF, GTC, Laferrari, Mondial, Portofino, Testarossa
Marke: Fiat , Model: 124, 126, 127, 130, 131, 500, 500L, 500X, 595, 850, Argenta, Barchetta, Bertone, Brava, Bravo, Cinquecento, Coupé, Croma, Dino, Doblo, Ducato, Fiorino, Freemont, Fullback, Idea, Marea, Multipla, Palio, Panda, Punto, Qubo, Regata, Ritmo, Scudo, Sedici, Seicento, Spider, Stilo, Strada, Talento, Tempra, Tipo, Topolino, Ulysse, Uno
Marke: Fisker , Model: Karma
Marke: Ford , Model: B-Max, Capri, C-Max, Cortina, Courier, Ecosport, Edge, Escort, Fiesta, Focus, Fusion, Galaxy, Granada, Grand C-MAX, KA, Kuga, Maverick, Mercury, Mondeo, Mustang, Orion, Puma, Ranger, Scorpio, Sierra, S-Max, Streetka, Taunus,Tourneo, Tourneo Connect, Tourneo Courier, Tourneo Custom, Transit, Transit Connect, Transit Courier, Transit Custom
Marke: Ford (USA) , Model: Aerostar, Cougar, Explorer, GT, Mustang, Probe, Taurus, Thunderbird, Windstar
Marke: Giotti Victoria , Model: Gladiator
Marke: Honda , Model: Accord, Acty, Ballade, Civic, Concerto, CR-V, CR-X, CR-Z, FR-V, HR-V, Insight, Integra, Jazz, Joker, Legend, Logo, NSX, Prelude, Quintet,S2000, Shuttle, Stream
Marke: HS , Model: Cobra
Marke: Hummer , Model: H2, H3
Marke: Hyundai , Model: Accent, Atos, Coupé, Elantra, Galloper, Genesis, Getz, Grandeur, H 100, H-1, H350, i10, i20, i30, i40, Ioniq, iX20, iX35, iX55, Kona, Lantra, Matrix, Nexo, Pony, Santa Fe, Santamo, Scoupe, Sonata, Starex, Terracan, Trajet, Tucson, Veloster, XG
Marke: Infiniti , Model: EX, FX, G, M, Q, QX
Marke: Innocenti , Model: Mini
Marke: Isuzu , Model: Campo, D-Max, KB, L, Midi, N, NKR, NLR, NMR, NNR, NPR, Pick Up, WFR, WFS
Marke: Isuzu (GM) , Model: Trooper
Marke: Isuzu (J) , Model: Gemini, Piazza
Marke: Iveco , Model: 50, Daily, Eco-daily, Massif, Turbo-daily
Marke: Jaguar , Model: Concept, E-Pace, E-Type, F-Pace, F-Type, I-Pace, MK, S-Type, XE, XF, XJ, XJR, XJ-S, XK, X-Type
Marke: Jeep , Model: Cherokee, CJ-7, CJ-8, Commander, Compass, Grand Cherokee, Kaiser, Patriot, Renegade, Wagoneer, Willys, Wrangler
Marke: KIA , Model: Carens, Carnival, Cee'd, Cerato, Clarus, K2500, K2900, Leo, Magentis, Niro, Opirus, Optima, Picanto, Pregio, Pride, Rio, Sephia, Shuma, Sorento, Soul, Sportage, Stinger, Stonic, Venga
Marke: KTM , Model: X-Bow
Marke: Lada , Model: 110, 111, 112, 1500, Kalina, Niva, Nova, Priora, Samara
Marke: Lamborghini , Model: Aventador, Countach, Diablo, Gallardo, Huracan, Jalpa, LM, Murciélago, Urus
Marke: Lancia , Model: A 112, Beta, Dedra, Delta, Flaminia, Flavia, Fulvia, Gamma, HPE, Kappa, Lybra, Musa, Phedra, Prisma, Thema, Thesis, Trevi, Voyager, Y, Ypsilon, Zeta
Marke: Land Rover , Model: 109, 110, 88, 90, Defender, Discovery, Evoque, Freelander, Range Rover, Range Rover Sport, Range Rover Velar
Marke: Lexus , Model: CT, GS, IS, LC, LS, NX, RC, RX, SC, UX
Marke: Ligier , Model: Be Up, Nova
Marke: Lotus , Model: 3-Eleven, Elan, Elise, Esprit, Europa, Evora, Exige, Super Seven
Marke: MAN , Model: TGE
Marke: Maserati , Model: 3200, Biturbo, Coupé, Ghibli, Grancabrio, Granturismo, Karif, Levante, Quattroporte, Shamal, Spyder
Marke: Mazda , Model: 121, 2, 3, 323, 5, 6, 626, 929, B 2500, BT-50, CX-3, CX-30, CX-5, CX-7, CX-9, Demio, E, MPV, MX-3, MX-5, MX-6, Premacy, RX-7, RX-8, Tribute, Xedos
Marke: Mclaren , Model: 540c, 570gt, 570s, 600lt, 650S, 675lt, 720S, MP4
Marke: Mega , Model: Mega
Marke: Mercedes-Benz , Model: 100, 190, 207, 208, 209, 210, 220, 230, 250, 280, 307, 308, 309, 310, 350, 400, 407, 409, 410, 450, 508, 510, 609, 611, 612, A-Klasse, B-Klasse, Citan, C-Klasse, CLA-Klasse, CLC-Klasse, CLK-Klasse, CL-Klasse, CLS-Klasse, E-Klasse, EQC, G-Klasse, GLA-Klasse, GLC-Klasse, GLE-Klasse, GLK-Klasse, GL-Klasse, GLS-Klasse, GT, Marco Polo, M-Klasse, R-Klasse, SEC-Klasse, S-Klasse, SLC-Klasse, SLK-Klasse,SL-Klasse, SLS-Klasse, Sprinter, Vaneo, Viano, Vito, V-Klasse, X-Klasse
Marke: MG , Model: F, Maestro, Metro, MGB, Montego, TF, ZR, ZS, ZT
Marke: Minelli , Model: TF
Marke: Mini , Model: Cabriolet, Clubman, Clubvan, Countryman, Coupé, Mini, Paceman, Roadster
Marke: Mitsubishi , Model: 3000 GT, ASX, Attrage, Canter, Carisma, Colt, Cordia, Eclipse, Galant, Grandis, i Miev, L 200, L 300, L 400, Lancer, Outlander, Pajero, Sapporo, Sigma, Space Gear, Space Runner, Space Star, Space Wagon, Starion, Tredia
Marke: Mitsubishi Fuso , Model: Canter
Marke: Morgan , Model: 4/4, Aero 8, Plus
Marke: Nissan , Model: Almera, Atleon, Bluebird, Cabstar, Cabstar E, Cherry, Cube, Double, e-NV200, GT-R, Interstar, Juke, King-/Double, Kubistar, Laurel, Leaf,Maxima, Micra, Murano, Navara, Note, Np300, NT400, Nv200, Nv300, Nv400, NX, Pathfinder, Patrol, Pixo, Prairie, Primastar, Primera, Pulsar, Qashqai, Serena, Silvia, Stanza, Sunny, SX, Terrano, Tiida, Trade, Urvan, Vanette, X-Trail, Z, ZX
Marke: Oldsmobile , Model: Cutlass, Delta 88, Ninety Eight, Toronado
Marke: Opel , Model: Adam, Admiral, Agila, Ampera, Antara, Ascona, Astra, Blitz, Calibra, Campo, Cascada, Combo, Commodore, Corsa, Crossland, Frontera, Grandland, GT, Insignia, Kadett, Karl, Manta, Meriva, Mokka, Monterey, Monza, Movano, Olympia, Omega, Rekord, Senator, Signum, Sintra, Speedster, Tigra, Vectra, Vivaro, Zafira
Marke: Peugeot , Model: 1007, 104, 106, 107, 108, 2008, 205, 206, 207, 208, 3008, 304, 305, 306, 307, 308, 309, 4007, 4008, 404, 405, 406, 407, 5008, 504, 505, 508, 604, 605, 607, 806, 807, Bipper, Boxer, Expert, iON, J 5, J 9, Partner, RCZ, Rifter, Traveler
Marke: Piaggio , Model: Maxxi, Porter, Quargo
Marke: Pontiac , Model: 6000, Bonneville, Firebird, Grand-AM, Grand-Prix, Trans Am, Trans Sport
Marke: Porsche , Model: 356, 911, 924, 928, 944, 968, Boxster, Carrera GT,Cayenne, Cayman, Macan, Panamera
Marke: Puch , Model: G, Haflinger, Pinzgauer
Marke: Qvale , Model: Mangusta
Marke: Reliant , Model: Scimitar
Marke: Renault , Model: 4 F, Alaskan, Alpine, Avantime, Captur, Clio, Espace, Express, Fluence, Fuego, Grand Espace, Grand Scénic, Kadjar, Kangoo, Koleos, Laguna, Latitude, Master, Mégane, Modus, R11, R12, R14, R16, R18, R19, R20, R21, R25, R30, R4, R5, R9, Safrane, Scénic, Spider, Super 5, Talisman, Trafic, Twingo, Twizy, Vel Satis, Wind, Zoe
Marke: Renault Trucks , Model: B 120, B 90, Mascott, Master, Maxity
Marke: Rolls-Royce , Model: Corniche, Cullinan, Dawn, Flying Spur, Ghost, Park Ward, Phantom, Silver Dawn, Silver Seraph, Silver Spirit, Silver Spur, Touring, Wraith
Marke: Rover , Model: 200, 25, 2600, 3500, 400, 45, 600, 75, 800, Mini, Streetwise
Marke: Saab , Model: 90, 900, 9000, 9-3, 9-4X, 9-5, 99
Marke: Seat , Model: Alhambra, Altea, Altea XL, Arona, Arosa, Ateca, Cordoba, Exeo, Ibiza, Inca, Leon, Malaga, Marbella, MII, Ronda, Tarraco, Terra, Toledo
Marke: Secma , Model: F16
Marke: Skoda , Model: 135, 136, Citigo, Fabia, Favorit, Felicia, Forman,Kamiq, Karoq, Kodiaq, Octavia, Praktik, Rapid, Roomster, Scala, Superb, Yeti
Marke: Smart , Model: Forfour, Fortwo, Roadster
Marke: Sokon , Model: Mini Truck
Marke: Ssang Yong , Model: Actyon, Korando, Kyron, Musso, Rexton, Rodius, Tivoli, XLV
Marke: Subaru , Model: 1600, 1800, 700, BRZ, E10, E12, Forester, G3X Justy, Impreza, Justy, Legacy, Levorg, Outback, SVX, Trezia, Tribeca, Vivio, WRX, XT, XV
Marke: Suzuki , Model: Alto, Baleno, Carry, Celerio, Grand Vitara, Ignis, Jimny, Kizashi, Liana, LJ, SA, Samurai, SJ, Splash, Swift, SX4, SX4 S-Cross, Vitara, Wagon R, X-90
Marke: Talbot , Model: Horizon, Matra, Samba, Solara, Tagora
Marke: Tata , Model: Indica, Pick Up, Safari
Marke: Tazzari , Model: Zero
Marke: Tesla , Model: MODEL 3, Model S, Model X, Roadster
Marke: Think , Model: City
Marke: Toyota , Model: 4-Runner, Auris, Auris Touring Sports, Avensis, Avensis Verso, Aygo, Camry, Carina, Celica, C-HR, Corolla, Corolla Verso, Cressida, Crown, Dyna, GT, Hi-ace, Hi-lux, iQ, Land Cruiser, Lite-ace, Mirai, Model - F, MR 2, Paseo, Picnic, Previa, Prius, Prius+, Proace, Proace Verso, RAV-4, Sienna, Starlet, Supra, Tercel, Urban Cruiser, Verso, Verso-S, Yaris, Yaris Verso
Marke: Triumph , Model: Spitfire, TR 6
Marke: TVR , Model: Cerbera, Chimaera, Griffith, S4, T350, Tuscan, V8
Marke: Venturi , Model: Cabriolet, Coupé
Marke: Volvo , Model: 144, 164, 240, 340, 360, 440, 460, 480, 740, 760, 780, 850, 940, 960, C30, C70, PV, S40, S60, S60 CC, S70, S80, S90, V40, V40 CC, V50, V60, V60 CC, V70, V90, V90 CC, XC40, XC60, XC70, XC90
Marke: VW , Model: Amarok, Arteon, Beetle, Bora, Buggy, Caddy, CC, Corrado, Crafter, Cross Caddy, Crossgolf, Crosspolo, Crosstouran, Crossup, Eos, Fox, Golf, Golf Plus, Golf Sportsvan, Grand California, Jetta, Käfer, Karmann, LT - Serie, Lupo,Passat, Phaeton, Polo, Santana, Scirocco, Sharan, T1, T2, T4, T5, T6, Taro - Serie, T-Cross, Tiguan, Touareg, Touran, T-ROC, Typ 2 - Serie, Up, Vento
Marke: Wiesmann , Model: Coupé, Roadster
Marke: Zagato , Model: Hyena
推荐阅读
- asp.net-core - ASP.NET Core:如何将 RazorPage 应用程序的控制器用作访问相同数据的移动应用程序的后端?
- python - 如何安装 optparse 模块?
- android - 以编程方式为自定义视图设置形状颜色
- javascript - Javascript 使用 window.innerWidth 和 window.innerHeight 缩放页面内容
- networking - 网络打印、不良标签打印的数据包分析
- javascript - 如何为每个帖子添加一个 eventListener(不使用 jQuery)?
- reactjs - 反应原生 - 超过一定长度的文本不显示
- r - 在 R 中计算行数时无法按日期分组
- angular - 如何在 Amcharts4 中加载外部数据
- react-native - React-Native 通过映射键隐藏组件