时间戳

首页 > 解决方案 > 无法动态生成和使用标头中的某些令牌的值

python - 无法动态生成和使用标头中的某些令牌的值

问题描述

我正在尝试使用请求模块创建一个脚本来从网页中获取一些内容。我的意图是在我发送带有适当参数的 post http 请求时获得 200 状态代码。所以，这是站点地址，这是我为生成结果而填写的表格。我只是从 from 的两个下拉列表中选择了两个选项，然后点击Select My vehicle按钮来填充结果。

我在下面尝试的方式仅在我硬编码cookie和digital_token在标题中有效。请注意username和password是静态的。

由于 和 的值cookie是digital_token动态的，因此有必要动态生成它们，否则如果我对它们的值进行硬编码，它们会在几分钟内过期。

如何digital_token即时生成和使用价值？

我试过：

import requests
from bs4 import BeautifulSoup

base = 'https://www.avis.com.au/en/home'
URL = 'https://www.avis.com.au/webapi/reservation/vehicles'

payload = {"rqHeader":{"brand":"","locale":"en_AU"},"nonUSShop":False,"pickInfo":"SYD","pickCountry":"AU","pickDate":"26/01/2021","pickTime":"9:00 AM","dropInfo":"SYD","dropDate":"27/01/2021","dropTime":"9:00 AM","couponNumber":"","couponInstances":"","couponRateCode":"","discountNumber":"","rateType":"","residency":"AU","age":25,"wizardNumber":"","lastName":"","userSelectedCurrency":"","selDiscountNum":"","promotionalCoupon":"","preferredCarClass":"","membershipId":"","noMembershipAvailable":False,"corporateBookingType":"","enableStrikethrough":"true","picLocTruckIndicator":False}

headers = {
    'content-type': 'application/json',
    'cookie': 'JSESSIONID=7-PfFQTdLoe4Allx772MXjNRHg-proLZhT7Wkn5k.w04vprecmapp02; akaalb_production_config=~op=avis_au:avis-au-us-wdc|~rv=36~m=avis-au-us-wdc:0|~os=7f956ca2417c5e686d715889b6a30f65~id=227f46458e1d74f397bc3732493d29f5; SessionPersistence=PROFILEDATA%3A%3DauthorizableId%253Danonymous; datacenter=cwdc; visitorId=cwdc-A6dcb0c81-876f-4200-b8b4-cf01d6665ad9; _gcl_au=1.1.1537375693.1611743635; _ga=GA1.3.666679016.1611743635; _gid=GA1.3.1276090703.1611743635; IR_gbd=avis.com.au; _fbp=fb.2.1611743635775.709997675; __qca=P0-236558422-1611743636477; APISID=f4bf5c40-9387-4319-b347-67a23e62ef8a; DIGITAL_TOKEN=86c21933-e3fe-4867-855e-256991d6c099-02-cwdc-ho4905; IR_8968=1611781534175%7C0%7C1611781534175%7C%7C; _uetsid=27fa7d10608b11eb994047dca2e1810a; _uetvid=27faeca0608b11eb8fe3e90805023a70; _gat_UA-6997633-40=1',
    'digital_token': 'eyJjdHkiOiJKV1QiLCJlbmMiOiJBMjU2R0NNIiwiYWxnIjoiZGlyIn0..mPE_QHWlQV4uOqy6.3cB4yCaW_mvSRbigEXpvvB-Y_zxrixxddIdq0kzoNyQlJEGgsefnR-8cSHXxWgDWU6Tp5Wl6WFfOjfPETeh9gnFmQpokP2-vZ0pX6k288PAjqg7GbJoaiYo7fBsDAs3nQfu_lsRtvtoqh4ku5jptU8juVLYl7DuBzSs3KtmJobP6esWh8mrx0ezClpg1Qt0E4jpfmSYmCx0dbDgSyKYxp-fB3YD8DDBB1xrkCWFwCW4P4HQqtt6S1EYbW_FKsFmwI-wnyouSXfMXCCeEc8_Ib6PO1LBU3PaWfCGyzEW2M-Z8QeKLYXxp5GCmgQDRylm5JiVhIhWPDLh9TLcQGyWKABOR9wrM7FkUSUBh9fGX-O-HRc63AxDSZ5zn81Wo9roqPLt_BoAm-RYxbmSjxD298x8i10beEv8vu3IDjuqGmbldmEpvdST2gE_-KRKZAwKDcM8NCzU9LuG3Q3G6eSOJRL3jfMx37uqVpOSe9qMQEYBtMn_tUhKWmThsYLzvmH6bgvmCKaAwOgd2TzSI_kcieAidjOH-OJLEbJ9zrGNw6orGu6x7_UcC9r8fy5T8JbsNXrOFQE_LL3jkzNymlTXQRZOO51AOgoxuVrNJBM8TrzHftY7y5HVa_gQleLFG_uN_RGg3mr9fzKxjcGEHFt15RqqtY2V70LOEpQeKrPjC7B-D-UI0XHa3lOhBp3bIDz4tX-loFP3QOwn2e5Nmqk_O0pr9Lfzjbc0jnk-6iUIFsQzKl4G-eJwaUkN5n549BpeaFrpfwInNNloVdVTqDwGLYA-xcYnkNBozXFuNjkNwYjv66Zlwaj3_bC6sz-SySyu7rtwhdW85O8DREhj8I6xb-VvwDDGsb6-h0GgeCjGN7JZ4qkpZ4BvLyPu-YxeB6mz-xcO22-t9lyx8abABlm8oLdhOkYBDgA8x7exGzNMbUiDe_D61bsgnkTu6XIMKrm4HMOcT9ZsGcXkK2BZ909i_cXyTDpq26YLgTjEcH5Ob5NRC-eSAw4eeddDmeAxWlL4TixCESaKsbxC8sXRC7tURQDIasW6xLAAkVSTCnlkOV34luR3VOU6KaHvoa6RaAdKgFw5VXvCddFkhA_mqWhOO60Y1ow9wRWu65Z8M6VjqsU_rRvW8HMlfUqJ9I8EHQJ4yG3PcbNRxx4qzpXOs6TGsk9OMhQzPOaEoo4-hzaV4f54HeyZDFUx2gI8CEopbd1Shg1e0p1Mk0gEuM12550A--M6dvZ0oMLL4lv9XzEEe2mGy16gmiRsRkvEoFcumbM1XUuX3IkuCVHk9OcPNNlVPD2jIGT8prY8aHDP8aJ6wl0b1GtoiwDA6CAA9ePzRpLN4G617Wnotttmao5YVrcE_b1O0CQbfYqLQ_lPvTl1n4-P1p-YuiRwYAdw2BQEpOv0LyS1ekycEzWHN4-BOFr9MxQ.bZcUICoU7nJzaxeT0rENtg',
    'password': 'AVISCOM',
    'user-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36',
    'username': 'AVISCOM'
}

with requests.Session() as s:
    s.headers.update(headers)
    r = s.post(URL,json=payload)
    print(r.status_code)

标签： pythonpython-3.xweb-scrapingpython-requests

解决方案

正如评论中已经提到的，这个令牌是 JS 生成的，不能使用requestsorcurl或其他方式获取。

一个简单（但不是非常高效）的方法是使用例如 Firefox 的 Selenium：

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True   # Don't open a visible pop-up window
driver = webdriver.Firefox(options=options)

driver.get('https://www.avis.com.au/en/home')
cookies = driver.get_cookies()

driver.quit()

print(cookies)

[{'name': 'akaalb_production_config', 'value': '~op=avis_au:avis-au-us-dal|~rv=37~m=avis-au-us-dal:0|~os=7f956ca2417c5e686d715889b6a30f65~id=a09a33378351c4e0fb1683926d7cf558', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': False}, {'name': 'APISID', 'value': 'd3879ba0-bdec-488e-94f5-0c517b8d300f', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True}, {'name': 'DIGITAL_TOKEN', 'value': '89a7cd03-276c-4f2f-942f-07a5171a13d2-02-cdal-ho3904', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True}, {'name': 'datacenter', 'value': 'cdal', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True}, {'name': 'visitorId', 'value': 'cdal-A96653921-88e6-431a-9b08-2582dfe47a75', 'path': '/', 'domain': 'www.avis.com.au', 'secure': True, 'httpOnly': True, 'expiry': 1927796685}, {'name': 'SessionPersistence', 'value': 'PROFILEDATA%3A%3DauthorizableId%253Danonymous', 'path': '/', 'domain': 'www.avis.com.au', 'secure': False, 'httpOnly': False}]

在这里，您已经看到了您的令牌。例如通过以下方式获取它：

print([ c for c in cookies if 'DIGITAL_TOKEN' in c.values() ][0]['value'])

89a7cd03-276c-4f2f-942f-07a5171a13d2-02-cdal-ho3904

推荐阅读