python - scrapy shell如何爬取特定网站

Question

我想通过scrapy shell来抓取这个网站，我试过：

$ scrapy shell 'https://aaav2.hinet.net/A1/AuthScreen.jsp'

加上用户代理

$ scrapy shell -s USER_AGENT='Mozilla/5.0 (Macintosh; Intel Mac OS X
10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2
Safari/601.3.9' 'https://aaav2.hinet.net/A1/AuthScreen.jsp'

然后view(response)一无所获。

有人能帮我纠正view(response)一下吗，就像我直接在浏览器中打开这个 URL 一样？

标签： pythonscrapy

Answer 1

您正在重定向到另一个 URL。做：

$ scrapy shell "https://aaav2.hinet.net/A1/error.jsp?aa-eurl=edc68fe62571d6617ef5f42113d9068aa9f6600e320d55084d75fbf2cd244155e02b9b684284ed94c52ee591d2edde9a&mesg=aa-version+parameter+is+required%21%3Cbr+%2F%3Eaa-productid+parameter+is+required%21%3Cbr+%2F%3Eaa-curl+parameter+is+required%21%3Cbr+%2F%3Eaa-eurl+parameter+is+required%21%3Cbr+%2F%3Eaa-fee+parameter+is+required%21%3Cbr+%2F%3E&aa-eurlDesc=&aa-device=pc&aa-usage=&aa-propertiesKey=&aa-language="

view(response)这将向您显示一个页面，就像您的浏览器一样。