时间戳

首页 > 解决方案 > R循环遍历每个URL，抓取，解析，提取节点并放入数据框中

r - R循环遍历每个URL，抓取，解析，提取节点并放入数据框中

问题描述

使用以下包： require(stringr) require(RCurl) require(XML)

我能够连接到所需的网页，并提取所需的信息。

> url="https://www.realtor.com/realestateagents/33415/pg-1" doc =
> getURLContent(url, verbose = TRUE) #gets the doc , verbose = show me
> me what you are doing) doc = htmlParse(doc)
> # name =  getNodeSet(doc,  "//div[@itemprop = 'name']") name = sapply(name, xmlValue)
> # phone =  getNodeSet(doc,  "//div[@itemprop= 'telephone']") phone = sapply(phone, xmlValue)

我生成了一个网址列表

urlList = c("https://www.realtor.com/realestateagents/33415/pg-1",
                "https://www.realtor.com/realestateagents/33415/pg-2")

    urlList = as.list(urlList)

我想遍历每个 url，捕获相同的节点并将结果放在一个数据框中，该数据框由名为 Name 和 Phone 的列组成。

我尝试了以下但没有成功

Reduce(function(...) merge(..., all=T), 
       lapply(urls_list, function(x) {
         data.frame(urlList=x, 

                     # d<- htmlParse(getURLContent(x))
                    d<-htmlParse(d)
                    d1 =  getNodeSet(d,  "//div[@itemprop = 'name']")
                    name = sapply(name, xmlValue)

       })) -> results

谢谢你的帮助

标签： rxmlweb-scrapingweb-crawlerrcurl

解决方案

我认为这样的事情应该可以为您提供所需的信息。

library(rvest)

zip.codes <- c("33415", "33413")

results <- list()

result.index <- 0

for(zip in zip.codes){

  url <- paste0("https://www.realtor.com/realestateagents/", zip ,"/pg-1" )

  page <- read_html(url)

  max.pages <- as.numeric(max(page %>% 
                                html_nodes(xpath = '//*[@class="page"]') %>% 
                                html_nodes("a") %>% 
                                html_text))

  for(i in c(1:max.pages)){
    print(paste("Processing Zip Code", zip, "- Page", i, "of", max.pages))

    result.index <- result.index + 1

    url <- paste0("https://www.realtor.com/realestateagents/", zip,"/pg-", i)

    page <- read_html(url)

    df <- data.frame(AgentID = page %>% 
                               html_nodes(xpath = '//*[@id="call_inquiry_cta"]') %>% 
                               xml_attr("data-agent-id"),
                     AgentName = page %>% 
                               html_nodes(xpath = '//*[@id="call_inquiry_cta"]') %>% 
                               xml_attr("data-agent-name"),
                     AgentAddr = page %>% 
                               html_nodes(xpath = '//*[@id="call_inquiry_cta"]') %>% 
                               xml_attr("data-agent-address"),
                     AgentPhone = sub("tel:", "", page %>% 
                                                  html_nodes(xpath = '//*[@id="call_inquiry_cta"]') %>% 
                                                  xml_attr("href")),
                     PhoneType = page %>% 
                                 html_nodes(xpath = '//*[@id="call_inquiry_cta"]') %>% 
                                 xml_attr("data-agent-num-type"),
                     AgentWebSite = page %>% 
                                    html_nodes(xpath = '//*[@id="call_inquiry_cta"]') %>% 
                                    xml_attr("data-agent-web-url"))

    results[[result.index]] <- df
  }
}

df <- do.call(rbind, results)

推荐阅读