swift - 使用正则表达式或替代方法快速抓取网页
问题描述
先看下面的更新。
我正在尝试在 reddit 上为指定的子 reddit 抓取所有版主。API 仅允许您获取子 reddit 的所有版主用户名,因此最初我已获取所有这些用户名,然后对每个配置文件执行额外请求以获取头像 url。这最终超出了 API 限制。
因此,我只想获取下一页的来源并在收集每页上的 10 个用户名和头像 url 的同时进行分页。这将最终以更少的请求轮询网站。我了解如何进行分页部分,但现在我正在尝试了解如何收集用户名和相邻的头像 URL。
所以采取以下网址:
https://www.reddit.com/r/videos/about/moderators/
所以我会拉取整个页面源码,
将所有 mods 用户名和 url 添加到 mod 对象中,然后添加到数组中。
在我回来的字符串上使用正则表达式是个好主意吗?
到目前为止,这是我的代码,任何帮助都会很棒:
func tester() {
let url = URL(string: "https://www.reddit.com/r/videos/about/moderators")!
let task = URLSession.shared.dataTask(with: url) { data, response, error in
guard let data = data, error == nil else {
print("\(error)")
return
}
let string = String(data: data, encoding: .utf8)
let regexUsernames = try? NSRegularExpression(pattern: "href=\"/user/[a-z0-9]\"", options: .caseInsensitive)
var results = regexUsernames?.matches(in: string as String, options: [], range: NSRange(location: 0, length: string.length))
let regexProfileURLs = try? NSRegularExpression(pattern: "><img src=\"[a-z0-9]\" style", options: .caseInsensitive)
print("\(results)") // This shows as empty array
}
task.resume()
}
我也尝试了以下但得到这个错误:
Can't form Range with upperBound < lowerBound
代码:
func tester() {
let url = URL(string: "https://www.reddit.com/r/videos/about/moderators")!
let task = URLSession.shared.dataTask(with: url) { data, response, error in
guard let data = data, error == nil else {
print("data was nil")
return
}
guard let htmlString = String(data: data, encoding: .utf8) else {
print("cannot cast data into string")
return
}
let leftSideOfValue = "href=\"/user/"
let rightSideOfValue = "\""
guard let leftRange = htmlString.range(of: leftSideOfValue) else {
print("cannot find range left")
return
}
guard let rightRange = htmlString.range(of: rightSideOfValue) else {
print("cannot find range right")
return
}
let rangeOfTheValue = leftRange.upperBound..<rightRange.lowerBound
print(htmlString[rangeOfTheValue])
}
更新:
所以我已经到了它会给我第一个用户名的地步,但是我正在循环并且一遍又一遍地得到相同的用户名。推进每个增量步骤的最佳方式是什么?有没有办法像 let newHTMLString = htmlString.dropFirst(k: ?) 用我们刚刚得到的元素之后的子字符串替换 htmlString ?
func tester() {
let url = URL(string: "https://www.reddit.com/r/pics/about/moderators")!
let task = URLSession.shared.dataTask(with: url) { data, response, error in
guard let data = data, error == nil else {
print("data was nil")
return
}
guard let htmlString = String(data: data, encoding: .utf8) else {
print("cannot cast data into string")
return
}
let counter = htmlString.components(separatedBy:"href=\"/user/")
let count = counter.count
for i in 0...count {
let leftSideOfUsernameValue = "href=\"/user/"
let rightSideOfUsernameValue = "\""
let leftSideOfAvatarURLValue = "><img src=\""
let rightSideOfAvatarURLValue = "\">"
guard let leftRange = htmlString.range(of: leftSideOfUsernameValue) else {
print("cannot find range left")
return
}
guard let rightRange = htmlString.range(of: rightSideOfUsernameValue) else {
print("cannot find range right")
return
}
let username = htmlString.slice(from: leftSideOfUsernameValue, to: rightSideOfUsernameValue)
print(username)
guard let avatarURL = htmlString.slice(from: leftSideOfAvatarURLValue, to: rightSideOfAvatarURLValue) else {
print("Error")
return
}
print(avatarURL)
}
}
task.resume()
}
我也试过:
let endString = String(avatarURL + rightSideOfAvatarURLValue)
let endIndex = htmlString.index(endString.endIndex, offsetBy: 0)
let substringer = htmlString[endIndex...]
htmlString = String(substringer)
解决方案
您应该能够通过执行以下操作调用简单的正则表达式,将所有名称和 url 拉入两个单独的数组:
func tester() {
let url = URL(string: "https://www.reddit.com/r/pics/about/moderators")!
let task = URLSession.shared.dataTask(with: url) { data, response, error in
guard let data = data, error == nil else { return }
guard let htmlString = String(data: data, encoding: .utf8) else { return }
let names = htmlString.matching(regex: "href=\"/user/(.*?)\"")
let imageUrls = htmlString.matching(regex: "><img src=\"(.*?)\" style")
print(names)
print(imageUrls)
}
task.resume()
}
extension String {
func matching(regex: String) -> [String] {
guard let regex = try? NSRegularExpression(pattern: regex, options: []) else { return [] }
let result = regex.matches(in: self, options: [], range: NSMakeRange(0, self.count))
return result.map {
return String(self[Range($0.range, in: self)!])
}
}
}
或者,您可以为每个对象创建一个对象,<div class="_1sIhmckJjyRyuR_z7M5kbI">
然后根据需要获取要使用的名称和 URL。
推荐阅读
- swift - Swift 排序的数组总是按升序排列
- javascript - React/ES6 减少对象
- java - 我无法在 Android Studio 上启动模拟器
- javascript - 未捕获的错误:意外的行:第 541 行的 THREE.OBJLoader.parse 中的“n 0 0 1”
- c# - 在 asp.net 身份中向帐户控制器中的登录操作添加附加参数的正确方法是什么?
- angular - 将对象成员分配给上下文菜单命令 - TypeScript
- redis - redis-server 命令行选项的文档
- javascript - 单击滚动顶部按钮时如何消除页面上的这种白色效果
- html - 在多个影子主机之间共享同一个影子 DOM?
- css - 如何在 Tab 组件中更改 Icon 组件的位置?