java - 解码 html 响应
问题描述
我正在尝试从以下网页中抓取一些数据:https ://bitcoin.pl/ 我收到来自服务器的响应并提取正文。我想从正文中提取链接。但是,不能这样做,因为 body 没有被正确解码并且包含转义字符。
我尝试了以下一些解决方案:
How to unescape HTML character entity in Java?
https://howtodoinjava.com/java/string/unescape-html-to-string/
下面我提供我写的代码:
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class test_scraper {
public static void main(String[] args) throws Exception {
final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36";
Unirest.setDefaultHeader("User-Agent",USER_AGENT);
final HttpResponse<String> response = Unirest.post("https://bitcoin.pl/?ajax-request=jnews")
.header("Accept","application/json, text/javascript, */*; q=0.01")
.header("Content-Type","application/x-www-form-urlencoded; charset=UTF-8")
.header("Referer","https://bitcoin.pl/")
.header("user-agent",USER_AGENT)
.header("accept-language", "en-US,en;q=0.9")
.header("X-Requested-With","XMLHttpRequest")
.header("accept-encoding:", "gzip, deflate, br")
.queryString("lang","pl_PL")
.queryString("action","jnews_module_ajax_jnews_block_5")
.queryString("data[current_page]",2)
.queryString("data[attribute][number_post]", 1)
.asString();
//System.out.println(response.getHeaders());
//System.out.println(response.getBody());
final Document html = Jsoup.parseBodyFragment(response.getBody());
System.out.println(Jsoup.parse(response.getBody()));
}
}
我收到了与浏览器完全相同的响应(检查器模式 -> 网络 -> XHR -> 响应)但是我想从已经解码的预览中获取 HTML。
我收到的是(部分回复):
<html>
<head></head>
<body>
{"content":"
<div class="\"jeg_posts" jeg_load_more_flag\">
\n
<article class="\"jeg_post" jeg_pl_lg_2 post-9771 post type-post status-publish format-standard has-post-thumbnail hentry category-kryptowaluty tag-bitcoin tag-chinski-bank-ludowy tag-chinski-banki-centralny tag-chiny tag-cyfrowa-waluta tag-libra tag-token-pboc\">
\n
<div class="\"jeg_thumb\"">
\n \n
<a href="\"https:\/\/bitcoin.pl\/chiny-data-emisji-waluty\/\""></a>
<div class="\"thumbnail-container" animate-lazy size-715 \">
<a href="\"https:\/\/bitcoin.pl\/chiny-data-emisji-waluty\/\""><img width="\"350\"" height="\"250\"" src="\"https:\/\/bitcoin.pl\/wp-content\/themes\/jnews\/assets\/img\/jeg-empty.png\"" class="\"attachment-jnews-350x250" size-jnews-350x250 lazyload wp-post-image\" alt="\"chiny\"" data-src="\"https:\/\/bitcoin.pl\/wp-content\/uploads\/2019\/09\/chiny-350x250.jpg\"" data-sizes="\"auto\"" data-srcset="\"https:\/\/bitcoin.pl\/wp-content\/uploads\/2019\/09\/chiny-350x250.jpg" 350w, https:\ \ bitcoin.pl\ wp-content\ uploads\ 2019\ 09\ chiny-120x86.jpg 120w, chiny-750x536.jpg 750w\" data-expand="\"700\"" data-animate="\"0\""><\/div><\/a>\n
<div class="\"jeg_post_category\"">
\n
<span><a href="\"https:\/\/bitcoin.pl\/category\/kryptowaluty\/\"" class="\"category-kryptowaluty\"">Kryptowaluty<\/a><\/span>\n <\/div>\n <\/div>\n </a>
如何正确解码以上以获得正确的 HTML?
解决方案
该服务返回一个 JSON:
{
"content": "<div class=...",
"next": false,
"prev": true
}
不需要Jsoup ,因为此 HTML 嵌入到 JSON 对象中。改用杰克逊:
ObjectMapper mapper = new ObjectMapper();
Map map = mapper.readValue(body, Map.class);
String content = map.get("content").toString();
System.out.println(content);
你会得到没有任何转义的普通 HTML:
<div class="jeg_posts jeg_load_more_flag">
<article class="jeg_post ...
<div class="jeg_thumb">
...
上面的类ObjectMapper
是com.fasterxml.jackson.databind.ObjectMapper
,不要将它与 Unirest 的类似类混淆。
要使用 Jackson,请将以下依赖项添加到您的 Gradle 文件中,在 Maven 中是类似的:
implementation 'com.fasterxml.jackson.core:jackson-databind:2.10.0.pr3'
推荐阅读
- css - JavaFX CornerRadiiConverter - 它在哪里?
- flutter - 容器中的子图像不遵循容器边框
- vue.js - 如何创建一个端点到 axios POST
- algorithm - 使用一次对 n 个元素进行排序的函数对 2n 个元素的数组进行排序
- javascript - 递归方式中层次树的构造逻辑
- spring - 数据未使用 H2 DB 从 Spring Boot 返回
- c++ - “const char *”类型的参数与“char *”类型的参数不兼容
- java - 有没有办法自动重新加载列表而不会避免闪烁?
- groovy - Groovy:读取属性文件,然后用原始注释写入文件
- windows-10 - 从 Docker 修剪未使用的图像后无法释放空间