时间戳

首页 > 解决方案 > 获取所有可见的纯文本并找出每段文本属于哪个 HTML 标记或 DOM 元素

puppeteer - 获取所有可见的纯文本并找出每段文本属于哪个 HTML 标记或 DOM 元素

问题描述

我知道如何在页面上获取所有可见的纯文本：

const text = await page.$eval('*', el => el.innerText);

但我还需要知道每段文本属于页面的哪个元素，但我找不到办法做到这一点。

标签： puppeteer

解决方案

在客户端，您可以使用TreeWalker以保留顺序的方式执行此操作。这是来自Web Scraper Testing Ground的示例内容示例：

const IGNORE = ["style", "script"];

const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);

const pairs = [];

let node;

while ((node = walker.nextNode()) !== null) {
  const parent = node.parentNode.tagName;

  if (IGNORE.includes(parent)) {
    continue;
  }

  const value = node.nodeValue.trim();

  if (value.length === 0) {
    continue;
  }

  pairs.push([parent.toLowerCase(), value]);
}

console.log(pairs);
<div id="topbar"></div>
		<a href="/" style="text-decoration: none">
		    <div id="title">WEB SCRAPER TESTING GROUND</div>
		    <div id="logo"></div>
		</a>
		<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
	<ol>
		<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
		<li>Scrape discounted products only</li>
		<li>Scrape products with red prices only</li>
	</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
	</p><ul>
		<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
		</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
	</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
	<li><a href="/blocks?ver=1">Price list 1</a></li>
	<li><a href="/blocks?ver=2">Price list 2</a></li>
	<li><a href="/blocks?ver=3">Price list 3</a></li>
	<li><a href="/blocks?ver=4">Price list 4</a></li>
	<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>

<div id="case_blocks">

<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>

<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>

</div>
<br><br><br>
		</div>

evaluate根据格兰特米勒的回答，在 Puppeteer 中使用它来称呼它：

const pairs = await page.evaluate(() => {
  const IGNORE = ["style", "script"];
  const NONWHITESPACE_RE = /\S/;

  const result = document.evaluate(
    "//*[child::text()]",
    document,
    null,
    XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
    null
  );

  const pairs = [];

  for (let i = 0, j = result.snapshotLength; i < j; i++) {
    const element = result.snapshotItem(i);

    if (IGNORE.includes(element.tagName.toLowerCase())) {
      continue;
    }

    const nodes = [...element.childNodes];

    for (const node of nodes) {
      if (node.nodeType !== document.TEXT_NODE) {
        continue;
      }

      if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
        continue;
      }

      pairs.push({
        tag: element.tagName.toLowerCase(),
        text: node.nodeValue.trim()
      });
    }
  }

  return pairs;
});

console.log(pairs);

这是客户端函数的原始版本，它使用 XPath，但总是将节点的直接子节点放在其间接子节点之前：

const IGNORE = ["style", "script"];
const NONWHITESPACE_RE = /\S/;

// get all text nodes in the document
const result = document.evaluate(
  // matches any node in the document that has at least one direct
  // text node child, including whitespace-only nodes
  "//*[child::text()]",
  document,
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null
);

// the result doesn't use the JavaScript iterator protocol, so we have
// to manually iterate over the elements
const pairs = [];

for (let i = 0, j = result.snapshotLength; i < j; i++) {
  const element = result.snapshotItem(i);

  if (IGNORE.includes(element.tagName.toLowerCase())) {
    continue;
  }

  const nodes = [...element.childNodes];

  for (const node of nodes) {
    if (node.nodeType !== document.TEXT_NODE) {
      continue;
    }

    // filter out whitespace-only nodes
    if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
      continue;
    }

    pairs.push({
      tag: element.tagName.toLowerCase(),
      // remove the `.trim()` to preserve leading & trailing whitespace
      text: node.nodeValue.trim()
    });
  }
}

console.log(pairs);
        <div id="topbar"></div>
		<a href="/" style="text-decoration: none">
		    <div id="title">WEB SCRAPER TESTING GROUND</div>
		    <div id="logo"></div>
		</a>
		<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
	<ol>
		<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
		<li>Scrape discounted products only</li>
		<li>Scrape products with red prices only</li>
	</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
	</p><ul>
		<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
		</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
	</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
	<li><a href="/blocks?ver=1">Price list 1</a></li>
	<li><a href="/blocks?ver=2">Price list 2</a></li>
	<li><a href="/blocks?ver=3">Price list 3</a></li>
	<li><a href="/blocks?ver=4">Price list 4</a></li>
	<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>

<div id="case_blocks">

<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>

<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>

</div>
<br><br><br>
		</div>

推荐阅读