时间戳

首页 > 解决方案 > Traditional pdf indexing solution compared to graph-based version

gremlin - Traditional pdf indexing solution compared to graph-based version

问题描述

My intention is to index an arbitrary directory containing pdf files (among other file types) with keywords stored in a list. I have a traditional solution and I heard that graph based solutions using e.g. SimpleGraph could be more elegant/efficient and independent of directory structures.

What would a graph-based solution (e.g. SimpleGraph) look like?

Traditional solution 

// https://stackoverflow.com/a/14051951/1497139
List<File> pdfFiles = this.explorePath(TestPDFFiles.RFC_DIRECTORY, "pdf");
List<PDFFile> pdfs = this.getPdfsFromFileList(pdfFiles);
…
for (PDFFile pdf:pdfs) {
     // https://stackoverflow.com/a/9560307/1497139
     if (org.apache.commons.lang3.StringUtils.containsIgnoreCase(pdf.getText(), keyWord)) {
          foundList.add(pdf.file.getName()); // here we access by structure (early binding)
          // - in the graph solution by name (late binding)
     }
}

标签： gremlinsimplegraph

解决方案

基本上使用 SimpleGraph 你会使用模块的组合

  1. 文件系统
  2. PDF系统

使用 FileSystem 模块，您可以收集目录中的文件图并过滤它以仅包含扩展名为 pdf 的文件 - 然后使用 PDFSystem 分析 PDF 以获取页面/文本结构 - 已经有一个测试用例simplegraph-bundle 模块展示了它如何使用一些 RFC pdf 作为输入。

TestPDFFiles.java

我现在添加了索引测试，见下文。

核心功能取自旧测试，搜索单个关键字并允许将其作为参数：

List<Object> founds = pdfSystem.g().V().hasLabel("page")
      .has("text", RegexPredicate.regex(".*" + keyWord + ".*")).in("pages")
      .dedup().values("name").toList();

这是一个 gremlin 查询，只需一次调用即可通过搜索整个 PDF 文件树来完成大部分工作。我认为这更优雅，因为您不必关心输入的结构（树/图形/文件系统/数据库等......）

JUnit 测试用例

 @Test
  /**
   * test for https://github.com/BITPlan/com.bitplan.simplegraph/issues/12
   */
  public void testPDFIndexing() throws Exception {
    FileSystem fs = getFileSystem(RFC_DIRECTORY);
    int limit = Integer.MAX_VALUE;
    PdfSystem pdfSystem = getPdfSystemForFileSystem(fs, limit);
    Map<String, List<String>> index = this.getIndex(pdfSystem, "ARPA",
        "proposal", "plan");
    // debug=true;
    if (debug) {
      for (Entry<String, List<String>> indexEntry : index.entrySet()) {
        List<String> fileNameList = indexEntry.getValue();
        System.out.println(String.format("%15s=%3d %s", indexEntry.getKey(),
            fileNameList.size(), fileNameList));
      }
    }
    assertEquals(14,index.get("ARPA").size());
    assertEquals(9,index.get("plan").size());
    assertEquals(8,index.get("proposal").size());
  }

推荐阅读