python - 使用稀疏向量进行查询的 Elasticsearch 运行时错误

问题描述

最近，Elasticsearch 实现了基于向量的查询。这意味着每个文档都包含一个向量作为字段，我们可以使用一个新向量在我们的语料库中找到匹配项。

您可以在此链接中找到更多信息。Elasticsearch 团队在那里解释了它应该如何工作，甚至提供了一个查询字符串：

{
  "query": {
    "script_score": {
      "query": {
        "match_all": {}
      },
      "script": {
        "source": "cosineSimilaritySparse(params.queryVector, doc['my_sparse_vector'])",
        "params": {
          "queryVector": {"2": 0.5, "10" : 111.3, "50": -1.3, "113": 14.8, "4545": 156.0}
        }
      }
    }
  }
}

我已经安装了最新的 Elasticsearch 版本，特别是curl -XGET 'http://localhost:9200'给了我这个信息：

"version" : {
"number" : "7.3.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "de777fa",
"build_date" : "2019-07-24T18:30:11.767338Z",
"build_snapshot" : false,
"lucene_version" : "8.1.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
}

我正在使用 Python 库elasticsearch（elasticsearch_dsl同样，但还没有用于这些查询）。我可以设置我的 Elasticsearch 索引、加载文档和进行查询。例如，这有效：

query_body = {
  "query": {
    "query_string": {
      "query": "Some text",
      "default_field": "some_field"
    }
  }
}

es.search(index=my_index, body=query_body)

但是，当我尝试使用与官方示例几乎相同的查询的相同代码时，它不起作用。

我的查询：

query_body = {
  "query": {
    "script_score": {
      "query": {
        "match_all": {}
      },
      "script": {
        "source": "cosineSimilaritySparse(params.queryVector, doc['my_embedding_field_name'])",
        "params": {
          "queryVector": {"1703": 0.0261, "1698": 0.0261, "2283": 0.0459, "2263": 0.0523, "3741": 0.0349}
        }
      }
    }
  }
}

请注意，查询中的稀疏向量是我制作的一个示例，确保至少在我的一个文档的嵌入向量中找到键（我不确定这应该有问题，但以防万一）。

错误：

elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'runtime error')

该错误消息对我的前进没有太大帮助，而且由于这是一个非常新的功能，我无法在网上找到其他帮助。

更新：下面是使用 curl 进行查询时产生的更完整的错误消息。

错误的核心是：

"type" : "illegal_argument_exception",
"reason" : "Variable [embedding] is not defined."

完整的消息是：

"error" : {
"root_cause" : [
  {
    "type" : "script_exception",
    "reason" : "compile error",
    "script_stack" : [
      "... (params.queryVector, doc[embedding])",
      "                             ^---- HERE"
    ],
    "script" : "cosineSimilaritySparse(params.queryVector, doc[embedding])",
    "lang" : "painless"
  },
  {
    "type" : "script_exception",
    "reason" : "compile error",
    "script_stack" : [
      "... (params.queryVector, doc[embedding])",
      "                             ^---- HERE"
    ],
    "script" : "cosineSimilaritySparse(params.queryVector, doc[embedding])",
    "lang" : "painless"
  }
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
  {
    "shard" : 0,
    "index" : "test-index",
    "node" : "216BQPYoQ-SIzcrV1jzMOQ",
    "reason" : {
      "type" : "query_shard_exception",
      "reason" : "script_score: the script could not be loaded",
      "index_uuid" : "e1kpygbHRai9UL8_0Lbsdw",
      "index" : "test-index",
      "caused_by" : {
        "type" : "script_exception",
        "reason" : "compile error",
        "script_stack" : [
          "... (params.queryVector, doc[embedding])",
          "                             ^---- HERE"
        ],
        "script" : "cosineSimilaritySparse(params.queryVector, doc[embedding])",
        "lang" : "painless",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Variable [embedding] is not defined."
        }
      }
    }
  },
  {
    "shard" : 0,
    "index" : "tutorial",
    "node" : "216BQPYoQ-SIzcrV1jzMOQ",
    "reason" : {
      "type" : "query_shard_exception",
      "reason" : "script_score: the script could not be loaded",
      "index_uuid" : "n2FNFgAFRiyB_efJKfsGPA",
      "index" : "tutorial",
      "caused_by" : {
        "type" : "script_exception",
        "reason" : "compile error",
        "script_stack" : [
          "... (params.queryVector, doc[embedding])",
          "                             ^---- HERE"
        ],
        "script" : "cosineSimilaritySparse(params.queryVector, doc[embedding])",
        "lang" : "painless",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Variable [embedding] is not defined."
        }
      }
    }
  }
],
"caused_by" : {
  "type" : "script_exception",
  "reason" : "compile error",
  "script_stack" : [
    "... (params.queryVector, doc[embedding])",
    "                             ^---- HERE"
  ],
  "script" : "cosineSimilaritySparse(params.queryVector, doc[embedding])",
  "lang" : "painless",
  "caused_by" : {
    "type" : "illegal_argument_exception",
    "reason" : "Variable [embedding] is not defined."
  }
} }, "status" : 400}

更新 2：我的文档具有以下结构：

{"name": "doc_name", "field_1": "doc_id", "field_2": "a_keyword", "text": "a rather long text", "embedding": {"4655": 0.040158602078116556, "4640": 0.040158602078116556}}

更新 3：我在创建索引后传递了一个映射，其中：

"properties": {
    "name": {
        "type": "keyword"
    },
    "field_1": {
        "type": "keyword"
    },
    "field_2": {
        "type": "keyword"
    },
    "text": {
        "type": "text"
    },
    "embedding": {
        "type": "sparse_vector"
    }
}

这消除了抱怨太多字段的错误（嵌入中的每个键都被视为一个字段）。但是查询错误是一样的。

标签： pythonelasticsearch

python - 使用稀疏向量进行查询的 Elasticsearch 运行时错误

问题描述

解决方案

推荐阅读