Elasticsearch：使用 huggingface 模型的 NLP 文本搜索-Toy模板网

这篇具有很好参考价值的文章主要介绍了Elasticsearch：使用 huggingface 模型的 NLP 文本搜索。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

本博文使用由 Elastic 博客 title 组成的简单数据集在 Elasticsearch 中实现 NLP 文本搜索。你将为博客文档建立索引，并使用摄取管道生成文本嵌入。通过使用 NLP 模型，你将使用自然语言在博客文档上查询文档。

安装

Elasticsearch 及 Kibana

如果你还没有安装好自己的 Elasticsearch 及 Kibana，请参考如下的链接来进行安装：

如何在 Linux，MacOS 及 Windows 上进行安装 Elasticsearch
Kibana：如何在 Linux，MacOS 及 Windows 上安装 Elastic 栈中的 Kibana

在安装的时候，我们可以选择 Elastic Stack 8.x 的安装指南来进行安装。在本博文中，我将使用最新的 Elastic Stack 8.10 来进行展示。

在安装 Elasticsearch 的过程中，我们需要记下如下的信息：

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

由于我们将要使用到 eland 来上传模型。这是一个收费的功能。我们需要启动试用功能：

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

Python 安装包

在本演示中，我们将使用 Python 来进行展示。我们需要安装访问 Elasticsearch 相应的安装包 elasticsearch：

python3 -m pip install -qU sentence-transformers eland elasticsearch transformers

我们将使用 Jupyter Notebook 来进行展示。

$ pwd
/Users/liuxg/python/elser
$ jupyter notebook

准备数据

我们在项目的根目录下，创建如下的一个数据文件: data.json：

data.json

[
   {
      "title":"Pulp Fiction",
      "runtime":"154",
      "plot":"The lives of two mob hitmen, a boxer, a gangster and his wife, and a pair of diner bandits intertwine in four tales of violence and redemption.",
      "keyScene":"John Travolta is forced to inject adrenaline directly into Uma Thurman's heart after she overdoses on heroin.",
      "genre":"Crime, Drama",
      "released":"1994"
   },
   {
      "title":"The Dark Knight",
      "runtime":"152",
      "plot":"When the menace known as the Joker wreaks havoc and chaos on the people of Gotham, Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice.",
      "keyScene":"Batman angrily responds 'I’m Batman' when asked who he is by Falcone.",
      "genre":"Action, Crime, Drama, Thriller",
      "released":"2008"
   },
   {
      "title":"Fight Club",
      "runtime":"139",
      "plot":"An insomniac office worker and a devil-may-care soapmaker form an underground fight club that evolves into something much, much more.",
      "keyScene":"Brad Pitt explains the rules of Fight Club to Edward Norton. The first rule of Fight Club is: You do not talk about Fight Club. The second rule of Fight Club is: You do not talk about Fight Club.",
      "genre":"Drama",
      "released":"1999"
   },
   {
      "title":"Inception",
      "runtime":"148",
      "plot":"A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into thed of a C.E.O.",
      "keyScene":"Leonardo DiCaprio explains the concept of inception to Ellen Page by using a child's spinning top.",
      "genre":"Action, Adventure, Sci-Fi, Thriller",
      "released":"2010"
   },
   {
      "title":"The Matrix",
      "runtime":"136",
      "plot":"A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.",
      "keyScene":"Red pill or blue pill? Morpheus offers Neo a choice between the red pill, which will allow him to learn the truth about the Matrix, or the blue pill, which will return him to his former life.",
      "genre":"Action, Sci-Fi",
      "released":"1999"
   },
   {
      "title":"The Shawshank Redemption",
      "runtime":"142",
      "plot":"Two imprisoned men bond over a number of years, finding solace and eventual redemption through acts of common decency.",
      "keyScene":"Andy Dufresne escapes from Shawshank prison by crawling through a sewer pipe.",
      "genre":"Drama",
      "released":"1994"
   },
   {
      "title":"Goodfellas",
      "runtime":"146",
      "plot":"The story of Henry Hill and his life in the mob, covering his relationship with his wife Karen Hill and his mob partners Jimmy Conway and Tommy DeVito in the Italian-American crime syndicate.",
      "keyScene":"Joe Pesci's character Tommy DeVito shoots young Spider in the foot for not getting him a drink.",
      "genre":"Biography, Crime, Drama",
      "released":"1990"
   },
   {
      "title":"Se7en",
      "runtime":"127",
      "plot":"Two detectives, a rookie and a veteran, hunt a serial killer who uses the seven deadly sins as his motives.",
      "keyScene":"Brad Pitt's character David Mills shoots John Doe after he reveals that he murdered Mills' wife.",
      "genre":"Crime, Drama, Mystery, Thriller",
      "released":"1995"
   },
   {
      "title":"The Silence of the Lambs",
      "runtime":"118",
      "plot":"A young F.B.I. cadet must receive the help of an incarcerated and manipulative cannibal killer to help catch another serial killer, a madman who skins his victims.",
      "keyScene":"Hannibal Lecter explains to Clarice Starling that he ate a census taker's liver with some fava beans and a nice Chianti.",
      "genre":"Crime, Drama, Thriller",
      "released":"1991"
   },
   {
      "title":"The Godfather",
      "runtime":"175",
      "plot":"An organized crime dynasty's aging patriarch transfers control of his clandestine empire to his reluctant son.",
      "keyScene":"James Caan's character Sonny Corleone is shot to death at a toll booth by a number of machine gun toting enemies.",
      "genre":"Crime, Drama",
      "released":"1972"
   },
   {
      "title":"The Departed",
      "runtime":"151",
      "plot":"An undercover cop and a mole in the police attempt to identify each other while infiltrating an Irish gang in South Boston.",
      "keyScene":"Leonardo DiCaprio's character Billy Costigan is shot to death by Matt Damon's character Colin Sullivan.",
      "genre":"Crime, Drama, Thriller",
      "released":"2006"
   },
   {
      "title":"The Usual Suspects",
      "runtime":"106",
      "plot":"A sole survivor tells of the twisty events leading up to a horrific gun battle on a boat, which began when five criminals met at a seemingly random police lineup.",
      "keyScene":"Kevin Spacey's character Verbal Kint is revealed to be the mastermind behind the crime, when his limp disappears as he walks away from the police station.",
      "genre":"Crime, Mystery, Thriller",
      "released":"1995"
   }
]

$ pwd
/Users/liuxg/python/elser
$ ls
Multilingual semantic search.ipynb
NLP text search using hugging face transformer model.ipynb
Semantic search - ELSER.ipynb
data.json

创建应用并演示

import modules

import pandas as pd, json
from elasticsearch import Elasticsearch
from getpass import getpass
from urllib.request import urlopen

部署 NLP 模型

我们将使用 eland 工具来安装 text_embedding 模型。对于我们的模型，我们使用 all-MiniLM-L6-v2 将搜索文本转换为密集向量。

该模型会将你的搜索查询转换为向量，该向量将用于对 Elasticsearch 中存储的文档集进行搜索。

我们在 terminal 中打入如下的命令：

eland_import_hub_model --url https://elastic:vXDWYtL*my3vnKY9zCfL@localhost:9200 \
      --hub-model-id sentence-transformers/all-MiniLM-L6-v2 \
      --task-type text_embedding \
      --ca-cert /Users/liuxg/elastic/elasticsearch-8.10.0/config/certs/http_ca.crt \
      --start

请注意：

我们需要根据自己的部署来替换上面的 elastic 超级用户的密码
我们需要根据自己的 Elasticsearch 集群的部署来替换上面的 Elasticsearch 访问地址
我们需要根据自己的部署的证书来替换上面的证书路径

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

我们回到 Kibana 的界面：

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

连接到 Elasticsearch

我们创建一个客户端连接：

ELASTCSEARCH_CERT_PATH = "/Users/liuxg/elastic/elasticsearch-8.10.0/config/certs/http_ca.crt"
 
es = Elasticsearch(  ['https://localhost:9200'],
    basic_auth = ('elastic', 'vXDWYtL*my3vnKY9zCfL'),
    ca_certs = ELASTCSEARCH_CERT_PATH,
    verify_certs = True)
print(es.info())

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

创建 ingest pipeline

我们需要创建一个文本嵌入提取管道来生成 title 字段的向量（文本）嵌入。

下面的管道定义了一个用于 NLP 模型的 inference 处理器。

# ingest pipeline definition
PIPELINE_ID="vectorize_blogs"

es.ingest.put_pipeline(id=PIPELINE_ID, processors=[{
        "inference": {
          "model_id": "sentence-transformers__all-minilm-l6-v2",
          "target_field": "text_embedding",
          "field_map": {
            "title": "text_field"
          }
        }
      }])

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

创建带有映射的索引

现在，在索引文档之前，我们将创建一个具有正确映射的 Elasticsearch 索引。我们添加 text_embedding 以包含 model_id 和 Predicted_value 来存储嵌入。

# define index name
INDEX_NAME="blogs"

# flag to check if index has to be deleted before creating
SHOULD_DELETE_INDEX=True

# define index mapping
INDEX_MAPPING = {
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      
      "text_embedding": {
        "properties": {
          "is_truncated": {
            "type": "boolean"
          },
          "model_id": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "predicted_value": {
            "type": "dense_vector",
            "dims": 384,
            "index": True,
            "similarity": "l2_norm"
          }
        }
      }
    }
  }

INDEX_SETTINGS = {
    "index": {
      "number_of_replicas": "1",
      "number_of_shards": "1",
      "default_pipeline": PIPELINE_ID
    }
}

# check if we want to delete index before creating the index
if(SHOULD_DELETE_INDEX):
  if es.indices.exists(index=INDEX_NAME):
    print("Deleting existing %s" % INDEX_NAME)
    es.options(ignore_status=[400, 404]).indices.delete(index=INDEX_NAME)

print("Creating index %s" % INDEX_NAME)
es.options(ignore_status=[400,404]).indices.create(index=INDEX_NAME, mappings=INDEX_MAPPING, settings=INDEX_SETTINGS)

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

摄入数据到 Elasticsearch

让我们使用摄取管道对示例博客数据进行索引。

注意：在我们开始索引之前，请确保你已启动训练模型部署。

from elasticsearch import helpers

# Load data into a JSON object
with open('data.json') as f:
   data_json = json.load(f)
 
print(data_json)
 
# Prepare the documents to be indexed
documents = []
for doc in data_json:
    documents.append({
        "_index": "blogs",
        "_source": doc,
    })
 
# Use helpers.bulk to index
helpers.bulk(es, documents)

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

我们可以回到 Kibana 的界面查看被写入的文档：

GET blogs/_search

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能

查询数据集

下一步是运行查询来搜索相关博客。该示例查询使用我们上传到 Elasticsearch Sentence-transformers__all-minilm-l6-v2 的模型来搜索 “model_text”: “scientific fiction”。

该过程是一个查询，尽管它内部包含两个任务。首先，查询将使用 NLP 模型为您的搜索文本生成一个向量，然后传递该向量以在数据集上进行搜索。

结果，输出显示按照与搜索查询的接近度排序的查询文档列表。

INDEX_NAME="blogs"

source_fields = [ "id", "title"]

query = {
  "field": "text_embedding.predicted_value",
  "k": 10,
  "num_candidates": 50,
  "query_vector_builder": {
    "text_embedding": {"model_id": "sentence-transformers__all-minilm-l6-v2",
      "model_text": "scientific fiction"
    }
  }
}

response = es.search(
    index=INDEX_NAME,
    fields=source_fields,
    knn=query,
    source=False)


results = pd.json_normalize(json.loads(json.dumps(response.body['hits']['hits'])))

# shows the result
results[['_id', '_score', 'fields.title']]

上面命令显示的结果为：

Elasticsearch：使用 huggingface 模型的 NLP 文本搜索,Elasticsearch,AI,Elastic,elasticsearch,自然语言处理,大数据,搜索引擎,全文检索,人工智能