这篇博客是之前文章：
- Elasticsearch：使用 Open AI 和 Langchain 的 RAG - Retrieval Augmented Generation （一）
- Elasticsearch：使用 Open AI 和 Langchain 的 RAG - Retrieval Augmented Generation （二）
-
Elasticsearch：使用 Open AI 和 Langchain 的 RAG - Retrieval Augmented Generation （三）
的续篇。在这篇文章中，我们将学习如何把从 Elasticsearch 搜索到的结果传递到大数据模型以得到更好的结果。
如果你还没有创建好自己的环境，请参考第一篇文章进行详细地安装。
针对大文本的文档，我们可以采用如下的架构：
创建应用并展示
安装包
#!pip3 install langchain导入包
from dotenv import load_dotenv
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import ElasticsearchStore
from langchain.text_splitter import CharacterTextSplitter
from langchain.prompts import ChatPromptTemplate
from langchain.prompts import PromptTemplate
from langchain.chat_models import ChatOpenAI
from langchain.schema.output_parser import StrOutputParser
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.runnable import RunnableLambda
from langchain.schema import HumanMessage
from urllib.request import urlopen
import os, json
load_dotenv()
openai_api_key=os.getenv('OPENAI_API_KEY')
elastic_user=os.getenv('ES_USER')
elastic_password=os.getenv('ES_PASSWORD')
elastic_endpoint=os.getenv("ES_ENDPOINT")
elastic_index_name='langchain-rag'添加文档并将文档分成段落
with open('workplace-docs.json') as f:
workplace_docs = json.load(f)
print(f"Successfully loaded {len(workplace_docs)} documents")
metadata = []
content = []
for doc in workplace_docs:
content.append(doc["content"])
metadata.append({
"name": doc["name"],
"summary": doc["summary"],
"rolePermissions":doc["rolePermissions"]
})
text_splitter = CharacterTextSplitter(chunk_size=50, chunk_overlap=0)
docs = text_splitter.create_documents(content, metadatas=metadata)
Index Documents using ELSER - SparseVectorRetrievalStrategy()
from elasticsearch import Elasticsearch
url = f"https://{elastic_user}:{elastic_password}@{elastic_endpoint}:9200"
connection = Elasticsearch(url, ca_certs = "./http_ca.crt", verify_certs = True)
es = ElasticsearchStore.from_documents(
docs,
es_url = url,
es_connection = connection,
es_user=elastic_user,
es_password=elastic_password,
index_name=elastic_index_name,
strategy=ElasticsearchStore.SparseVectorRetrievalStrategy()
)
如果你还没有配置好自己的 ELSER，请参考之前的文章 “ Elasticsearch：使用 Open AI 和 Langchain 的 RAG - Retrieval Augmented Generation （三）”。
在执行完上面的命令后，我们可以在 Kibana 中进行查看：
展示结果
def showResults(output):
print("Total results: ", len(output))
for index in range(len(output)):
print(output[index])Search
r = es.similarity_search("work from home policy")
showResults(r)
RAG with Elasticsearch - Method 1 (Using Retriever)
retriever = es.as_retriever(search_kwargs={"k": 4})
template = """Answer the question based only on the following context:
{context}
Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)
chain = (
{"context": retriever, "question": RunnablePassthrough()}
| prompt
| ChatOpenAI()
| StrOutputParser()
)
chain.invoke("vacation policy")
RAG with Elasticsearch - Method 2 (Without Retriever)
Add Context
def add_context(question: str):
r = es.similarity_search(question)
context = "\n".join(x.page_content for x in r)
return contextChain
template = """Answer the question based only on the following context:
{context}
Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)
chain = (
{"context": RunnableLambda(add_context), "question": RunnablePassthrough()}
| prompt
| ChatOpenAI()
| StrOutputParser()
)
chain.invoke("canada employees guidelines")
Compare with RAG and without RAG
q = input("Ask Question: ")
## Question to OpenAI
chat = ChatOpenAI()
messages = [
HumanMessage(
content=q
)
]
gpt_res = chat(messages)
# Question with RAG
gpt_rag_res = chain.invoke(q)
# Responses
s = f"""
ChatGPT Response:
{gpt_res}
ChatGPT with RAG Response:
{gpt_rag_res}
"""
print(s)
文章来源：https://www.toymoban.com/news/detail-755428.html
上面的 jupyter notebook 的代码可以在地址 https://github.com/liu-xiao-guo/semantic_search_es/blob/main/RAG-langchain-elasticsearch.ipynb 下载。文章来源地址https://www.toymoban.com/news/detail-755428.html
到了这里，关于Elasticsearch：使用 Open AI 和 Langchain 的 RAG - Retrieval Augmented Generation （四）的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！
