描述
数据量共约3000万+,在使用es进行term聚合的时候,发现执行耗费时间巨大,因此采用了msearch的检索方式
多搜索接口编辑 多搜索 API 从单个 API 请求执行多个搜索。 请求的格式类似于批量 API 格式,并使用 换行符分隔的 JSON (NDJSON) 格式。
结构类型于下
GET my-index-000001/_msearch
{ }
{"query" : {"match" : { "message": "this is a test"}}}
{"index": "my-index-000002"}
{"query" : {"match_all" : {}}}
kibana查询操作
使用kibana进行msearch操作
POST cqu_dev_journal_paper/_msearch
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"dfs_query_then_fetch","ccs_minimize_roundtrips":true}
{"from":10,"size":10,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id","sid","title","author","journalId","journalVolumeId","publisherName","doi","pageInfo"],"excludes":[]},"sort":[{"createTime":{"order":"desc"}}],"track_total_hits":2147483647}
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true}
{"from":0,"size":1,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id"],"excludes":[]},"aggregations":{"year":{"terms":{"field":"year","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}],"collect_mode":"breadth_first"}}}}
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true}
{"from":0,"size":1,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id"],"excludes":[]},"aggregations":{"subject":{"terms":{"field":"subject","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}],"collect_mode":"breadth_first"}}}}
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true}
{"from":0,"size":1,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id"],"excludes":[]},"aggregations":{"publisherName":{"terms":{"field":"publisherName.raw","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}],"collect_mode":"breadth_first"}}}}
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true}
{"from":0,"size":1,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id"],"excludes":[]},"aggregations":{"journalId":{"terms":{"field":"journalId","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}],"collect_mode":"breadth_first"}}}}
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true}
{"from":0,"size":1,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id"],"excludes":[]},"aggregations":{"publishCountry":{"terms":{"field":"publishCountry.raw","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}],"collect_mode":"breadth_first"}}}}
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"query_then_fetch","ccs_minimize_roundtrips":true}
{"from":0,"size":1,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id"],"excludes":[]},"aggregations":{"language":{"terms":{"field":"language.raw","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}],"collect_mode":"breadth_first"}}}}
语句解析
msearch查询的数据结构如下,
header\n
body\n
header\n
body\n
因此下面语句中第一个json就是header,第二个是请求的body,
{"index":["cqu_dev_journal_paper"],"types":[],"search_type":"dfs_query_then_fetch","ccs_minimize_roundtrips":true}
{"from":10,"size":10,"query":{"bool":{"must":[{"exists":{"field":"title","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"explain":false,"_source":{"includes":["id","sid","title","author","journalId","journalVolumeId","publisherName","doi","pageInfo"],"excludes":[]},"sort":[{"createTime":{"order":"desc"}}],"track_total_hits":2147483647}
官网对于请求的说明:
地址: es 多搜索接口
请求正文包含以换行符分隔的搜索列表和 搜索对象。
<header> (必填,对象) 用于限制或更改搜索的参数。
此对象对于每个搜索正文都是必需的,但可以为空 () 或空白 线。{}
<body>
(可选,对象) 包含搜索请求的参数:
对象的属性<body>
aggregations
(可选,聚合对象) 您希望在搜索期间运行的聚合。请参阅聚合。
query
(可选,查询 DSL 对象)您希望在 搜索。响应中将返回与此查询匹配的命中。
from
(可选,整数) 返回命中的起始偏移量。默认值为 。0
size
(可选,整数) 要返回的命中数。默认值为 。10
代码操作验证对比
使用SpringDataElaticsearch进行对es检索的操作,代码如下:
MultiSearch 检索
//MultiSearch 查询
List<Query> queryList = new ArrayList<>();
NativeSearchQueryBuilder listNativeSearchQueryBuilder = new NativeSearchQueryBuilder();
//返回数据总数量
listNativeSearchQueryBuilder.withTrackTotalHits(true);
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must(QueryBuilders.existsQuery("title"));
//设置检索词和聚合项的查询条件
listNativeSearchQueryBuilder.withQuery(boolQueryBuilder);
//构造分页排序
Pageable pageable = PageRequest.of(1, 10, Sort.by(Sort.Direction.DESC, "createTime"));
listNativeSearchQueryBuilder.withPageable(pageable);
//设置返回字段
listNativeSearchQueryBuilder.withFields("id", "sid", "title", "author", "journalId", "journalVolumeId", "publisherName", "doi", "pageInfo");
//列表查询的query放入查询列表
queryList.add(listNativeSearchQueryBuilder.build());
HashMap<String, String> aggFieldMap = new HashMap<>();
aggFieldMap.put("subject", "subject");
aggFieldMap.put("year", "year");
aggFieldMap.put("journalId", "journalId");
aggFieldMap.put("publisherName", "publisherName.raw");
aggFieldMap.put("language", "language.raw");
aggFieldMap.put("publishCountry", "publishCountry.raw");
NativeSearchQueryBuilder aggNativeSearchQueryBuilder;
for (String name : aggFieldMap.keySet()) {
aggNativeSearchQueryBuilder = new NativeSearchQueryBuilder();
aggNativeSearchQueryBuilder.withQuery(boolQueryBuilder);
//表示不返回列表数据
aggNativeSearchQueryBuilder.withPageable(PageRequest.of(0, 1));
aggNativeSearchQueryBuilder.withSearchType(SearchType.QUERY_THEN_FETCH);
aggNativeSearchQueryBuilder.withFields("id");
aggNativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms(name).field(aggFieldMap.get(name))
.collectMode(Aggregator.SubAggCollectionMode.BREADTH_FIRST).size(50));
queryList.add(aggNativeSearchQueryBuilder.build());
}
long pageStart = System.currentTimeMillis();
//multiSearch:允许在一次请求中执行多个查询操作,并将查询结果一起返回
List<SearchHits<JournalPaperEsBean>> searchHitsList = elasticsearchOperations.multiSearch(queryList, JournalPaperEsBean.class);
System.out.println("==========es agg search:" + (System.currentTimeMillis() - pageStart));
}
执行耗费时间:3723ms
search检索
@Test
void test() {
//search 查询
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
//返回数据总数量
nativeSearchQueryBuilder.withTrackTotalHits(true);
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must(QueryBuilders.existsQuery("title"));
//设置检索词和聚合项的查询条件
nativeSearchQueryBuilder.withQuery(boolQueryBuilder);
//构造分页排序
Pageable pageable = PageRequest.of(1, 10, Sort.by(Sort.Direction.DESC, "createTime"));
nativeSearchQueryBuilder.withPageable(pageable);
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("subject").field("subject").size(50));
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("year").field("year").size(50).order(BucketOrder.key(false)));
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("journalId").field("journalId").size(50));
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("publisherName").field("publisherName.raw").size(50));
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("language").field("language.raw").size(50));
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("publishCountry").field("publishCountry.raw").size(50));
//设置返回字段
nativeSearchQueryBuilder.withFields("id", "sid", "title", "author", "journalId", "journalVolumeId", "publisherName", "doi", "pageInfo");
long start = System.currentTimeMillis();
//执行检索操作
SearchHits<JournalPaperEsBean> searchHits = elasticsearchOperations.search(nativeSearchQueryBuilder.build(), JournalPaperEsBean.class);
System.out.println("=====agg:" + (System.currentTimeMillis() - start));
}
耗时:15298ms文章来源:https://www.toymoban.com/news/detail-521961.html
对比可以发现:使用msearch比search效率提高了近5倍!文章来源地址https://www.toymoban.com/news/detail-521961.html
到了这里,关于Elasticsearch使用msearch提高聚合效率(与search检索对比)的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!