序
本文主要研究一下elasticsearch的data-streams
data-streams
主要特性
- 首先data streams是由一个或者多个自动生成的隐藏索引组成的，它的格式为
.ds-<data-stream>-<yyyy.MM.dd>-<generation>
示例.ds-web-server-logs-2099.03.07-000034，generation是一个6位的数字，默认从
000001开始
-
必须包含@timestamp字段，映射为date或者date_nanos字段类型，如果index template没有定义类型的话，则elasticsearch默认将其定义为date类型
-
读请求会自动路由到关联到的所有索引，而写请求的话则是添加到最新的索引，旧的索引不支持添加数据
-
rollover会根据指定条件来创建新索引，一般是推荐使用ILM自动取rollover
使用
创建mappings和settings
# Creates a component template for mappings
PUT _component_template/my-mappings
{
"template": {
"mappings": {
"properties": {
"@timestamp": {
"type": "date",
"format": "date_optional_time||epoch_millis"
},
"message": {
"type": "wildcard"
}
}
}
},
"_meta": {
"description": "Mappings for @timestamp and message fields",
"my-custom-meta-field": "More arbitrary metadata"
}
}
# Creates a component template for index settings
PUT _component_template/my-settings
{
"template": {
"settings": {
"index.lifecycle.name": "my-lifecycle-policy"
}
},
"_meta": {
"description": "Settings for ILM",
"my-custom-meta-field": "More arbitrary metadata"
}
}
主要是利用_component_template创建mappings和settings，方面下面创建index_template使用
创建index template
PUT _index_template/my-index-template
{
"index_patterns": ["my-data-stream*"],
"data_stream": { },
"composed_of": [ "my-mappings", "my-settings" ],
"priority": 500,
"_meta": {
"description": "Template for my time series data",
"my-custom-meta-field": "More arbitrary metadata"
}
}
创建data stream
PUT /_data_stream/my-data-stream-1/
查询data stream
GET /_data_stream/my-data-stream-1
{
"data_streams": [
{
"name": "my-data-stream-1",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-1-2023.08.06-000001",
"index_uuid": "ByCb4bPGSEOXfVf3Txpiiw"
}
],
"generation": 1,
"_meta": {
"my-custom-meta-field": "More arbitrary metadata",
"description": "Template for my time series data"
},
"status": "YELLOW",
"template": "my-data-stream",
"ilm_policy": "my-lifecycle-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false
}
]
}
创建数据
POST my-data-stream-1/_doc
{
"@timestamp": "2099-05-06T16:21:15.000Z",
"message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
返回
{
"_index": ".ds-my-data-stream-1-2023.08.06-000001",
"_id": "bHTfyIkBwVE4kI2xm5nL",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
查询索引数据
POST my-data-stream-1/_search
{ "query": { "match_all": {} } }
filebeat
filebeat默认output到elasticsearch创建的就是data streams，如果不想使用其自动加载的模版，则可以设置setup.template.enabled=false，那么创建的则是普通的index。文章来源：https://www.toymoban.com/news/detail-691503.html
小结
elasticsearch7.9版本以xpack的形式推出了data streams，主要是针对持续产生的时间序列数据提供了一种更为简单的方式去对索引进行数据切分和统一查询的方式。文章来源地址https://www.toymoban.com/news/detail-691503.html
doc
- data-streams
到了这里，关于聊聊elasticsearch的data-streams的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！
