1. Toy模板网首页
  2. > Toy博客

【记录】Telegraf-解析nginx日志

10月前 作者：dnice 分类：Toy博客 阅读(26) 违法举报

这篇具有很好参考价值的文章主要介绍了【记录】Telegraf-解析nginx日志。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

本文档只记录解析nginx access.log相关内容，关于telegraf后期整理一篇文档单独介绍。

1、Tail Input Plugin

Tail Input Plugin 是 Telegraf项目 众多Inputs插件之一。
Tail Input Plugin 监听一个日志文件并解析该文件的每个日志变更。
默认情况下，tail 插件的行为类似于以下 Unix tail 命令：

	tail -F --lines=0 myfile.log

Telegraf收集nginx access.log 相当于Unix tail命令：

	tail -F --lines=0 access.log

并使用grok格式来解析日志。
除此之外，Telegraf Tail Input Plugin支持多种 Data Format
细节的东西可以跟下源码，下面直接记录运行配置。文章来源地址https://www.toymoban.com/news/detail-821832.html

2、配置

(1) telegraf.conf

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false

[[inputs.tail]]
  files = ["/Users/enic/github/telegraf/testdata/access.log"]
#   from_beginning = false
  from_beginning = true
  grok_patterns = ["%{COMMON_LOG_FORMAT}"]
  name_override = "nginx_access_log"
  grok_custom_pattern_files = []
  grok_custom_patterns = '''
      NGINX_ACCESS_LOG %{IP:remote_addr} - (-|%{WORD:remote_user}) \[%{HTTPDATE:time_local}\] %{QS:request} %{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{QS:referrer} %{QS:agent} %{IPORHOST:xforwardedfor} %{IPORHOST:host} %{QS:upstream_addr} (-|%{NUMBER:upstream_status:int}) %{BASE10NUM:request_time:float} (-|%{BASE10NUM:upstream_response_time:float}) %{QS:cookie_uin} %{QS:cookie_luin}
  '''
  grok_timezone = "Local"
  data_format = "grok"

[[outputs.file]]
  files = ["/Users/enic/github/telegraf/testdata/outputs.log"]

(2)access.log

10.10.10.10 - - [19/Apr/2022:03:17:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:44 +0800] "POST /qps HTTP/1.1" 504 3693 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:17:54 +0800] "POST /latency HTTP/1.1" 504 3693 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.100 - - [19/Apr/2022:03:18:37 +0800] "GET / HTTP/1.1" 200 1839 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" "-"
10.10.10.10 - - [19/Apr/2022:03:18:44 +0800] "POST /qps HTTP/1.1" 504 3693 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:54 +0800] "POST /latency HTTP/1.1" 504 3693 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:19:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"
10.10.10.10 - - [19/Apr/2022:03:18:34 +0800] "POST /user/login HTTP/1.1" 400 63 "-" "Go-http-client/1.1" "-"

3、操作

cd telegraf
make
./telegraf --config=/Users/enic/github/telegraf/testdata/telegraf.conf

4、output展示

nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",http_version=1.1,ident="-",auth="-",resp_bytes=63i,client_ip="10.10.10.10" 1650309454000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",resp_bytes=63i,client_ip="10.10.10.10",ident="-",auth="-",http_version=1.1 1650309454001000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",request="/user/login",auth="-",http_version=1.1,resp_bytes=63i,client_ip="10.10.10.10" 1650309454002000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,client_ip="10.10.10.10",ident="-",resp_bytes=63i,auth="-",request="/user/login" 1650309454003000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip="10.10.10.10",auth="-",ident="-",request="/user/login",http_version=1.1,resp_bytes=63i 1650309454004000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",auth="-",request="/user/login",resp_bytes=63i,client_ip="10.10.10.10",http_version=1.1 1650309454005000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST auth="-",request="/qps",http_version=1.1,resp_bytes=3693i,client_ip="10.10.10.10",ident="-" 1650309464000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST ident="-",auth="-",request="/latency",http_version=1.1,resp_bytes=3693i,client_ip="10.10.10.10" 1650309474000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth="-",request="/user/login",resp_bytes=63i,client_ip="10.10.10.10",http_version=1.1,ident="-" 1650309514000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",auth="-",request="/user/login",client_ip="10.10.10.10",resp_bytes=63i,http_version=1.1 1650309514001000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip="10.10.10.10",ident="-",auth="-",request="/user/login",http_version=1.1 1650309514002000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",resp_bytes=63i,ident="-",auth="-",http_version=1.1,client_ip="10.10.10.10" 1650309514003000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip="10.10.10.10",auth="-",ident="-",request="/user/login",http_version=1.1,resp_bytes=63i 1650309514004000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,ident="-",request="/user/login",client_ip="10.10.10.10",auth="-",http_version=1.1 1650309514005000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=200,verb=GET ident="-",request="/",resp_bytes=1839i,auth="-",http_version=1.1,client_ip="10.10.10.100" 1650309517000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST request="/qps",auth="-",ident="-",http_version=1.1,resp_bytes=3693i,client_ip="10.10.10.10" 1650309524000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST resp_bytes=3693i,auth="-",request="/latency",http_version=1.1,client_ip="10.10.10.10",ident="-" 1650309534000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,resp_bytes=63i,auth="-",client_ip="10.10.10.10",ident="-",request="/user/login" 1650309574000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",auth="-",resp_bytes=63i,client_ip="10.10.10.10",request="/user/login",http_version=1.1 1650309574001000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",request="/user/login",auth="-",http_version=1.1,resp_bytes=63i,client_ip="10.10.10.10" 1650309574002000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth="-",request="/user/login",client_ip="10.10.10.10",ident="-",http_version=1.1,resp_bytes=63i 1650309574003000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,resp_bytes=63i,client_ip="10.10.10.10",auth="-",request="/user/login",ident="-" 1650309514000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",auth="-",client_ip="10.10.10.10",request="/user/login",http_version=1.1,resp_bytes=63i 1650309514001000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,client_ip="10.10.10.10",request="/user/login",resp_bytes=63i,auth="-",ident="-" 1650309514002000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,ident="-",request="/user/login",http_version=1.1,client_ip="10.10.10.10",auth="-" 1650309514003000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip="10.10.10.10",ident="-",auth="-",request="/user/login",http_version=1.1 1650309574000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip="10.10.10.10",ident="-",auth="-",resp_bytes=63i,request="/user/login",http_version=1.1 1650309574001000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip="10.10.10.10",resp_bytes=63i,ident="-",auth="-",request="/user/login",http_version=1.1 1650309574002000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip="10.10.10.10",ident="-",auth="-",request="/user/login",http_version=1.1 1650309574003000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",resp_bytes=63i,client_ip="10.10.10.10",auth="-",http_version=1.1,ident="-" 1650309514000000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth="-",http_version=1.1,resp_bytes=63i,client_ip="10.10.10.10",request="/user/login",ident="-" 1650309514001000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip="10.10.10.10",auth="-",resp_bytes=63i,ident="-",request="/user/login",http_version=1.1 1650309514002000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth="-",resp_bytes=63i,request="/user/login",http_version=1.1,client_ip="10.10.10.10",ident="-" 1650309514003000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",resp_bytes=63i,client_ip="10.10.10.10",ident="-",http_version=1.1,auth="-" 1650309514004000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",client_ip="10.10.10.10",http_version=1.1,resp_bytes=63i,ident="-",auth="-" 1650309514005000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request="/user/login",http_version=1.1,resp_bytes=63i,client_ip="10.10.10.10",ident="-",auth="-" 1650309514006000000
nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident="-",request="/user/login",resp_bytes=63i,client_ip="10.10.10.10",auth="-",http_version=1.1 1650309514007000000

到了这里，关于【记录】Telegraf-解析nginx日志的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处： 如若内容造成侵权/违法违规/事实不符，请点击违法举报进行投诉反馈，一经查实，立即删除！

分享到：
领支付宝红包 赞助服务器费用

相关文章

  • Fabric V2.5 test-network启动日志分析

    背景 分析Fabric网络的启动日志可以帮我们理解搭建区块链网络的过程。本文对输出日志进行详细的分析。 启动网络与创建通道 安装链码 链码的生命周期：https://hyperledger-fabric.readthedocs.io/en/release-2.5/chaincode_lifecycle.html

    2024年02月21日
    浏览(38)
  • F12 浏览器调试模式页面刷新 network 日志刷新消失的解决办法

    F12 浏览器调试模式页面刷新 network 日志刷新消失的解决办法

    每次请求刷新后都把之前的请求记录刷新掉了，把preserve log勾选上后，所有的请求都会保留，再也不怕抓不到记录了。

    2024年02月15日
    浏览(61)
  • 记录swagger提示Network Error，postman访问正常

    记录swagger提示Network Error，postman访问正常

    Access to XMLHttpRequest at’http://127.0.0.1:9999/freightlogistics/carMain/list’ from origin ‘http://localhost:9999’ has been blocked by CORS policy: The’Access-Control-Allow-Origin’ header contains multiple values’http://localhost:9999, http://localhost:9999’, but only one is allowed.   在Swagger配置中，如果设置了网关的地址为

    2024年04月28日
    浏览(38)
  • Lighting Network（闪电网络）大白话解析

    Lighting Network（闪电网络）大白话解析

    通道(Channel)，通过在主网宣布通道建立，而后交易双方转至链下交易，把多次交易在链下完成，不占用主网资源，交易完成后在主网广播最终交易结果，无需更改主网机制即可实现吞吐量的提高。 “通道”是一个逻辑上的概念，实际使用过程中并没有“通道”，即使在数据传

    2024年02月04日
    浏览(44)
  • LLaMA Pro: Progressive LLaMA with Block Expansion

    LLaMA Pro: Progressive LLaMA with Block Expansion

    Q : 这篇论文试图解决什么问题？ A : 这篇论文试图解决大型语言模型（LLMs）在特定领域（如编程、数学、生物医学或金融）能力不足的问题。尽管LLMs在多种现实世界任务中表现出色，但在某些特定领域仍然存在局限性，这阻碍了开发通用语言代理以应用于更广泛场景的进展

    2024年01月16日
    浏览(31)
  • 金字塔场景解析网络（Pyramid Scene Parsing Network ）

    金字塔场景解析网络（Pyramid Scene Parsing Network ）

    基于语义分割的 场景解析 是计算机视觉中的基本主题。 目标是为图像中的每个像素分配一个类别标签 。场景解析提供了对场景的完整理解。它预测每个元素的标签、位置和形状。 场景解析的难度与场景和标签的多样性密切相关 。 最先进的场景解析框架主要基于全卷积网络

    2024年02月08日
    浏览(43)
  • nginx（二）---Nginx 日志Log

    nginx（二）---Nginx 日志Log

    日志配置 日志轮转/切割 日志分析         1.1日志模块 官方文档    http://nginx.org/en/docs/http/ngx_http_log_module.html 日志模块的名称            ngx_http_log_module         1.2相关指令 log_format         日志格式 access_log         访问日志 error_log          错误日志 open_log_file_

    2024年02月15日
    浏览(36)
  • GCNet: Global Context Network（ICCV 2019）原理与代码解析

    GCNet: Global Context Network（ICCV 2019）原理与代码解析

    paper： GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond official implementaion： https://github.com/xvjiarui/GCNet Third party implementation： https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/bricks/context_block.py 通过捕获long-range dependency提取全局信息，对各种视觉任务都是很有帮助的。Non-local Ne

    2024年02月12日
    浏览(38)
  • Python 日志记录：6大日志记录库的比较

    Python 日志记录：6大日志记录库的比较

    日志记录框架是一种工具，可帮助您标准化应用程序中的日志记录过程。虽然某些编程语言提供内置日志记录模块作为其标准库的一部分，但大多数日志记录框架都是第三方库，例如logging (Python)、Log4j (Java)、 Zerolog (Go) 或 Winston (Node.js)。有时，组织会选择开发自定义日志记录

    2024年02月14日
    浏览(42)
  • 使用Microsoft Network Monitor分析Wireshark无法解析的SSL流量包

    使用Microsoft Network Monitor分析Wireshark无法解析的SSL流量包

    在某次实验中发现wireshark抓到的ssl流量包无法政策解析Server Hello的数据。 在朋友的建议下使用Microsoft Network Monitor工具打开流量包，但Wireshark所抓的流量包格式为pcapng，Microsoft Network Monitor无法正常解析。所以需要使用Wireshark将其另存为Microsoft Network Monitor能解析的cap格式流量包

    2024年02月04日
    浏览(39)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包，优惠每天领

二维码1

领取红包

二维码2

领红包