--- title: "查询请求流量日志分析" date: 2021-12-27 lastmod: 2021-12-27 description: "极限网关可跟踪记录所有请求，分析发送给Elasticsearch的请求性能及业务运行情况。通过配置路由、日志流程、管道任务、索引集群等，实现请求日志的记录与分析，并导入仪表板查看效果。" tags: ["INFINI Gateway", "Console"] summary: "极限网关能够跟踪记录经过网关的所有请求，可用来分析发送给 Elasticsearch 的请求情况，用于分析请求性能和了解业务运行情况。 设置网关路由 # 如果需要开启极限网关的查询日志分析，需要在路由上面配置 tracing_flow 参数，设置一个流程来记录请求日志。 router: - name: default tracing_flow: request_logging default_flow: cache_first 上面的配置定义了一个名为 default 的路由，默认的请求流程为 cache_first，用于日志记录的流程为 request_logging。 定义日志流程 # 日志处理流程配置 request_logging 的定义如下： flow: - name: request_logging filter: - request_path_filter: must_not: # any match will be filtered prefix: - /favicon.ico - request_header_filter: exclude: - app: kibana # in order to filter kibana's access log, config `elasticsearch.customHeaders: { "app": "kibana" }` to your kibana's config `/config/kibana.yml` - logging: queue_name: request_logging 上面的流程里面使用了若干个过滤器：" --- 极限网关能够跟踪记录经过网关的所有请求，可用来分析发送给 Elasticsearch 的请求情况，用于分析请求性能和了解业务运行情况。 {{% load-img "/img/blog/request-logging/dashboard-1.jpg" "" %}} ## 设置网关路由 如果需要开启极限网关的查询日志分析，需要在路由上面配置 `tracing_flow` 参数，设置一个流程来记录请求日志。 ``` router: - name: default tracing_flow: request_logging default_flow: cache_first ``` 上面的配置定义了一个名为 `default` 的路由，默认的请求流程为 `cache_first`，用于日志记录的流程为 `request_logging`。 ## 定义日志流程 日志处理流程配置 `request_logging` 的定义如下： ``` flow: - name: request_logging filter: - request_path_filter: must_not: # any match will be filtered prefix: - /favicon.ico - request_header_filter: exclude: - app: kibana # in order to filter kibana's access log, config `elasticsearch.customHeaders: { "app": "kibana" }` to your kibana's config `/config/kibana.yml` - logging: queue_name: request_logging ``` 上面的流程里面使用了若干个过滤器： - request_path_filter 过滤了无用的 `/favicon.ico` 请求 - request_header_filter，过滤了来自 Kibana 的请求 - logging，将请求日志记录到本地磁盘队列 `request_logging`，供后续管道来消费并创建索引 ## 定义日志管道 极限网关使用管道任务来异步消费这些日志，并创建索引，具体的定义配置如下： ``` pipeline: - name: request_logging_index auto_start: true keep_running: true processor: - json_indexing: index_name: "gateway_requests" elasticsearch: "dev" input_queue: "request_logging" idle_timeout_in_seconds: 1 worker_size: 1 bulk_size_in_mb: 10 #in MB ``` 上面的配置里面，定义了一个名为 `request_logging_index` 的处理管道，设置了消费的磁盘队列名称 `request_logging` 和索引的目标集群 `dev` 和索引名 `gateway_requests`，使用了一个工作线程，批次提交大小为 10MB。 ## 定义索引集群 接下来配置索引集群，如下： ``` elasticsearch: - name: dev enabled: true endpoint: https://192.168.3.98:9200 # if your elasticsearch is using https, your gateway should be listen on as https as well basic_auth: #used to discovery full cluster nodes, or check elasticsearch's health and versions username: elastic password: pass discovery: # auto discovery elasticsearch cluster nodes enabled: true refresh: enabled: true ``` 上面的配置定义了一个名为 `dev` 的 Elasticsearch 集群，并且开启 Elastic 模块来处理集群的自动配置。 ## 配置索引模板 然后就可以配置 Elasticsearch 集群的索引模板了，在 `dev` 集群上执行下面的命令创建日志索引的模板。 {{% load-img "/img/blog/request-logging/create-template.jpg" "" %}} {{< expand "展开查看 Elasticsearch 的模板定义" "..." >}} ``` PUT _template/.infini-gateway-default { "order": 100000, "index_patterns": [ "gateway_requests" ], "settings": { "index": { "max_result_window": "10000000", "number_of_shards": "1" } }, "mappings": { "dynamic_templates": [ { "strings": { "mapping": { "ignore_above": 256, "type": "keyword" }, "match_mapping_type": "string" } } ], "properties": { "request": { "properties": { "body": { "type": "text" } } }, "response": { "properties": { "body": { "type": "text" } } } } }, "aliases": {} } ``` {{< /expand >}} ## 配置索引生命周期 {{< expand "展开查看索引生命周期的定义" "..." >}} ``` PUT _ilm/policy/30days-retention { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "30d", "max_size": "50gb" }, "set_priority": { "priority": 100 } } }, "warm": { "actions": { "forcemerge": { "max_num_segments": 1 }, "set_priority": { "priority": 50 } } }, "cold": { "min_age": "3d", "actions": { "allocate": { "number_of_replicas": 1, "include": {}, "exclude": {}, "require": { "box_type": "warm" } }, "set_priority": { "priority": 0 } } }, "delete": { "min_age": "30d", "actions": { "delete": { "delete_searchable_snapshot": true } } } } } } PUT _template/gateway_requests-rollover { "order" : 100000, "index_patterns" : [ "gateway_requests-*" ], "settings" : { "index" : { "format" : "7", "lifecycle" : { "name" : "30days-retention", "rollover_alias" : "gateway_requests" }, "codec" : "best_compression", "routing" : { "allocation" : { "require" : { "box_type" : "hot" }, "total_shards_per_node" : "1" } }, "number_of_shards" : "1" } }, "mappings" : { "dynamic_templates" : [ { "strings" : { "mapping" : { "ignore_above" : 256, "type" : "keyword" }, "match_mapping_type" : "string" } } ] }, "aliases" : { } } DELETE gateway_requests-00001 PUT gateway_requests-00001 { "settings": { "index.lifecycle.rollover_alias":"gateway_requests" , "refresh_interval": "5s" }, "aliases":{ "gateway_requests":{ "is_write_index":true } } } ``` {{< /expand >}} ## 导入仪表板 下载面向 Kibana 7.9 的最新的仪表板 [INFINI-Gateway-7.9.2-2021-01-15.ndjson.zip](https://github.com/medcl/infini-gateway/releases)，在 `dev` 集群的 Kibana 里面导入，如下： {{% load-img "/img/blog/request-logging/import-dashboard.jpg" "" %}} ## 启动网关 接下来，就可以启动网关，。 ``` ➜ ./bin/gateway ___ _ _____ __ __ __ _ / _ \ /_\ /__ \/__\/ / /\ \ \/_\ /\_/\ / /_\///_\\ / /\/_\ \ \/ \/ //_\\\_ _/ / /_\\/ _ \/ / //__ \ /\ / _ \/ \ \____/\_/ \_/\/ \__/ \/ \/\_/ \_/\_/ [GATEWAY] A light-weight, powerful and high-performance elasticsearch gateway. [GATEWAY] 1.0.0_SNAPSHOT, a17be4c, Wed Feb 3 00:12:02 2021 +0800, medcl, add extra retry for bulk_indexing [02-03 13:51:35] [INF] [instance.go:24] workspace: data/gateway/nodes/0 [02-03 13:51:35] [INF] [api.go:255] api server listen at: http://0.0.0.0:2900 [02-03 13:51:35] [INF] [runner.go:59] pipeline: request_logging_index started with 1 instances [02-03 13:51:35] [INF] [entry.go:267] entry [es_gateway] listen at: http://0.0.0.0:8000 [02-03 13:51:35] [INF] [app.go:297] gateway now started. ``` ## 修改应用配置 将之前指向 Elasticsearch 地址的应用（如 Beats、Logstash、Kibana 等）换成网关的地址。 假设网关 IP 是 `192.168.3.98`，则修改 Kibana 配置如下： ``` # The Kibana server's name. This is used for display purposes. #server.name: "your-hostname" # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["https://192.168.3.98:8000"] elasticsearch.customHeaders: { "app": "kibana" } # When this setting's value is true Kibana uses the hostname specified in the server.host # setting. When the value of this setting is false, Kibana uses the hostname of the host # that connects to this Kibana instance. #elasticsearch.preserveHost: true # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. #kibana.index: ".kibana" # The default application to load. #kibana.defaultAppId: "home" ``` 保存配置并重启 Kibana。 ## 查看效果 现在任何通过网关访问 Elasticsearch 的请求都能被监控到了。 {{% load-img "/img/blog/request-logging/dashboard-1.jpg" "" %}} {{% load-img "/img/blog/request-logging/dashboard-2.jpg" "" %}} {{% load-img "/img/blog/request-logging/dashboard-3.jpg" "" %}}