專案

一般

配置概況

Wiki » Graylog »

利用Content Packs快速建立對Nginx服務日誌的資料分析

Post created on：2023-10-25
本文操作基於 Graylog V5.1.6

Nginx伺服器端的配置

於nginx.conf中定義日誌格式

log_format graylog_json escape=json '{ "nginx_timestamp": "$time_iso8601", '
        '"remote_addr": "$remote_addr", '
        '"connection": "$connection", '
        '"connection_requests": $connection_requests, '
        '"pipe": "$pipe", '
        '"body_bytes_sent": $body_bytes_sent, '
        '"request_length": $request_length, '
        '"request_time": $request_time, '
        '"response_status": $status, '
        '"request": "$request", '
        '"request_method": "$request_method", '
        '"host": "$host", '
        '"upstream_cache_status": "$upstream_cache_status", '
        '"upstream_addr": "$upstream_addr", '
        '"http_x_forwarded_for": "$http_x_forwarded_for", '
        '"http_referrer": "$http_referer", '
        '"http_user_agent": "$http_user_agent", '
        '"http_version": "$server_protocol", '
        '"remote_user": "$remote_user", '
        '"http_x_forwarded_proto": "$http_x_forwarded_proto", '
        '"upstream_response_time": "$upstream_response_time", '
        '"nginx_access": true }';

於要記錄的站點裡添加日誌的配置，其中graylog_json對應nginx.conf中定義的日誌格式名稱，graylog.example.com為graylog伺服器的位置，port號按照graylog input的配置決定

server {
    listen 443 ssl;
    server_name www.example.com
    access_log syslog:server=graylog.example.com:12401 graylog_json;
    ...
    ...
}

重新載入nginx

systemctl reload nginx

Graylog伺服器端的配置

前往 nginx-syslog 下載最新版本的內容包後解壓縮，登入Graylog WEBUI後，前往 System>Content Packs，並上傳「content_pack.json」

隨後可以在Content Packs清單中看到「nginx-syslog」，點擊其右方的「Install」按鈕

我安裝的是V4版本，內容包將會替Graylog安裝以下元件：

  • Input
  • Stream
  • Pipelines

前往 System>Inputs，可以看到內容包替我們安裝好了一個「nginx-syslog」的Global inputs，如果有需要調整預設值，可以在該Input的More actions中選擇「Edit input」進行參數的變更，這裡我變更的有Time Zone，Update後生效

確認nginx-syslog所創建的Stream有資料量在流動

效果

沒有使用nginx-syslog前，查閱主機上的nginx日誌時，閱讀困難，非常眼花撩亂

65.108.0.71 - - [24/Oct/2023:16:36:15 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,hours,issue:desc HTTP/1.1" 200 18692 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:18 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,hours,project:desc HTTP/1.1" 200 18700 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:21 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,hours:desc,activity:desc HTTP/1.1" 200 18730 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:25 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,hours:desc,issue:desc HTTP/1.1" 200 18718 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:28 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,hours:desc,spent_on HTTP/1.1" 200 18704 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:31 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,issue,project:desc HTTP/1.1" 200 18700 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:34 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,issue:desc,activity:desc HTTP/1.1" 200 18730 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:37 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,issue:desc,hours:desc HTTP/1.1" 200 18718 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:40 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,issue:desc,spent_on HTTP/1.1" 200 18704 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-" 
65.108.0.71 - - [24/Oct/2023:16:36:43 +0800] "GET /time_entries/report?set_filter=1&sort=issue.category:desc,project,issue:desc HTTP/1.1" 200 18700 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" "-" "-"

使用nginx-syslog後，前往 Search頁面，能看到nginx的日誌已經變成了json格式紀錄，非常容易閱讀，在篩選出JSON的key/value後，還會將原本的資料存成一份full message作為參考，更重要的是這種經過整理後的JSON日誌，由於有明確的屬性和數值，才能夠被使用於 Dashboard 上

透過畫面上方的Search Filters、以指定的時間日期加上簡單的日誌屬性指定（可包含AND、OR等判斷式），可以快速的搜索所需日誌

Reference

  1. graylog-content-pack-nginx-syslog
  2. NGINX Logs to Graylog - Quick and Dirty

回到頁首