Log analysis plays a very important role in troubleshooting and performance analysis in web systems. The focus of the tool is not on the usual PV, UV, etc. display, but on the fine-grained (minimum minute level, ie, one-minute log abstraction and summary) exception location and performance analysis over a specified time period.
Environmental installation
Python 3.4+
Pymongo 3.4.0+
MongoDB server
Clear several terms first
Uri refers to the part of the request that does not contain parameters; request_uri refers to the original request, with or without parameters; args refers to the part of the parameter in the request. (Refer to the definition in nginx) uri_abs and args_abs are strings that are abstracted for uri and args (for classification), for example: "/sub/0/100414/4070?channel=ios&version=1.4.5" Processing is converted to uri_abs: "/sub/*/*/*", args_abs:"channel=*&version=*"
Characteristics
Provide a general portal for log analysis: Through this portal, you can view a summary analysis of all server generated logs at a site; you can also filter based on time period and server dimensions.
Supports analysis of requesturi, IP and responsecode, based on three major dimensions: number of requests, response size, and response time. Different sub-items have their own characteristics.
(Core idea) is analyzed by a class of uri or its corresponding args, that is, the requesturi is abstracted and divided into two parts: uriabs and args_abs.
The abstract collation thought mentioned in 3, the default abstract method can satisfy most of the requirements; in addition, it also provides the option of custom abstract rules, based on which you can flexibly specify whether any part of the request should be abstracted.
The requesturi analysis can visually show which types of requests are large, which types of requests take more time, and which types of requests account for traffic. In addition, the distribution of indicators of a certain type of request in different granularities (minute, tenmin, hour, day) over time can be displayed. Change; the distribution of different argsabs indicators can also be analyzed for a certain uriabs
IP analysis divides all requests into three sources (fromcdn/proxy, fromreverseproxy, fromclientdirectly), each of which displays the IP address of the top N of its traffic, and displays the distribution of the metrics of an IP access over time; The distribution of different uriabs metrics that can be generated for an IP analysis
A more accurate description of response time and response size is achieved by the quartile concept, since the reference value of the arithmetic mean is not significant for the response time in the log.
High performance: In line with the idea of ​​who generated the log, the log analysis script loganalyse is scheduled to run on the web server, so loganalyse's high efficiency and low resources are also the top priority. After testing, on the author's server (disk: 3 * 7200rpm group RAID5, Gigabit LAN), for different log files, the processing speed is between 20,000 lines / s ~ 30,000 lines / s
Implementation ideas
The analysis script ( log_analyse.py ) is deployed to each web server and is scheduled to run through crontab. Log_analyse.py uses python's re module to analyze and process the log through regular expressions, and obtains information such as uri, args, time current, status code, response size, response time, server name, etc., and performs preliminary processing and then stores it into MongoDB. View the script ( log_show.py) as an entry to analyze and view the logs of all web servers. As for real-time, it depends on the execution frequency of the log_analyse.py script on the web server.
Premise specification
The log files of each server are stored in a unified path.
The log format and log naming rules are consistent (the format specified in the code is xxx.access.log)
0 o'clock log cutting every day
The log format determines the regular expression in the code, which can be customized according to the regular definition in analyse_config.py. The predefined log format in the project corresponds to the following:
Log_format access '$remote_addr - [$time_local] "$request" '
'$status $body_bytes_sent $request_time "$http_referer" '
'"$http_user_agent" - $http_x_forwarded_for';
For other formats of nginx logs or Apache logs, follow the above guidelines and use the tool for analysis and processing.
Processing of exception logs
If you want to divide the paragraphs by spaces or double quotation marks, the main problem is that when faced with various kinds of irregular records (both reasons and many styles), various exceptions cannot be taken into account. So the reason for using the re module instead of the simple split() function in the project. Some "tolerable" exception records in the code are processed by some decision logic; for "unacceptable" exception records, an empty string is returned and the log is recorded in a file. In fact, the best way to do these non-standard requests is to use a special character as a separator when defining the log format in nginx, such as "|". This eliminates the need for a re module, and direct string segmentation can correctly get the segments (performance will be better).
Log_show.py instructions for use:
help information
[ljk@demo ~]$ log_show --help
Usage:
Log_show
Log_show
Log_show
Options:
-h --help Showthis screen.
-f --from
-t --to
-l --limit
-s --server
-g --group_by
Valid values:"minute","ten_min","hour","day".[default: hour]
Distribution Show distribution(about hits,bytes,time,etc) of:
All or specific 'request', the specific 'ip', the specific 'error_code'in every period.
Periodis specific by--group_by
Detail Show details of:
Detail 'args' analyse of the specific 'uri'(if it has args);
Detail 'uri' analyse of the specific 'ip'or'error_code'
Notice: it's best to put 'request_uri', 'uri' and 'ip' in quotation marks.
All examples can filter the start time and the specified server with the -f, -t, -s parameters.
Request subcommand
Analyze data that has been logged in at the specified site today
[ljk@demo ~]$ log_show api request -l 3
=====================
Total_hits:999205 invalid_hits:581
=====================
Hits percent time_distribution(s) bytes_distribution(B) uri_abs
430210 43.06%%25<0.01%50<0.03%75<0.06%100<2.82 %25<42%50<61%75<63%100<155 /api/record/getR
183367 18.35%%25<0.02%50<0.03%75<0.06%100<1.73 %25<34%50<196%75<221%100<344 /api/getR/com/*/*/*
102299 10.24%%25<0.02%50<0.02%75<0.05%100<1.77 %25<3263%50<3862%75<3982%100<4512 /view/*/*/*/*.js
=====================
Total_bytes: 1.91 GB
=====================
Bytes percent time_distribution(s) bytes_distribution(B) uri_abs
1.23 GB 64.61% %25<0.03 %50<0.04 %75<0.1 %100<1.96 %25<2549 %50<17296 %75<31054 %100<691666 /api/NewCom/list
319.05 MB 16.32% %25<0.02 %50<0.02 %75<0.05 %100<1.77 %25<3263 %50<3862 %75<3982 %100<4512 /view/*/*/*/*.js
167.12 MB 8.55% %25<0.15 %50<0.19 %75<0.55 %100<2.93 %25<2791 %50<3078 %75<3213 %100<11327 /api/getR/com/*/*
=====================
Total_time: 117048s
=====================
Cum. time percent time_distribution(s) bytes_distribution(B) uri_abs
38747 33.10%%25<0.01%50<0.03%75<0.06%100<2.82 %25<42%50<61%75<63%100<155 /api/record/getR
22092 18.87%%25<0.02%50<0.03%75<0.06%100<1.73 %25<34%50<196%75<221%100<344 /api/getR/com/*/*/*
17959 15.34%%25<0.15%50<0.19%75<0.55%100<2.93 %25<2791%50<3078%75<3213%100<11327 /api/getRInfo/com/*/*
The above example can observe the ranking of the three dimensions of hits/bytes/time within the specified time (default 0 to the current time) and the distribution of response time and response size. For example, if you see that a uriabs has fewer hits that do produce larger bytes or consume more time, is it worth paying attention to the uriabs?
Ip subcommand
Display analysis results based on ip address
[ljk@demo ~]$ log_show.py api ip -l 2
=====================
From_cdn/Proxy: hits hits(%) bytes bytes(%) time(%)
==================== 199870 99.94 570.51 MB 99.99 99.99
Last_cdn_ip
Xxx.57.xxx.189 1914 0.96 696.18 KB 0.12 0.68
Xxx.206.xxx.154 1741 0.87 1.56 MB 0.27 0.98
User_ip_via_cdn
Xxx.249.xxx.56 787 0.39 154.82 KB 0.03 0.23
Xxx.60.xxx.86 183 0.09 1.05 MB 0.18 0.13
=====================
From_reverse_proxy: hits hits(%) bytes bytes(%) time(%)
==================== 66 0.03 68.83 KB 0.01 0.01
User_ip_via_proxy
Xxx.188.xxx.21 2 0.00 1.53 KB 0.00 0.00
Xxx.5.xxx.4 2 0.00 324.00 B 0.00 0.00
=====================
From_client_directly: hits hits(%) bytes bytes(%) time(%)
==================== 64 0.03 8.32 KB 0.00 0.00
Remote_addr
192.168.1.202 29 0.01 58.00 B 0.00 0.00
192.168.1.200 29 0.01 58.00 B 0.00 0.00
The idea of ​​IP analysis is to classify requests into three categories by source: Fromcdn/Proxy, Fromreverseproxy, Fromclient_directly, and then sort the IP addresses by number of requests within each category.
Distribution subcommand
Aggregate statistics for "all requests" or "specified uri/request_uri" by "minute/min/time/day"
Aggregate statistics for "specified IP" by "minute/min/time/day"
Applicable scenario: Check the changes of various indicators in the aggregated granularity of request/IP over time. For example, if the number of requests (or bandwidth) of a request is increased for a certain uri, it can be observed by the distribution subcommand. Big, it’s still getting bigger and bigger.
# Example 1: Analyze the distribution of the specified request, specify group aggregation by minute, and display 5 lines by default.
[ljk@demo ~]$ python log_show.py api request distribution "/view/*/*.json"-g minute
=====================
Uri_abs:/view/*/*.json
Total_hits: 17130 Total_bytes: 23.92 MB
=====================
Minute hits hits(%) bytes bytes(%) time_distribution(s) bytes_distribution(B)
1803091654 1543 9.01% 2.15 MB 8.98% %25<0.03 %50<0.03 %75<0.05 %100<1.07 %25<1532 %50<1593 %75<1645 %100<1982
1803091655 1527 8.91% 2.13 MB 8.88% %25<0.03 %50<0.04 %75<0.05 %100<1.04 %25<1538 %50<1592 %75<1642 %100<2143
1803091656 1464 8.55% 2.05 MB 8.57% %25<0.03 %50<0.04 %75<0.05 %100<1.03 %25<1536 %50<1592 %75<1642 %100<1952
1803091657 1551 9.05% 2.15 MB 8.97% %25<0.02 %50<0.03 %75<0.04 %100<0.89 %25<1534 %50<1594 %75<1639 %100<1977
1803091658 1458 8.51% 2.06 MB 8.61% %25<0.02 %50<0.03 %75<0.04 %100<2.35 %25<1540 %50<1596 %75<1644 %100<2146
Through the above example, the distribution of "/view/*/*.json" over a specified time period can be displayed, including the total amount of hits/bytes/time and the ratio of the indicators in each granularity to the total amount; The command also shows the "trends" of the indicators over time.
Description: The minute field is the specified aggregate size, and 1803091654 means "18:54 on March 09, 18". The granularity of the aggregation (minute/tenmin/hour/day) distribution subcommand can be specified by the -g parameter. With specific uri/requesturi (showing the uri/request_uri to specify the distribution of granularity over time) or not with uri (showing all requests to specify the granularity over time)
# Example 2: Analyze the number of requests/bandwidths generated by a specified IP over time. The default aggregate granularity is hour.
[ljk@demo ~]$ python log_show.py api ip -t 180314 distribution "140.206.109.174"-l 0
=====================
IP: 140.206.109.174
Total_hits: 10999 Total_bytes: 4.83 MB
=====================
Hour hits hits(%) bytes bytes(%)
18031306 1273 11.57% 765.40 KB 15.47%
18031307 2133 19.39%1004.74 KB 20.31%
18031308 2211 20.10% 1.00 MB 20.74%
18031309 2334 21.22% 1.05 MB 21.72%
18031310 2421 22.01% 850.79 KB 17.20%
18031311 627 5.70% 226.30 KB 4.57%
Description: The hour field indicates the default aggregation granularity, and 18031306 means "06 o'clock on March 13, 2016" - l 0 means that the number of output lines is not limited (that is, all results are output)
Detail subcommand:
Perform a detailed analysis of a uri to see the distribution of its various parameters (args)
Perform a detailed analysis of an IP to see the distribution of requests generated between different uri_abs
Applicable scenarios: For example, if a certain type of uriabs is abnormal in some aspect (hits/bytes/time), the uriabs can be further analyzed by the detail subcommand to accurately locate which parameter (args_abs). ) caused an exception; or observed an IP access exception, you can go deeper into the IP is a general access, or only interested in some uri.
# Example 1:
[ljk@demo ~]$ python log_show.py api -f 180201 request detail "/recommend/update"-l 3
=====================
Uri_abs:/recommend/batchUpdate
Total_hits: 10069 Total_bytes: 7.62 MB
=====================
Hits hits(%) bytes bytes(%) time(%) time_distribution(s) bytes_distribution(B) args_abs
4568 45.37% 3.46 MB 45.44% 47.96%%25<0.04%50<0.06%75<0.07%100<0.47 %25<755%50<795%75<845%100<1484 uid=*&category_id=*&channel=* &version=*
4333 43.03% 3.25 MB 42.64% 42.30%%25<0.03%50<0.05%75<0.07%100<0.48 %25<752%50<791%75<840%100<1447 category_id=*&channel=*&uid=* &version=*
389 3.86%314.15 KB 4.03% 0.88%%25<0.02%50<0.03%75<0.04%100<0.06 %25<766%50<802%75<850%100<1203 category_id=*&channel=*&version=*
In the above example, you can observe the status of each parameter of the different parameters corresponding to the uri "/recommend/update". There is also a side-by-side discovery: the same combination of parameters in the development of writing parameters is not written in the same order, although it does not affect the function, but it will cause some troubles when accurately monitoring the application performance.
Description: The detail subcommand is followed by uri (without parameters, parameters will be ignored if parameters are included)
# Example 2: Observe how many requests are generated for an IP, and the hits/bytes/time indicator for each request.
[ljk@demo ~]$ python log_show.py m -t 180314 ip detail "1.2.3.4"
=====================
IP: 140.206.109.174
Total_hits: 10999 Total_bytes: 4.83 MB
=====================
Hits hits(%) bytes bytes(%) time(%) uri_abs
10536 95.79% 405.47 KB 8.19% 92.01%/introduction/watch
147 1.34% 1.90 MB 39.31% 1.93%/view/*/*.html
138 1.25% 407.42 KB 8.23% 2.41% /chapinfo/*/*.html
42 0.38% 644.88 KB 13.03% 1.38%/info/*.html
30 0.27% 229.98 KB 4.65% 1.14%/classify/*.json
Loganalyse.py deployment instructions: The script is designed to be placed in the web server's scheduled tasks, scheduled (for example, every 30 minutes or 10 minutes, custom), and analyzed by logshow.py when needed. .
*/15 * * * * export LANG=zh_CN.UTF-8;python3 /home/ljk/log_analyse.py &>/tmp/log_analyse.log
Note
Where uri_abs and args_abs are the result of abstracting uri and args (abstracting a specific request pattern, that is, looking at the request classification). The default rules are as follows: uri: split request_uri into several segments with "/" and "." If a segment is all composed of numbers, abstract it as a "*" args: replace all values ​​with "*"
There are some other interesting functions in common/common.py
Magnetic Buzzer Self-drive Type
The magnetic buzzers (Self-drive Type) offer optimal
sound and performance for all types of audible alert and identification. Our
magnetic Buzzer solutions are offered with various mounting options. We also
provide you with a washable version for your preferred soldering method. Our magnetic buzzers, also known
as indicators, are designed with an internal drive circuit for easy application
integration. During operation, current is driven through a voice coil to
produce a magnetic field. When a voltage is applied, the coil generates a
magnetic field and then allows the diaphragm to vibrate and produce sound. This
buzzer type has a low operating voltage ranging from 1.5 – 12V+. Our magnetic
buzzers are desirable for applications requiring a lower sound pressure level
(SPL) and frequency.
Passive Buzzer,Dc Magnetic Buzzer,Electro Magnetic Buzzer,Magnetic Buzzer Self Drive Type
Jiangsu Huawha Electronices Co.,Ltd , https://www.hnbuzzer.com