squidlog

README

Squid log monitoring with Netdata

Squid is a caching and forwarding HTTP web proxy.

This module parses Squid access logs.

Metrics

All metrics have "squidlog." prefix.

Metric	Scope	Dimensions	Units
requests	global	requests	requests/s
excluded_requests	global	unmatched	requests/s
type_requests	global	success, bad, redirect, error	requests/s
http_status_code_class_responses	global	1xx, 2xx, 3xx, 4xx, 5xx	responses/s
http_status_code_responses	global	a dimension per HTTP response code	responses/s
bandwidth	global	sent	kilobits/s
response_time	global	min, max, avg	milliseconds
uniq_clients	global	clients	clients
cache_result_code_requests	global	a dimension per cache result code	requests/s
cache_result_code_transport_tag_requests	global	a dimension per cache result delivery transport tag	requests/s
cache_result_code_handling_tag_requests	global	a dimension per cache result handling tag	requests/s
cache_code_object_tag_requests	global	a dimension per cache result produced object tag	requests/s
cache_code_load_source_tag_requests	global	a dimension per cache result load source tag	requests/s
cache_code_error_tag_requests	global	a dimension per cache result error tag	requests/s
http_method_requests	global	a dimension per HTTP method	requests/s
mime_type_requests	global	a dimension per MIME type	requests/s
hier_code_requests	global	a dimension per hierarchy code	requests/s
server_address_forwarded_requests	global	a dimension per server address	requests/s

Log Parsers

Squidlog supports 3 log parsers:

• CSV
• LTSV
• RegExp

RegExp is the slowest among them, but it is very likely you will need to use it if your log format is not default.

Known Fields

These are Squid log format codes.

Squidlog is aware how to parse and interpret following codes:

field	squid format code	description
resp_time	%tr	Response time (milliseconds).
client_address	%>a	Client source IP address.
client_address	%>A	Client FQDN.
cache_code	%Ss	Squid request status (TCP_MISS etc).
http_code	%>Hs	The HTTP response status code from Content Gateway to client.
resp_size	%<st	Total size of reply sent to client (after adaptation).
req_method	%rm	Request method (GET/POST etc).
hier_code	%Sh	Squid hierarchy status (DEFAULT_PARENT etc).
server_address	%<a	Server IP address of the last server or peer connection.
server_address	%<A	Server FQDN or peer name.
mime_type	%mt	MIME content type.

In addition, to make Squid native log format csv parsable, squidlog understands these groups of codes:

field	squid format code	description
result_code	%Ss/%>Hs	Cache code and http code.
hierarchy	%Sh/%<a	Hierarchy code and server address.

Custom Log Format

Custom log format is easy. Use known fields to construct your log format.

• If using CSV parser

Note: can be used only if all known squid format codes are separated by csv delimiter. For example, if you have %Ss:%Sh, csv parser cant extract %Ss and %Sh from it, and you need to use RegExp parser.

Copy your current log format. Replace all known squid format codes with corresponding known fields. Replaces others with "-".

jobs:
  - name: squid_log_custom_csv_exampla
    path: /var/log/squid/access.log
    log_type: csv
    csv_config:
      format: '- resp_time client_address result_code resp_size req_method - - hierarchy mime_type'

• If using LTSV parser

Provide fields mapping. You need to map your label names to known fields.

  - name: squid_log_custom_ltsv_exampla
    path: /var/log/squid/access.log
    log_type: ltsv
    ltsv_config:
      mapping:
        label1: resp_time
        label2: client_address
        ...

• If using RegExp parser

Use pattern with subexpressions names. These names should be known by squidlog. We recommend to use https://regex101.com/ to test your regular expression.

jobs:
  - name: squid_log_custom_regexp_exampla
    path: /var/log/squid/access.log
    log_type: regexp
    regexp_config:
      format: '^[0-9.]+\s+(?P<resp_time>[0-9]+) (?P<client_address>[\da-f.:]+) (?P<cache_code>[A-Z_]+)\/(?P<http_code>[0-9]+) (?P<resp_size>[0-9]+) (?P<req_method>[A-Z]+) [^ ]+ [^ ]+ (?P<hier_code>[A-Z_]+)\/[\da-z.:-]+ (?P<mime_type>[A-Za-z-]+)'

Configuration

Edit the go.d/squidlog.conf configuration file using edit-config from the Netdata config directory, which is typically at /etc/netdata.

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/squidlog.conf

This module needs only path to log file if you use native log format. If you use custom log format you need to set it manually.

jobs:
  - name: squid
    path: /var/log/squid/access.log

For all available options, please see the module configuration file.

Troubleshooting

To troubleshoot issues with the squid_log collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/
  

• Switch to the netdata user.

  sudo -u netdata -s
  

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m squid_log
  

Documentation

Index

• type Chart
• type Charts
• type Config
• type Dim
• type Dims
• type SquidLog
  • func New() *SquidLog
  • func (s *SquidLog) Charts() *module.Charts
  • func (s *SquidLog) Check() bool
  • func (s *SquidLog) Cleanup()
  • func (s *SquidLog) Collect() map[string]int64
  • func (s *SquidLog) Init() bool

Types

type Chart

type Chart = module.Chart

type Charts

type Charts = module.Charts

type Config

type Config struct {
	Parser      logs.ParserConfig `yaml:",inline"`
	Path        string            `yaml:"path"`
	ExcludePath string            `yaml:"exclude_path"`
}

type Dim

type Dim = module.Dim

type Dims

type Dims = module.Dims

type SquidLog

type SquidLog struct {
	module.Base
	Config `yaml:",inline"`
	// contains filtered or unexported fields
}

func New

func New() *SquidLog

func (*SquidLog) Charts

func (s *SquidLog) Charts() *module.Charts

func (*SquidLog) Check

func (s *SquidLog) Check() bool

func (*SquidLog) Cleanup

func (s *SquidLog) Cleanup()

func (*SquidLog) Collect

func (s *SquidLog) Collect() map[string]int64

func (*SquidLog) Init

func (s *SquidLog) Init() bool