logpipe

Documentation

Overview

Package logpipe implements reading csv logs from PostgreSQL logging_collector (https://www.postgresql.org/docs/current/runtime-config-logging.html) and convert them to JSON.

Package logpipe implements reading csv logs from PostgreSQL logging_collector (https://www.postgresql.org/docs/current/runtime-config-logging.html) and convert them to JSON.

Index

• Constants
• type CSVReadWriter
• type CSVRecordParser
• type CSVRecordReadWriter
  • func NewCSVRecordReadWriter(fieldsPerRecord int) *CSVRecordReadWriter
• type ErrFieldCountExtended
  • func LogFieldValidator(fields int) *ErrFieldCountExtended
  • func (err *ErrFieldCountExtended) Cause() error
  • func (err *ErrFieldCountExtended) Error() string
• type FieldsValidator
• type LineLogPipe
  • func NewJSONLineLogPipe(fileName string) *LineLogPipe
  • func NewRawLineLogPipe(fileName, name string) *LineLogPipe
  • func (p *LineLogPipe) GetExecutedCondition() *concurrency.Executed
  • func (p *LineLogPipe) GetExitedCondition() *concurrency.Executed
  • func (p *LineLogPipe) Start(ctx context.Context) error
• type LogPipe
  • func NewLogPipe() *LogPipe
  • func (p *LogPipe) GetExitedCondition() *concurrency.Executed
  • func (p *LogPipe) GetInitializedCondition() *concurrency.Executed
  • func (p *LogPipe) Start(ctx context.Context) error
• type LogRecordWriter
  • func (writer *LogRecordWriter) Write(record NamedRecord)
• type LoggingRecord
  • func (r *LoggingRecord) FromCSV(content []string) NamedRecord
  • func (r *LoggingRecord) GetName() string
• type NamedRecord
• type PgAuditLoggingDecorator
  • func NewPgAuditLoggingDecorator() *PgAuditLoggingDecorator
  • func (r *PgAuditLoggingDecorator) FromCSV(content []string) NamedRecord
  • func (r *PgAuditLoggingDecorator) GetName() string
• type PgAuditRecord
• type RecordWriter

Constants

const FieldsPerRecord12 int = 23

FieldsPerRecord12 is the number of fields in a CSV log line in PostgreSQL 12 or below

const FieldsPerRecord13 int = 24

FieldsPerRecord13 is the number of fields in a CSV log line since PostgreSQL 13

const FieldsPerRecord14 int = 26

FieldsPerRecord14 is the number of fields in a CSV log line since PostgreSQL 14

const LoggingCollectorRecordName = "postgres"

LoggingCollectorRecordName is the value of the logger field for logging_collector

const PgAuditFieldsPerRecord int = 9

PgAuditFieldsPerRecord is the number of fields in a pgaudit log line

const PgAuditRecordName = "pgaudit"

PgAuditRecordName is the value of the logger field for pgaudit

Types

type CSVReadWriter

type CSVReadWriter interface {
	io.Writer
	Read() (record []string, err error)
}

CSVReadWriter is the interface for structs that are able to receive writes and parse CSV lines

type CSVRecordParser

type CSVRecordParser interface {
	FromCSV(content []string) NamedRecord
	NamedRecord
}

CSVRecordParser is implemented by structs that can be filled when parsing a CSV line. The FromCSV method just stores the CSV record fields inside the struct fields. A validation check of the CSV fields should be performed by the caller. Also handling recover from panic should be provided by the caller, in order to take care of runtime error, e.g. index out of range because of CSV malformation

type CSVRecordReadWriter

type CSVRecordReadWriter struct {
	io.Writer
	*csv.Reader
}

CSVRecordReadWriter wraps a csv.Reader and implements io.Writer. It parses CSV lines, that are then read through the csv.Reader.

func NewCSVRecordReadWriter

func NewCSVRecordReadWriter(fieldsPerRecord int) *CSVRecordReadWriter

NewCSVRecordReadWriter returns a new CSVRecordReadWriter which parses CSV lines with an expected number of fields. It uses a single record for memory efficiency.

type ErrFieldCountExtended

type ErrFieldCountExtended struct {
	Fields   []string
	Err      error
	Expected int
}

ErrFieldCountExtended is returned when the CSV line has an invalid number of fields

func LogFieldValidator

func LogFieldValidator(fields int) *ErrFieldCountExtended

LogFieldValidator checks if the provided number of fields is valid or not for logging_collector logs

func (*ErrFieldCountExtended) Cause

func (err *ErrFieldCountExtended) Cause() error

Cause returns the parent error

func (*ErrFieldCountExtended) Error

func (err *ErrFieldCountExtended) Error() string

Error returns a description of the invalid record

type FieldsValidator

type FieldsValidator func(int) *ErrFieldCountExtended

FieldsValidator is a function validating the number of fields for a specific log line to be parsed

type LineLogPipe

type LineLogPipe struct {
	// contains filtered or unexported fields
}

LineLogPipe a pipe for a given format

func NewJSONLineLogPipe

func NewJSONLineLogPipe(fileName string) *LineLogPipe

NewJSONLineLogPipe returns a logPipe for json format

func NewRawLineLogPipe

func NewRawLineLogPipe(fileName, name string) *LineLogPipe

NewRawLineLogPipe returns a logPipe for raw output

func (*LineLogPipe) GetExecutedCondition

func (p *LineLogPipe) GetExecutedCondition() *concurrency.Executed

GetExecutedCondition returns the condition that can be checked in order to be sure initialization has been done

func (*LineLogPipe) GetExitedCondition

func (p *LineLogPipe) GetExitedCondition() *concurrency.Executed

GetExitedCondition returns the condition that can be checked in order to be sure initialization has been done

func (*LineLogPipe) Start

func (p *LineLogPipe) Start(ctx context.Context) error

Start a new goroutine running the logging collector core, reading from a process logging raw strings to a file and redirecting its content to stdout in JSON format. The goroutine is started just once for a given file. All successive calls, that are referencing the same filename, will just check its existence

type LogPipe

type LogPipe struct {
	// contains filtered or unexported fields
}

LogPipe creates a pipe for a given file

func NewLogPipe

func NewLogPipe() *LogPipe

NewLogPipe returns a new LogPipe

func (*LogPipe) GetExitedCondition

func (p *LogPipe) GetExitedCondition() *concurrency.Executed

GetExitedCondition returns the condition that can be checked in order to be sure the process has been executed

func (*LogPipe) GetInitializedCondition

func (p *LogPipe) GetInitializedCondition() *concurrency.Executed

GetInitializedCondition returns the condition that can be checked in order to be sure initialization has been done

func (*LogPipe) Start

func (p *LogPipe) Start(ctx context.Context) error

Start a new goroutine running the logging collector core, reading from a process logging in CSV to a file and redirecting its content to stdout in JSON format. The goroutine is started just once for a given file. All successive calls, that are referencing the same filename, will just check its existence

type LogRecordWriter

type LogRecordWriter struct{}

LogRecordWriter implements the `RecordWriter` interface writing to the instance manager logger

func (*LogRecordWriter) Write

func (writer *LogRecordWriter) Write(record NamedRecord)

Write writes the PostgreSQL log record to the instance manager logger

type LoggingRecord

type LoggingRecord struct {
	LogTime              string `json:"log_time,omitempty"`
	Username             string `json:"user_name,omitempty"`
	DatabaseName         string `json:"database_name,omitempty"`
	ProcessID            string `json:"process_id,omitempty"`
	ConnectionFrom       string `json:"connection_from,omitempty"`
	SessionID            string `json:"session_id,omitempty"`
	SessionLineNum       string `json:"session_line_num,omitempty"`
	CommandTag           string `json:"command_tag,omitempty"`
	SessionStartTime     string `json:"session_start_time,omitempty"`
	VirtualTransactionID string `json:"virtual_transaction_id,omitempty"`
	TransactionID        string `json:"transaction_id,omitempty"`
	ErrorSeverity        string `json:"error_severity,omitempty"`
	SQLStateCode         string `json:"sql_state_code,omitempty"`
	Message              string `json:"message,omitempty"`
	Detail               string `json:"detail,omitempty"`
	Hint                 string `json:"hint,omitempty"`
	InternalQuery        string `json:"internal_query,omitempty"`
	InternalQueryPos     string `json:"internal_query_pos,omitempty"`
	Context              string `json:"context,omitempty"`
	Query                string `json:"query,omitempty"`
	QueryPos             string `json:"query_pos,omitempty"`
	Location             string `json:"location,omitempty"`
	ApplicationName      string `json:"application_name,omitempty"`
	BackendType          string `json:"backend_type,omitempty"`
	LeaderPid            string `json:"leader_pid,omitempty"`
	QueryID              string `json:"query_id,omitempty"`
}

LoggingRecord is used to store all the fields of the logging_collector CSV format

func (*LoggingRecord) FromCSV

func (r *LoggingRecord) FromCSV(content []string) NamedRecord

FromCSV stores inside the record structure the relative fields of the CSV log record.

See https://www.postgresql.org/docs/current/runtime-config-logging.html section "19.8.4. Using CSV-Format Log Output".

func (*LoggingRecord) GetName

func (r *LoggingRecord) GetName() string

GetName implements the NamedRecord interface

type NamedRecord

type NamedRecord interface {
	GetName() string
}

NamedRecord is the interface for structs that have a name

type PgAuditLoggingDecorator

type PgAuditLoggingDecorator struct {
	*LoggingRecord
	Audit         *PgAuditRecord `json:"audit,omitempty"`
	CSVReadWriter `json:"-"`
}

PgAuditLoggingDecorator stores all the fields of pgaudit CSV format

func NewPgAuditLoggingDecorator

func NewPgAuditLoggingDecorator() *PgAuditLoggingDecorator

NewPgAuditLoggingDecorator builds PgAuditLoggingDecorator

func (*PgAuditLoggingDecorator) FromCSV

func (r *PgAuditLoggingDecorator) FromCSV(content []string) NamedRecord

FromCSV implements the CSVRecordParser interface, parsing a LoggingRecord and then

func (*PgAuditLoggingDecorator) GetName

func (r *PgAuditLoggingDecorator) GetName() string

GetName implements the NamedRecord interface

type PgAuditRecord

type PgAuditRecord struct {
	AuditType      string `json:"audit_type,omitempty"`
	StatementID    string `json:"statement_id,omitempty"`
	SubstatementID string `json:"substatement_id,omitempty"`
	Class          string `json:"class,omitempty"`
	Command        string `json:"command,omitempty"`
	ObjectType     string `json:"object_type,omitempty"`
	ObjectName     string `json:"object_name,omitempty"`
	Statement      string `json:"statement,omitempty"`
	Parameter      string `json:"parameter,omitempty"`
}

PgAuditRecord stores all the fields of a pgaudit log line

type RecordWriter

type RecordWriter interface {
	Write(r NamedRecord)
}

RecordWriter is the interface