pii

package module
v0.0.0-...-1ecd88d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 17, 2018 License: AGPL-3.0 Imports: 11 Imported by: 1

README

pii

PII Searching

Description

pii is a small library and command line utility that aims to make detecting PII a little easier for security analysts by doing the following:

  • Attempts to standardize well-known PII regular expressions to avoid discrepencies
  • Allows the creation of combinatorial logic around multiple regular expressions for more granularity
  • Proccesses files in parallel, giving performance enhancements over Python / Ruby / etc.
  • Is written as a library allowing for easy integration into existing Go tools

The tool is very much evolving as people contribute. This is one of those tools that should get better over time as improvements are made and the patterns are tuned even further.

Installation

go get github.com/gen0cide/pii/cmd/pii

Usage

pii search foo.txt
Options
  • pii -j/--json will output the data in JSON format.
  • pii search --find-matches will attempt to extract the findings from the files (expirimental)
Contact

Twitter: @alexlevinson

Slack: LOTR Slack (gandalf, #pii)

Documentation

Index

Constants

View Source 
const Version = `0.0.1`

Version represents the version of the PII library

Variables

View Source 
var (
	// DefaultRuleSet provides a rule set of default PII rules
	DefaultRuleSet = RuleSet{
		"phone_number":   defaultPhoneRule,
		"ssn":            defaultSSNRule,
		"email_address":  defaultEmailRule,
		"ip_address":     defaultIPRule,
		"credit_card":    defaultCreditCardRule,
		"street_address": defaultAddressRule,
		"banking_info":   defaultBankInfoRule,
		"uuid":           defaultUUIDRule,
		"vin":            defaultVINRule,
	}
)

Functions

func ExportAddresses 

func ExportAddresses(s string) (matches []string)

ExportAddresses is not really working - disregard for the meantime.

func ExportBankInfos 

func ExportBankInfos(s string) (matches []string)

ExportBankInfos attempts to extract all bank info matches from a string

func ExportCreditCards 

func ExportCreditCards(s string) (matches []string)

ExportCreditCards attempts to extract all credit card numbers from a string

func ExportEmails 

func ExportEmails(s string) (matches []string)

ExportEmails attempts to extract all email address matches from a string

func ExportIPs 

func ExportIPs(s string) (matches []string)

ExportIPs attempts to extract all IP Address matches from a string 

func ExportLinks(s string) (matches []string)

ExportLinks attempts to grab all links and URLs from a string

func ExportPhones 

func ExportPhones(s string) (matches []string)

ExportPhones attempts to extract all phone matches from a string

func ExportSSNs 

func ExportSSNs(s string) (matches []string)

ExportSSNs attempts to extract all SSN and India PAN number matches from a string

func ExportUUIDs 

func ExportUUIDs(s string) (matches []string)

ExportUUIDs attempts to extract all UUID matches from a string

func ExportVINs 

func ExportVINs(s string) (matches []string)

ExportVINs attempts to extract all VIN matches from a string

Types

type Exporter 

type Exporter func(s string) []string

Exporter is an extraction helper function to pull features matched by a Matcher

type File 

type File struct {
	sync.RWMutex
	Metric
	Filename     string              `json:"filename,omitempty" csv:"filename"`
	Path         string              `json:"path,omitempty" csv:"path"`
	Hits         map[string]bool     `json:"hits,omitempty" csv:"hits"`
	Matches      map[string][]string `json:"matches,omitempty" csv:"matches"`
	Errored      bool                `json:"errored,omitempty" csv:"errored"`
	ErrorMessage string              `json:"error_message,omitempty" csv:"error_message"`
	RuleCache    []Rule              `json:"-"`
	Data         []byte              `json:"-"`
}

File defines a specific file that is to be tested

func NewFile 

func NewFile(f string) *File

NewFile returns a generic file object

func (*File) Detect 

func (f *File) Detect(w *waiter.Waiter)

Detect performs all cached detection rules on a given file

func (*File) DetectOnRule 

func (f *File) DetectOnRule(rule Rule, w *waiter.Waiter)

DetectOnRule performs a single detection on a file with a given rule

func (*File) Error 

func (f *File) Error() string

func (*File) Find 

func (f *File) Find(w *waiter.Waiter)

Find performs all cached match rules on a given file

func (*File) FindOnRule 

func (f *File) FindOnRule(rule Rule, w *waiter.Waiter)

FindOnRule performs a single export on a file with a given rule

func (*File) Load 

func (f *File) Load(w *waiter.Waiter, errChan chan error)

Load loads the file into memory and proceeds with the test

type Matcher 

type Matcher func(string) bool

Matcher is an evaluation type

func Address 

func Address() Matcher

Address returns a matcher for identifying street address, po boxes, and zip codes

func All 

func All(funcs ...Matcher) Matcher

All returns a meta-matcher that requires all supplied matchers evaluate to true

func And 

func And(a, b Matcher) Matcher

And creates a meta matcher for requiring a logical AND between supplied matchers

func Any 

func Any(funcs ...Matcher) Matcher

Any creates a meta matcher for any possible hits against a set of matchers

func AtLeastN 

func AtLeastN(n int, funcs ...Matcher) Matcher

AtLeastN creates an at least n rule against a set of matchers

func BankInfo 

func BankInfo() Matcher

BankInfo returns a matcher for identifying either IBANs or US Routing #s

func CreditCard 

func CreditCard() Matcher

CreditCard returns a matcher for identifying major credit card numbers

func Email 

func Email() Matcher

Email returns a matcher for identifying email addresses

func HaltLangDetect 

func HaltLangDetect() Matcher

HaltLangDetect is a special matcher for preventing language detection from running

func IP 

func IP() Matcher

IP returns a matcher for identifying IP addresses 

func Link() Matcher

Link returns a matcher for identifying URLs and links that are not emails

func Not 

func Not(f Matcher) Matcher

Not is the logical negation of a Matcher

func Or 

func Or(a, b Matcher) Matcher

Or creates a meta matcher for performing a logical OR on two matchers

func Phone 

func Phone() Matcher

Phone returns a matcher for identifying international phone numbers

func SSN 

func SSN() Matcher

SSN returns a matcher for identifying US social security numbers

func UUID 

func UUID() Matcher

UUID returns a matcher for identifying GUIDs, UUIDs, v3, v4, and v5

func VIN 

func VIN() Matcher

VIN generates a matcher for identifying vehicle identification numbers

type Metric 

type Metric struct {
	DetectionLatencies map[string]float64 `json:"detection_latencies,omitempty" csv:"detection_latencies"`
	MatchLatencies     map[string]float64 `json:"match_latencies,omitempty" csv:"match_latencies"`
}

Metric is used to record the efficencies of matches

type Rule 

type Rule struct {
	Name        string   `json:"name,omitempty" csv:"name"`
	Description string   `json:"description,omitempty" csv:"description"`
	Severity    int      `json:"severity,omitempty" csv:"severity"`
	Filter      Matcher  `json:"-"`
	Exporter    Exporter `json:"-"`
}

Rule defines a matching requirement

type RuleSet 

type RuleSet map[string]Rule

RuleSet creates a map of multiple rules

func (RuleSet) Hits 

func (r RuleSet) Hits(s string) []Rule

Hits enumerates all rules within a ruleset returning any matching rules

type TestSettings 

type TestSettings struct {
	DetectPII bool `json:"detect_pii,omitempty" csv:"detect_pii"`
	FindPII   bool `json:"find_pii,omitempty" csv:"find_pii"`
}

TestSettings are used to set the test harness settings

type Tester 

type Tester struct {
	TestSettings
	Metric
	Rules []Rule  `json:"rules,omitempty" csv:"rules"`
	Files []*File `json:"files,omitempty" csv:"files"`
}

Tester defines a test harness for assessment

func NewDefaultTester 

func NewDefaultTester() *Tester

NewDefaultTester creates a new default Test harness with all default rules included

func NewEmptyTester 

func NewEmptyTester() *Tester

NewEmptyTester returns an empty Test harness with no rules loaded

func NewSubsetRuleTester 

func NewSubsetRuleTester(rules ...string) (*Tester, error)

NewSubsetRuleTester returns a TestHarness with only a section of rules

func (*Tester) AddNewFile 

func (t *Tester) AddNewFile(f string) error

AddNewFile adds a new file to the test harness

func (*Tester) AddRecursiveDirectory 

func (t *Tester) AddRecursiveDirectory(d string) error

AddRecursiveDirectory adds all files recursively in a directory to the test harness

func (*Tester) CalcAverages 

func (t *Tester) CalcAverages()

CalcAverages is used to calculate the averages of all the file matching metrics

func (*Tester) PerformDetections 

func (t *Tester) PerformDetections(w *waiter.Waiter) []error

PerformDetections performs all detections against a given Test Harness

func (*Tester) PerformMatches 

func (t *Tester) PerformMatches(w *waiter.Waiter)

PerformMatches performs all matches against a given Test Harness

func (*Tester) SetDetection 

func (t *Tester) SetDetection(s bool)

SetDetection sets the tester's detect PII setting

func (*Tester) SetFinder 

func (t *Tester) SetFinder(s bool)

SetFinder sets the tester's find PII setting

func (*Tester) Setup 

func (t *Tester) Setup()

Setup prepares the files for matching

Source Files

View all Source files

Directories

Path Synopsis
cmd
fakepiigen
pii

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.