README
¶
|Documentation Status|
=======
seekret
=======
Go library and command line to seek for secrets on various sources.
************
Command Line
************
Description
===========
``seekret`` inspect different sources (files into a directory or git
repositories) to seek for secrets. It can be used to prevent that secrets are
published in exposed locations.
Installing seekret
==================
``seekret`` can be directly installed by using go get.
::
go get github.com/18F/seekret/cmd/seekret
The requirements for a success installation are:
* pkg-config
* golang >= 1.6
* libgit2 >= 2.23
Usage
=====
General Options
~~~~~~~~~~~~~~~
::
NAME:
seekret - seek for secrets on various sources.
USAGE:
seekret [global options] command [command options] [arguments...]
VERSION:
0.0.1
AUTHOR(S):
Albert Puigsech Galicia <albert@puigsech.com>
COMMANDS:
seek:
git seek for seecrets on a git repository.
dir seek for seecrets on a directory.
GLOBAL OPTIONS:
--exception FILE, -x FILE load exceptions from FILE.
--rules PATH PATH with rules. [$SEEKRET_RULES_PATH]
--format value, -f value specify the output format. (default: "human")
--known FILE, -k FILE load known secrets from FILE.
--workers value, -w value number of workers used for the inspection (default: 4)
--help, -h show help
--version, -v print the version
``-x, --exception``
``--rules``
``-f, --format``
``-k, --known``
``-w, --workers``
Options for Git
~~~~~~~~~~~~~~~
::
NAME:
seekret git - seek for seecrets on a git repository.
USAGE:
seekret git [command options] [arguments...]
CATEGORY:
seek
OPTIONS:
--count value, -c value (default: 0)
``-c, --count``
Options for Dir
~~~~~~~~~~~~~~~
::
NAME:
seekret dir - seek for seecrets on a directory.
USAGE:
seekret dir [command options] [arguments...]
CATEGORY:
seek
OPTIONS:
--recursive, -r
--hidden
``-r, --recursive``
``-h, --hidden``
Examples
========
Scan all files from all commits in a local repo::
seekret git /path/to/repo
Scan all files from all commits in a remote repo::
seekret git http://github.com/18F/seekret-exposed
Scan all files from the last commit in a local repo::
seekret git --count 1 /path/to/repo
Scan all files (including hidden) in a local folder::
seekret dir --recursive --hidden /path/to/dir
Hands-On
========
The repository seekret-secrets is prepare to test seekret, and can be used to
perform the following hands-on examples:
1. Inspect remote git repository::
seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ git https://github.com/18F/seekret-secrets.git
2. Inspect local got repository::
git clone https://github.com/18F/seekret-secrets.git /tmp/seekret-secrets
seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ git /tmp/seekret-secrets
3. Inspect only the last 2 commits::
seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ git -c 2 /tmp/seekret-secrets
4. Inspect with exceptions::
seekret --rules $GOPATH/src/github.com/18F/seekret/rules/ -x /tmp/seekret-secrets/.exception_1 git /tmp/seekret-secrets
*******
Library
*******
Importing seekret Library
=========================
::
import seekret "github.com/18F/seekret/lib"
Init Seekret context
====================
::
s := seekret.NewSeekret()
Loading Rules
=============
::
s.LoadRulesFromPath("/path/to/main/rues:/path/to/other/rules:/path/to/more/rules")
::
s.LoadRulesFromDir("/path/to/rules")
::
s.LoadRulesFromFile("/path/to/file.rule")
Loading Objects
===============
::
opts := map[string]interface{} {
"hidden": true,
"recursive": false,
}
s.LoadObjects("dir", "/path/to/inspect", opts)
::
opts := map[string]interface{} {
"count": 10,
}
s.LoadObjects("dir", "/repo/to/inspect", opts)
Loading Exceptions
==================
::
s.LoadExceptionsFromFile("/path/to/exceptions/file")
Inspect
=======
::
s.Inspect(5)
Get Inspect Results
===================
::
secretsList := s.ListSecrets()
*****
Rules
*****
Secret identification is performed by using a set of rules specified on the
rules files. Those files, with '.rule' extension are defined by using YAML
following this format:
::
rulename:
match: [regexp]
unmatch:
- [regexp]
- [regexp]
- ...
For the contents of a file is considered a secret, it must comply with the
'match' regexp and not comply ANY of the 'unmatch' reg rule and comply match
ANY of the unmatch.
**********
Exceptions
**********
Exceptions determine conditions under which content should not be considered
a secret. The exceptions are specified by using a YAML file that follows this
format:
::
...
-
rule: [rulename]
object: [regexp]
line: [linenumber]
content: [regexp]
-
...
The conditions are optional, so it is not necessary to specify them all, but
for a content deemed exception must meet all the specified conditions.
The meaning of the various conditions explained:
``rule``
Contains the name of the rule.
``object``
Contains a regexp that should match the object name (usually the filename).
``line``
Contains the line number into the object.
``content``
Contains a regexp that should match the content.
.. |Documentation Status| image:: https://godoc.org/github.com/18F/seekret?status.svg
:target: https://godoc.org/github.com/18F/seekret
:width: 88px
:height: 20px
Documentation
¶
Overview ¶
Package seekret provides a framework to create tools to inspect information looking for sensitive information like passwords, tokens, private keys, certificates, etc.
Basics ¶
The current trend of automation of all things and de DevOps culture are very beneficial for efficiency but also come with several problems, being one of them the secret provisioning. Bootstrapping secrets into systems and applications may be complicated and sometimes the straightforward way is to store them into a insecure storage, like github repository, embedded into an artifact or system image, etc. That means that an AWS secret_key end up into a Github repository.
Seekret is an extensible framework that gelps in creating tools for detecting secrets on different sources. The secrets to detect are defined by a set of rules that can help detect passwords, tokens, private keys, certificates, etc.
Tools Using Seekret ¶
Seekret is extensible and can cover various use cases. Below there are some tools that uses seekret:
git-seekret: https://github.com/18F/git-seekret Git module that uses local hooks to help develepers to prevent leaking sensitive information in a commit.
Using It ¶
Seekret API is very simple and easy to use. This section shows some snippets of code that shows the basic operations you can do with it.
The first thing to be done is to create a new Seekret context:
s := seekret.NewSeekret()
Then the rules must to be loaded. They can be loaded from a path definition, a directory or a single file:
s.LoadRulesFromPath("/path/to/main/rues:/path/to/other/rules:/path/to/more/rules")
s.LoadRulesFromDir("/path/to/rules")
s.LoadRulesFromFile("/path/to/file.rule")
Optionally, exceptions (or false positives) can also be loaded from a file:
s.LoadExceptionsFromFile("/path/to/exceptions/file")
After that, must be loaded the objects to be inspected searching for secrets.
opts := map[string]interface{} {
// Loading options.
}
s.LoadObjects(sourceType, source, opts)
sourceType is an interface that implements the interface shown below. We offer sourceType's for Directories and Git Repositories, but you are able to extend it by creating your own.
type SourceType interface {
LoadObjects(source string, opt LoadOptions) ([]models.Object, error)
}
Currently, there are the following different sources supported:
Directories (and files): https://github.com/18F/seekret/sources/dir Load all files contained in a directory (and its sub-directories). Git Repositories: https://github.com/18F/seekret/sources/git Load git objects from commits or staging area.
Having all the rules, exceptions and objects loaded into the contects, it's possible to start the inspection with the following code:
s.Inspect(Nworkers)
Nworkers is an integuer that specify the number of goroutines used on the inspection. The recommended value is runtime.NumCPU().
Finally, it is possible to obtain the list of secrets located and do something with them:
secretsList := s.ListSecrets()
for secret := range secretsList {
// Do something
}
Index ¶
- func DefaultRulesPath() string
- type LoadOptions
- type Seekret
- func (s *Seekret) AddException(exception models.Exception)
- func (s *Seekret) AddRule(rule models.Rule, enabled bool)
- func (s *Seekret) DisableRule(name string) error
- func (s *Seekret) DisableRuleByRegexp(name string) int
- func (s *Seekret) EnableRule(name string) error
- func (s *Seekret) EnableRuleByRegexp(name string) int
- func (s *Seekret) GroupObjectsByMetadata(k string) map[string][]models.Object
- func (s *Seekret) GroupObjectsByPrimaryKeyHash() map[string][]models.Object
- func (s *Seekret) Inspect(Nworkers int)
- func (s *Seekret) ListRules() []models.Rule
- func (s *Seekret) ListSecrets() []models.Secret
- func (s *Seekret) LoadExceptionsFromFile(file string) error
- func (s *Seekret) LoadObjects(st SourceType, source string, opt LoadOptions) error
- func (s *Seekret) LoadRulesFromDir(dir string, defaulEnabled bool) error
- func (s *Seekret) LoadRulesFromFile(file string, defaulEnabled bool) error
- func (s *Seekret) LoadRulesFromPath(path string, defaulEnabled bool) error
- type SourceType
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultRulesPath ¶
func DefaultRulesPath() string
DefaultRulesPath return the default PATH that contains rules.
Types ¶
type LoadOptions ¶
type LoadOptions map[string]interface{}
type Seekret ¶
type Seekret struct {
// contains filtered or unexported fields
}
Seekret contains a seekret context and exposes the API to manipulate it.
func (*Seekret) AddException ¶
AddException adds a new exception into the context.
func (*Seekret) DisableRule ¶
DisableRule disables specific rule.
func (*Seekret) DisableRuleByRegexp ¶
DisableRule disables rules that match with a regular expression.
func (*Seekret) EnableRule ¶
EnableRule enables specific rule.
func (*Seekret) EnableRuleByRegexp ¶
EnableRule enables rules that match with a regular expression.
func (*Seekret) GroupObjectsByMetadata ¶
GroupObjectsByMetadata returns a map with all objects grouped by specific metadata key.
func (*Seekret) GroupObjectsByPrimaryKeyHash ¶
GroupObjectsByPrimaryKeyHash returns a map with all objects grouped by the primary key hash, that is calculated from all metadata keys with the primary attribute. All returned objects could have the same content, even if are not the same.
func (*Seekret) Inspect ¶
Inspect executes the inspection into all loaded objects, by checking all rules and exceptions loaded.
func (*Seekret) ListSecrets ¶
ListSecrets return an array with all found secrets after the inspection.
func (*Seekret) LoadExceptionsFromFile ¶
LoadExceptionsFromFile loads exceptions from a YAML file.
func (*Seekret) LoadObjects ¶
func (s *Seekret) LoadObjects(st SourceType, source string, opt LoadOptions) error
LoadObjects loads objects form an specific source. It can load objects from different source types, that are implemented following the SourceType interface.
func (*Seekret) LoadRulesFromDir ¶
LoadRulesFromFile loads rules from all YAML files inside a directory.
func (*Seekret) LoadRulesFromFile ¶
LoadRulesFromFile loads rules from a YAML file.
type SourceType ¶
type SourceType interface {
LoadObjects(source string, opt LoadOptions) ([]models.Object, error)
}
Source Files
Directories