keywhizfs

package module
v0.0.0-...-0f463f4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 8, 2015 License: Apache-2.0 Imports: 19 Imported by: 0

README

Overview

license build

KeywhizFs is a client for Keywhiz which represents accessible secrets as a userland filesystem (using FUSE). This client will mount a directory which contains files for each secret that is accessible.

Exposing secrets as a filesystem has many benefits.

  1. Consumers of secrets require no special libraries or code.
  2. Unix user and group permissions restrict which processes can read a secret.

Transparently, authentication is performed with a Keywhiz server using mutually-authenticated TLS. A client certificate, trusted by Keywhiz, is required and used to authenticate KeywhizFs. Refer to the Keywhiz documentation for generating and managing client access.

Directory structure

KeywhizFs will display all secrets under the top level directory of the mountpoint. Secrets may not begin with the '.' character, which is reserved for special control "files".

Control files

  • .running
  • This "file" contains the PID of the owner process.
  • .clear_cache
  • Deleting this empty "file" will cause the internal cache of KeywhizFs to be cleared. This should seldom be necessary in practice but has been useful at times.
  • .json/
  • This sub-directory mimics the REST API of Keywhiz. Reading files will directly communicate with the backend server and display the unparsed JSON response.

Filesystem permissions

Building

Run go build keywhizfs/main.go.

Testing

Simply run go test ./....

Running

/etc/fuse.conf

In order to allow keywhiz-fs to expose its filesystems to other users besides the owner of the process, fuse must be configured with the 'user_allow_other' option. Put the following snippet in /etc/fuse.conf.

# The following line was added by keywhiz-fs
user_allow_other

fusermount setuid permissions

The fusermount progam is used within the go-fuse library. Generally, it is installed setuid root, with group read/execute permissions for group 'fuse'. For KeywhizFs to work, the running user must be a member of the 'fuse' group.

CAP_IPC_LOCK capability

To prevent secrets from ending up in swap, KeywhizFs will attempt to mlockall memory. This is not required, but is beneficial. On Linux, set the proper capability on the KeywhizFs binary so memory can be locked without running as root. Example assumes your binary is at /sbin/keywhiz-fs.

setcap 'cap_ipc_lock=+ep' /sbin/keywhiz-fs

Usage

Usage: ./keywhiz-fs [options] url mountpoint
Options:
  -asuser="keywhiz": Default user to own files
  -ca="cacert.crt": PEM-encoded CA certificates file
  -cert="": PEM-encoded certificate file
  -debug=false: Enable debugging output
  -group="keywhiz": Default group to own files
  -key="client.key": PEM-encoded private key file
  -ping=false: Enable startup ping to server
  -timeout=20: Timeout for communication with server in seconds

The -cert option may be omitted if the -key option contains both a PEM-encoded certificate and key.

Running in Docker

We have included a Dockerfile so you can easily build and run keywhiz-fs with all of its dependencies. To build a kewhizfs Docker image run the following command:

docker build --rm -t keywhizfs .

After building, you can run the newly built image by running:

docker run --device /dev/fuse:/dev/fuse --cap-add=IPC_LOCK --cap-add=SYS_ADMIN keywhizfs -debug=true -ca=/go/src/github.com/square/keywhiz-fs/fixtures/cacert.crt -key=/go/src/github.com/square/keywhiz-fs/fixtures/client.pem https://localhost:443 /secrets/kwfs

Note that we have to pass --device /dev/fuse:/dev/fuse to mount the fuse device into the container, and give IPC_LOCK and SYS_ADMIN capabilities to the container, so it can set cap_ipc_lock on the keywhiz-fs binary, and so it can mount fuse-fs filesystems, respectively.

This build mounts the fuseFS at /secrets/kwfs/.

Caveats

Currently keywhiz-fs is not a 12 factor application, and it does not send unbuffered logs to stdout. It currently expects there to be a syslog server being ran locally.

We can follow this tutorial and run syslogd in a separate container, allowing us to use an external container as our syslogd. After that, we can use the external container as our keywhiz-fs syslog server:

docker run -v /tmp/syslogdev/log:/dev/log --device /dev/fuse:/dev/fuse --cap-add=IPC_LOCK --cap-add=SYS_ADMIN keywhizfs -debug=true -ca=/go/src/github.com/square/keywhiz-fs/fixtures/cacert.crt -key=/go/src/github.com/square/keywhiz-fs/fixtures/client.pem https://localhost:443 /secrets/kwfs

Additionally, if you see the following error: mlockall() failed with ENOMEM, you are probably running your docker deamon with aufs, which does not support capability extensions, making setcap 'cap_ipc_lock=+ep' /go/bin/keywhizfs fail. You should use overlayFS instead.

Contributing

Please contribute! And, please see CONTRIBUTING.md.

Documentation

Index

Constants

View Source 
const (
	VERSION = "2.0"
	EISDIR  = fuse.Status(unix.EISDIR)
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Cache 

type Cache struct {
	*log.Logger
	// contains filtered or unexported fields
}

Cache contains necessary state to return secrets, using previously cached content or retrieving from a server if necessary.

func NewCache 

func NewCache(backend SecretBackend, timeouts Timeouts, logConfig log.Config) *Cache

NewCache initializes a Cache.

func (*Cache) Add 

func (c *Cache) Add(s Secret)

Add inserts a secret into the cache. If a secret is already in the cache with a matching identifier, it will be overridden This method is most useful for testing since lookups may add data to the cache.

func (*Cache) Clear 

func (c *Cache) Clear()

Clear empties the internal cache.

func (*Cache) Len 

func (c *Cache) Len() int

Len returns the number of values stored in the cache. This method is most useful for testing.

func (*Cache) Secret 

func (c *Cache) Secret(name string) (*Secret, bool)

Secret retrieves a Secret by name from cache or a server.

Cache logic:

  • If cache hit and very recent: return cache entry
  • Ask backend w/ timeout
  • If backend returns fast: update cache, return
  • If timeout_backend_deadline AND cache hit: return cache entry, background update cache when backend returns
  • If timeout_max_wait: log error and pretend file doesn't exist

func (*Cache) SecretList 

func (c *Cache) SecretList() []Secret

SecretList returns a listing of Secrets from cache or a server.

Cache logic:

  • Ask backend w/ timeout
  • If backend returns fast: update cache, return
  • If timeout_backend_deadline: return cache entries, background update cache when backend returns
  • If timeout_max_wait: log error and pretend no files

type Client 

type Client struct {
	*klog.Logger
	// contains filtered or unexported fields
}

Client basic struct.

func NewClient 

func NewClient(certFile, keyFile, caFile, serverURL string, timeout time.Duration, logConfig klog.Config, ping bool) (client Client)

NewClient produces a read-to-use client struct given PEM-encoded certificate file, key file, and ca file with the list of trusted certificate authorities.

func (Client) RawSecret 

func (c Client) RawSecret(name string) (data []byte, ok bool)

RawSecret returns raw JSON from requesting a secret.

func (Client) RawSecretList 

func (c Client) RawSecretList() (data []byte, ok bool)

RawSecretList returns raw JSON from requesting a listing of secrets.

func (Client) Secret 

func (c Client) Secret(name string) (secret *Secret, ok bool)

Secret returns an unmarshalled Secret struct after requesting a secret.

func (Client) SecretList 

func (c Client) SecretList() (secrets []Secret, ok bool)

SecretList returns a slice of unmarshalled Secret structs after requesting a listing of secrets.

type KeywhizFs 

type KeywhizFs struct {
	pathfs.FileSystem
	*log.Logger
	Client    *Client
	Cache     *Cache
	StartTime time.Time
	Ownership Ownership
}

KeywhizFs is the central struct for dispatching filesystem operations.

func NewKeywhizFs 

func NewKeywhizFs(client *Client, ownership Ownership, timeouts Timeouts, logConfig log.Config) (kwfs *KeywhizFs, root nodefs.Node, err error)

NewKeywhizFs readies a KeywhizFs struct and its parent filesystem objects.

func (KeywhizFs) GetAttr 

func (kwfs KeywhizFs) GetAttr(name string, context *fuse.Context) (*fuse.Attr, fuse.Status)

GetAttr is a FUSE function which tells FUSE which files and directories exist.

name is empty when getting information on the base directory

func (KeywhizFs) Open 

func (kwfs KeywhizFs) Open(name string, flags uint32, context *fuse.Context) (nodefs.File, fuse.Status)

Open is a FUSE function where an in-memory open file struct is constructed.

func (KeywhizFs) OpenDir 

func (kwfs KeywhizFs) OpenDir(name string, context *fuse.Context) (stream []fuse.DirEntry, code fuse.Status)

OpenDir is a FUSE function called when performing a directory listing.

func (KeywhizFs) String 

func (kwfs KeywhizFs) String() string
func (kwfs KeywhizFs) Unlink(name string, context *fuse.Context) fuse.Status

Unlink is a FUSE function called when an object is deleted.

type Ownership 

type Ownership struct {
	Uid uint32
	Gid uint32
}

Ownership indicates the default ownership of filesystem entries.

func NewOwnership 

func NewOwnership(username, groupname string) Ownership

NewOwnership initializes default file ownership struct.

type Secret 

type Secret struct {
	Name        string
	Content     content   `json:"secret"`
	Length      uint64    `json:"secretLength"`
	CreatedAt   time.Time `json:"creationDate"`
	IsVersioned bool
	Mode        string
	Owner       string
	Group       string
}

Secret represents data returned after processing a server request.

json tags after fields indicate to json decoder the key name in JSON

func ParseSecret 

func ParseSecret(data []byte) (s *Secret, err error)

ParseSecret deserializes raw JSON into a Secret struct.

func ParseSecretList 

func ParseSecretList(data []byte) (secrets []Secret, err error)

ParseSecretList deserializes raw JSON into a list of Secret structs.

func (Secret) ModeValue 

func (s Secret) ModeValue() uint32

ModeValue function helps by converting a textual mode to the expected value for fuse.

type SecretBackend 

type SecretBackend interface {
	Secret(string) (secret *Secret, ok bool)
	SecretList() (secretList []Secret, ok bool)
}

SecretBackend represents an interface for storing secrets.

type SecretMap 

type SecretMap struct {
	// contains filtered or unexported fields
}

SecretMap is a thread-safe map for storing key -> secret mapping.

func NewSecretMap 

func NewSecretMap() *SecretMap

NewSecretMap initializes a new SecretMap.

func (*SecretMap) Get 

func (m *SecretMap) Get(key string) (s SecretTime, ok bool)

Get retrieves a values from the map and indicates if the lookup was ok.

func (*SecretMap) Len 

func (m *SecretMap) Len() int

Len returns the count of values stored.

func (*SecretMap) Overwrite 

func (m *SecretMap) Overwrite(m2 *SecretMap)

Overwrite will copy and overwrite data from another SecretMap.

func (*SecretMap) Put 

func (m *SecretMap) Put(key string, value Secret)

Put places a value in the map with a key, possibly overwriting an existing entry.

func (*SecretMap) PutIfAbsent 

func (m *SecretMap) PutIfAbsent(key string, value Secret) (put bool)

PutIfAbsent places a value in the map with a key, if that key did not exist. Returns whether the value was placed.

func (*SecretMap) Values 

func (m *SecretMap) Values() []SecretTime

Values returns a slice of stored secrets in no particular order.

type SecretTime 

type SecretTime struct {
	Secret Secret
	Time   time.Time
}

SecretTime contains a Secret record along with a timestamp when it was inserted.

type Timeouts 

type Timeouts struct {
	// FUSE may make many lookups in quick succession. If cached data is recent within the threshold,
	// a backend request is not attempted.
	Fresh time.Duration
	// BackendDeadline is distinct from the backend timeout. It is an optimistic timeout to wait
	// until resorting to cached data.
	BackendDeadline time.Duration
	MaxWait         time.Duration
}

Timeouts contains configuration for timeouts: timeout_backend_deadline: optimistic timeout to wait for cache timeout_max_wait: timeout for client to get data from server

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.