Documentation ¶
Overview ¶
ConsulFS implements a FUSE filesystem that is backed by a Consul Key-Value store.
API Usage --------- ConsulFS is implemented using the "bazil.org/fuse" package as a file system service. Refer to the fuse package for complete documentation on how to create a new FUSE connection that services a mount point. To have that mount point serve ConsulFS files, create a new instance of `ConsulFs`, give it to an "bazil.org/fuse/fs.Server", and call its Serve() method. The source code for the mounter at "github.com/bwester/consulfs/bin/consulfs" gives a full example of how to perform the mounting.
The `ConsulFs` instance contains common configuration data and is referenced by all file and directory inodes. Notably, 'Uid' and 'Gid' set the uid and gid ownership of all files. The `Consul` option is used to perform all Consul HTTP RPCs. The `CancelConsulKv` struct is included in this package as a wrapper around the standard Consul APIs. It is vital for system stability that the file system not get into an uninteruptable sleep waiting for a remote RPC to complete, so CancelConsulKv will abandon requests when needed.
Index ¶
- Constants
- Variables
- type CancelConsulKv
- func (cckv *CancelConsulKv) CAS(ctx context.Context, p *consul.KVPair, q *consul.WriteOptions) (bool, *consul.WriteMeta, error)
- func (cckv *CancelConsulKv) Delete(ctx context.Context, key string, w *consul.WriteOptions) (*consul.WriteMeta, error)
- func (cckv *CancelConsulKv) Get(ctx context.Context, key string, q *consul.QueryOptions) (*consul.KVPair, *consul.QueryMeta, error)
- func (cckv *CancelConsulKv) Keys(ctx context.Context, prefix string, separator string, q *consul.QueryOptions) ([]string, *consul.QueryMeta, error)
- func (cckv *CancelConsulKv) Put(ctx context.Context, p *consul.KVPair, q *consul.WriteOptions) (*consul.WriteMeta, error)
- type ConsulCanceler
- type ConsulFs
- type Dir
- func (dir *Dir) Attr(ctx context.Context, attr *fuse.Attr) error
- func (dir *Dir) Create(ctx context.Context, req *fuse.CreateRequest, resp *fuse.CreateResponse) (fs.Node, fs.Handle, error)
- func (dir *Dir) Lookup(ctx context.Context, name string) (fs.Node, error)
- func (dir *Dir) Mkdir(ctx context.Context, req *fuse.MkdirRequest) (fs.Node, error)
- func (dir *Dir) NewDir(prefix string) *Dir
- func (dir *Dir) NewFile(key string) *File
- func (dir *Dir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error)
- func (dir *Dir) Refresh(ctx context.Context) error
- func (dir *Dir) Remove(ctx context.Context, req *fuse.RemoveRequest) error
- func (dir *Dir) RemoveDir(ctx context.Context, req *fuse.RemoveRequest) error
- func (dir *Dir) RemoveFile(ctx context.Context, req *fuse.RemoveRequest) error
- func (dir *Dir) Rename(ctx context.Context, req *fuse.RenameRequest, newDirNode fs.Node) error
- type File
- func (file *File) Attr(ctx context.Context, attr *fuse.Attr) error
- func (file *File) BufferRead() ([]byte, bool)
- func (file *File) BufferTruncate(size uint64) bool
- func (file *File) BufferWrite(req *fuse.WriteRequest, resp *fuse.WriteResponse) bool
- func (file *File) Fsync(ctx context.Context, req *fuse.FsyncRequest) error
- func (file *File) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fs.Handle, error)
- func (file *File) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error
- func (file *File) ReadAll_(ctx context.Context) ([]byte, error)
- func (file *File) Release(ctx context.Context, req *fuse.ReleaseRequest) error
- func (file *File) SetDeleted(ctx context.Context) error
- func (file *File) Setattr(ctx context.Context, req *fuse.SetattrRequest, resp *fuse.SetattrResponse) error
- func (file *File) Truncate(ctx context.Context, size uint64) error
- func (file *File) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) error
Constants ¶
const MaxWriteAttempts = 10
The number of time a write will be attempted before returning an error
Variables ¶
var ErrCanceled = errors.New("operation canceled")
ErrCanceled is returned whenever a Consul operation is canceled.
Functions ¶
This section is empty.
Types ¶
type CancelConsulKv ¶
type CancelConsulKv struct { // The Consul client to use for executing operations. Client *consul.Client // Logger gets all the logging messages Logger *logrus.Logger }
CancelConsulKv is the concrete implementation of ConsulCanceler. It takes a Consul `Client` object and performs all operations using that client. When an operation is "canceled", the method call will return immediately with an ErrCanceled error returned. The underlying HTTP call is not aborted.
func (*CancelConsulKv) CAS ¶
func (cckv *CancelConsulKv) CAS( ctx context.Context, p *consul.KVPair, q *consul.WriteOptions, ) (bool, *consul.WriteMeta, error)
CAS performs a compare-and-swap on a key
func (*CancelConsulKv) Delete ¶
func (cckv *CancelConsulKv) Delete( ctx context.Context, key string, w *consul.WriteOptions, ) (*consul.WriteMeta, error)
Delete removes a key and its data.
func (*CancelConsulKv) Get ¶
func (cckv *CancelConsulKv) Get( ctx context.Context, key string, q *consul.QueryOptions, ) (*consul.KVPair, *consul.QueryMeta, error)
Get returns the current value of a key.
type ConsulCanceler ¶
type ConsulCanceler interface { CAS( ctx context.Context, p *consul.KVPair, q *consul.WriteOptions, ) (bool, *consul.WriteMeta, error) Delete( ctx context.Context, key string, w *consul.WriteOptions, ) (*consul.WriteMeta, error) Get( ctx context.Context, key string, q *consul.QueryOptions, ) (*consul.KVPair, *consul.QueryMeta, error) Keys( ctx context.Context, prefix string, separator string, q *consul.QueryOptions, ) ([]string, *consul.QueryMeta, error) Put( ctx context.Context, p *consul.KVPair, q *consul.WriteOptions, ) (*consul.WriteMeta, error) }
ConsulCanceler defines an API for accessing a Consul Key-Value store. It's mostly a clone of `github.com/hashicorp/consul/api.KV`, but it adds the ability to stop waiting for an operation if that operation has expired.
type ConsulFs ¶
type ConsulFs struct { // Consul contains a referene to the ConsulCanceler that should be used for all operations. Consul ConsulCanceler // Uid contains the UID that will own all the files in the file system. Uid uint32 // Gid contains the GID that will own all the files in the file system. Gid uint32 // Perms sets the file permission flags for all files and directories. If zero, a // default of 0600 will be used. Perms os.FileMode // RootPath contains the path to the root of the filesystem in Consul. This // string will be prefixed to all paths requested from Consul. A path // separator will be added if needed. RootPath string // Messages will be sent to this logger Logger *logrus.Logger }
ConsulFs is the main file system object that represents a Consul Key-Value store.
type Dir ¶
type Dir struct { ConsulFs *ConsulFs Prefix string Level uint // contains filtered or unexported fields }
Dir represents a directory inode in VFS. Directories don't actually exist in Consul. TODO: discuss the strategy used to fake dirs.
func (*Dir) Attr ¶
Attr implements the Node interface. It is called when fetching the inode attribute for this directory (e.g., to service stat(2)).
func (*Dir) Create ¶
func (dir *Dir) Create( ctx context.Context, req *fuse.CreateRequest, resp *fuse.CreateResponse, ) (fs.Node, fs.Handle, error)
Create implements the NodeCreater interface. It is called to create and open a new file. The kernel will first try to Lookup the name, and this method will only be called if the name didn't exist.
func (*Dir) Lookup ¶
Lookup implements the NodeStringLookuper interface, to look up a directory entry by name. This is called to get the inode for the given name. The name doesn't have to have been returned by ReadDirAll() for a process to attempt to find it!
func (*Dir) Mkdir ¶
Mkdir implements the NodeMkdirer interface. It is called to make a new directory.
func (*Dir) ReadDirAll ¶
ReadDirAll returns the entire contents of the directory when the directory is being listed (e.g., with "ls").
func (*Dir) Remove ¶
Remove implements the NodeRemover interface. It is called to remove files or directory from a directory's contents.
func (*Dir) RemoveFile ¶
RemoveFile is called to unlink a file.
func (*Dir) Rename ¶
Rename implements the NodeRenamer interface. It's called to rename a file from one name to another, possibly in another directory. There is no plan to support renaming directories at this time. Consul doesn't have a rename operation, so the new name is written and the old one deleted as two separate actions. If the new name already exists as a file, it is replaced atomically.
type File ¶
type File struct { ConsulFs *ConsulFs Key string // The full keyname in Consul // Mutex guards all mutable metadata Mutex sync.Mutex Ctime time.Time // File attr Mtime time.Time // File attr Atime time.Time // File attr IsOpen bool // Is there an open handle to this file? Size uint64 // If the file is open, the expected file size Deleted bool // Whether this file has been deleted Buf []byte // If the file is deleted, buffers data locally }
File is a single file's inode in the filesystem. It is backed by a key in Consul.
func (*File) Attr ¶
Attr implements the Node interface. It is called when fetching the inode attribute for this file (e.g., to service stat(2)).
func (*File) BufferRead ¶
BufferRead returns locally-buffered file contents, which will only be used if the file is deleted.
func (*File) BufferTruncate ¶
func (*File) BufferWrite ¶
func (file *File) BufferWrite(req *fuse.WriteRequest, resp *fuse.WriteResponse) bool
func (*File) Fsync ¶
Fsync implements the NodeFsyncer interface. It is called to explicitly flush cached data to storage (e.g., on a fsync(2) call). Since data is not cached, this is a no-op.
func (*File) Open ¶
func (file *File) Open( ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse, ) (fs.Handle, error)
Open implements the NodeOpener interface. It is called the first time a file is opened by any process. Further opens or FD duplications will reuse this handle. When all FDs have been closed, Release() will be called.
func (*File) Read ¶
func (file *File) Read( ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse, ) error
Read implements the HandleReader interface. It is called to handle every read request. Because the file is opened in DirectIO mode, the kernel will not cache any file data.
func (*File) ReadAll_ ¶
ReadAll_ handles every read request by fetching the key from the server. This leads to simple consistency guarantees, as there is no caching, but performance may suffer in distributed settings. It intentionally does not implement the ReadAller interface to avoid the caching inherent in that interface.
func (*File) Release ¶
Release implements the HandleReleaser interface. It is called when all file descriptors to the file have been closed.
func (*File) SetDeleted ¶
SetDeleted marks this file as deleted.
If the file is open, Posix says those processes should continue to operate on the file as if it exists, but when they close, it is removed. These semantics are preserved by caching a copy of the file and operating on that copy, letting the key on Consul be deleted eagerly.
func (*File) Setattr ¶
func (file *File) Setattr( ctx context.Context, req *fuse.SetattrRequest, resp *fuse.SetattrResponse, ) error
Setattr implements the fs.NodeSetattrer interface. This is used by the kernel to request metadata changes, including the file's size (used by ftruncate(2) or by open("...", O_TRUNC) to clear a file's content).
func (*File) Truncate ¶
Truncate sets a key's value to the given size, stripping data off the end or adding \0 as needed. Note that a Consul Key-Value pair has two data segments, "value" and "flags," and this operation only changes the value. So to preserve the flags, a full read-modify-write must be done, even when the value is cleared entirely.
func (*File) Write ¶
func (file *File) Write( ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse, ) error
Write implements the HandleWriter interface. It is called on *every* write (DirectIO mode) to allow this module to handle consistency itself. Current strategy is to read the file, change the written portions, then write it back atomically. If the key was updated between the read and the write, try again.