rwvfs

README

rwvfs

Package rwvfs augments vfs to support write operations.

• rwvfs documentation
• golang.org/x/tools/godoc/vfs documentation

HTTP VFS

This repository includes HTTP server and client implementations of a read-write virtual filesystem. The HTTP server is a Go http.Handler that exposes any underlying RWVFS via HTTP. The HTTP client is a Go implementation of rwvfs.FileSystem that communicates with an external HTTP server. They both speak the same API on top of HTTP; see the code for details.

To spin up a test client and server, follow these instructions. (They require Go and a $GOPATH set; as a quickstart, install the latest Go from golang.org and then run export GOPATH=$HOME; export PATH=$PATH:$GOPATH/bin.)

Server

go get sourcegraph.com/sourcegraph/rwvfs/httpvfs
httpvfs &

go get sourcegraph.com/sourcegraph/rwvfs/httpvfs-client
httpvfs-client ls .
echo hello | httpvfs-client put foo
httpvfs-client cat foo
httpvfs-client rm foo

Future roadmap

• Add corresponding implementations of rwvfs.FileSystem for mapfs and zipfs

Documentation

Overview

Package rwvfs augments vfs to support write operations.

Index

• Variables
• func Glob(wfs WalkableFileSystem, prefix, pattern string) (matches []string, err error)
• func HTTPHandler(fs FileSystem, logTo io.Writer) http.Handler
• func HTTPHandlerWithDelay(fs FileSystem, logTo io.Writer, delay time.Duration) http.Handler
• func MkdirAll(fs FileSystem, path string) error
• func OpenFetcher(fs RangeOpener, name string) (vfs.ReadSeekCloser, error)
• func StatAllRecursive(path string, wfs WalkableFileSystem) ([]os.FileInfo, error)
• type Fetcher
• type FetcherOpener
• type FileSystem
  • func HTTP(base *url.URL, httpClient *http.Client) FileSystem
  • func Logged(log *log.Logger, fs FileSystem) FileSystem
  • func Map(m map[string]string) FileSystem
  • func OS(root string) FileSystem
  • func OSPerm(root string, filePerm, dirPerm os.FileMode) FileSystem
  • func ReadOnly(fs vfs.FileSystem) FileSystem
  • func Sub(fs FileSystem, prefix string) FileSystem
  • func Union(fileSystems ...FileSystem) FileSystem
• type LinkFS
• type MkdirAllOverrider
• type RangeOpener
• type WalkableFileSystem
  • func Walkable(fs FileSystem) WalkableFileSystem

Variables

var ErrOutsideRoot = errors.New("link destination is outside of filesystem")

ErrOutsideRoot occurs when a symlink refers to a path that is not in the current VFS.

var ErrReadOnly = errors.New("read-only VFS")

ErrReadOnly occurs when a write method (Create, Mkdir, Remove) is called on a read-only VFS (i.e., one created by ReadOnly).

Functions

func Glob

func Glob(wfs WalkableFileSystem, prefix, pattern string) (matches []string, err error)

Glob returns the names of all files under prefix matching pattern or nil if there is no matching file. The syntax of patterns is the same as in path/filepath.Match.

func HTTPHandler

func HTTPHandler(fs FileSystem, logTo io.Writer) http.Handler

HTTPHandler creates an http.Handler that allows HTTP clients to access fs. It should be accessed by clients created using this package's HTTP func.

func HTTPHandlerWithDelay

func HTTPHandlerWithDelay(fs FileSystem, logTo io.Writer, delay time.Duration) http.Handler

func MkdirAll

func MkdirAll(fs FileSystem, path string) error

MkdirAll creates a directory named path, along with any necessary parents. If path is already a directory, MkdirAll does nothing and returns nil.

func OpenFetcher

func OpenFetcher(fs RangeOpener, name string) (vfs.ReadSeekCloser, error)

OpenFetcher creates a new vfs.ReadSeekCloser based on the named file in fs that optimistically buffers reads. It is intended to be used when fs is a network filesystem with relatively high RTT (10msec+).

func StatAllRecursive

func StatAllRecursive(path string, wfs WalkableFileSystem) ([]os.FileInfo, error)

StatAllRecursive recursively stats all files and dirs in fs, starting at path and descending. The Name methods of the returned FileInfos returns their full path, not just their filename.

Types

type Fetcher

type Fetcher interface {
	// Fetch fetches the specified byte range (start inclusive, end
	// exclusive) from a remote (network-like) underlying source. Only
	// these bytes ranges are available to be read.
	Fetch(start, end int64) error
}

Fetcher is implemented by files that require explicit fetching to buffer data from remote (network-like) underlying sources.

type FetcherOpener

type FetcherOpener interface {
	OpenFetcher(name string) (vfs.ReadSeekCloser, error)
}

FetcherOpener is implemented by FileSystems that support a mode of operation where OpenFetcher returns a lazily loaded file. This is useful for VFS implementations that are backed by slow (e.g., network) data sources, so you can explicit fetch byte ranges at a higher level than Go I/O buffering.

type FileSystem

type FileSystem interface {
	vfs.FileSystem

	// Create creates the named file, truncating it if it already exists.
	Create(path string) (io.WriteCloser, error)

	// Mkdir creates a new directory. If name is already a directory, Mkdir
	// returns an error (that can be detected using os.IsExist).
	Mkdir(name string) error

	// Remove removes the named file or directory.
	Remove(name string) error
}

func HTTP

func HTTP(base *url.URL, httpClient *http.Client) FileSystem

HTTP creates a new VFS that accesses paths on an HTTP server.

func Logged

func Logged(log *log.Logger, fs FileSystem) FileSystem

Logged creates a new VFS wrapper that logs and times calls to an underlying VFS.

func Map

func Map(m map[string]string) FileSystem

Map returns a new FileSystem from the provided map. Map keys should be forward slash-separated pathnames and not contain a leading slash.

func OS

func OS(root string) FileSystem

OS returns an implementation of FileSystem reading from the tree rooted at root.

func OSPerm

func OSPerm(root string, filePerm, dirPerm os.FileMode) FileSystem

func ReadOnly

func ReadOnly(fs vfs.FileSystem) FileSystem

ReadOnly returns a FileSystem whose write methods (Create, Mkdir, Remove) return errors. All other methods pass through to the read-only VFS.

func Sub

func Sub(fs FileSystem, prefix string) FileSystem

Sub returns an implementation of FileSystem mounted at prefix on the underlying fs. If fs doesn't have an existing directory at prefix, you can can call Mkdir("/") on the new filesystem to create it.

func Union

func Union(fileSystems ...FileSystem) FileSystem

Union returns a new FileSystem which is the union of the provided file systems. For read operations, vfs.NameSpace is used and its behavior is inherited. Write operations are applied to the first file system which contains the parent directory. The union file system is not thread-safe. Concurrent access to itself and/or its underlying file systems requires synchronization.

type LinkFS

type LinkFS interface {
	// Symlink creates newname as a symbolic link to oldname.
	Symlink(oldname, newname string) error

	// ReadLink returns the destination of the named symbolic link.
	ReadLink(name string) (string, error)
}

A LinkFS is a filesystem that supports creating and dereferencing symlinks.

type MkdirAllOverrider

type MkdirAllOverrider interface {
	MkdirAll(path string) error
}

MkdirAllOverrider can be implemented by VFSs for which MkdirAll requires special behavior (e.g., VFSs without any discrete notion of a directory, such as Amazon S3). When MkdirAll is called on MkdirAllOverrider implementations, it calls the interface's method and passes along the error (if any).

type RangeOpener

type RangeOpener interface {
	// OpenRange returns a vfs.ReadSeekCloser to read the specified
	// range from the named file. The rangeHeader parameter must be in
	// the format of HTTP Range headers (e.g., "bytes=123-456"). An
	// empty rangeHeader is interpreted to mean the entire file.
	OpenRange(name string, rangeHeader string) (vfs.ReadSeekCloser, error)
}

A RangeOpener is a filesystem that can efficiently open a range within a file (instead of the whole file).

type WalkableFileSystem

type WalkableFileSystem interface {
	FileSystem
	Join(elem ...string) string
}

func Walkable

func Walkable(fs FileSystem) WalkableFileSystem

Walkable creates a walkable VFS by wrapping fs.