filestore

README

filestore

The filestore package provides a facade that encapsulates the common operations of a readable/writable file system. This could be the actual underlying disk, an in-memory store, S3, whatever.

Currently, this package only ships with an implementation that uses the underlying disk file system. Over time, I might offer more options through plugins, but disk is all I needed when I wrote it, so that's what's available :)

WARNING

This is absolutely a work in progress, and I am VERY likely to completely change the interfaces as I use it in "real" projects and realize my abstractions could use some work. Should you stumble upon this and want to use it, go for it. I would, however, suggest that you version lock to a specific version as I'm quite likely to check in breaking changes to meet my own needs until I feel good about the API.

Getting Started

go get -u github.com/monadicstack/filestore

Basic Usage

You can create a filestore.FS that utilizes your local disk, and all of the operations are implicitly based in that location. From there, you have most of the standard file system operations you would have on the Unix command line:

fs := filestore.Disk("data")

List Files in a Directory

Your fs is already tied to the data/ directory, so if you wanted to list its contents, you would just do the following:

files, err := fs.List(".")
if err != nil {
    // handle error
}
for _, file := range files {
    fmt.Printf("%s [Dir=%v]\n", file.Name(), file.IsDir())
}

Additionally, you could list the contents of any child directory within data/ by specifying the path. For instance, to list the contents of data/images/logos/ it would look like this:

files, err := fs.List("images/logos")

Lastly, you can filter results down to just those that meet specific criteria. The filestore package ships with a few common filters for things like file name pattern or extension, but you can provide any function that matches the filter signature:

pngFiles, err := fs.List("images/logos", filestore.WithExt("png"))

Reading/Writing Files

The filestore package makes it easy to read/write files in the underlying file system. Let's say you wanted to read the file data/images/logos/splash-256.png, here's what that would look like. Notice, that the filestore.ReaderFile returned by Read() implements io.Reader, so you can use it with any of Go's standard stream processing operations:

file, err := fs.Read("images/logos/splash-256.png")
if err != nil {
	// handle error
}
defer file.Close()

img, err := png.Decode(file)

Writing files is just as idiomatic. The Write() operation returns a filestore.WriterFile which implements io.Writer, so you can hook into your favorite stream processing code.

// If the data/conf/ directory doesn't exist, Write() will
// lazily create it on the fly for you!!!
file, err := fs.Write("conf/config.json")
if err != nil {
    // handle error
}
defer file.Close()

file.Write([]byte(`{"timeout":"10s"}`))

Other Handy Operations

The idea this package is to give you most of the same tools/behaviors that you'd have on the command line.

Delete a single file or even a whole directory and everything in it...

// Single file
err := fs.Remove("conf/config.json")
// Whole directory
err := fs.Remove("tmp/uploads")

Move a file or directory

// Single file
err := fs.Move("uploads/tmp/upload.png", "images/logos/splash-256.png")
// Whole directory
err := fs.Move("uploads/processing", "uploads/done")

Get a filestore.FS that is scoped to a subdirectory...

logos := fs.ChangeDirectory("images/logos")
logoFiles, err := logos.List(".", filestore.WithExts("png", "jpg"))

// Prints "data/images/logos"
fmt.Println(logos.WorkingDirectory())

Check if a file exists or not...

if fs.Exists("conf/config.json") {
    // Do something interesting
}

Get name/size/type/etc. details about a file w/o reading it.

info, err := fs.Stat("conf/config.json")
fmt.Printf("%s [Size=%d][Dir=%v]\n", info.Name(), info.Size(), info.IsDir())

Documentation

Index

• func ChangeExtension(fileName string, ext string) string
• type DiskFS
  • func Disk(basePath string) *DiskFS
  • func (d DiskFS) ChangeDirectory(dir string) FS
  • func (d DiskFS) Exists(filePath string) bool
  • func (d DiskFS) List(dirPath string, filters ...FileFilter) ([]FileInfo, error)
  • func (d DiskFS) Move(fromPath string, toPath string) error
  • func (d DiskFS) Read(filePath string) (ReaderFile, error)
  • func (d DiskFS) Remove(fileOrDirPath string) error
  • func (d DiskFS) Stat(filePath string) (FileInfo, error)
  • func (d DiskFS) WorkingDirectory() string
  • func (d DiskFS) Write(filePath string) (WriterFile, error)
• type FS
• type FileFilter
  • func WithEverything() FileFilter
  • func WithExt(extension string) FileFilter
  • func WithExts(extensions ...string) FileFilter
  • func WithPattern(pattern string) FileFilter
• type FileInfo
• type ReaderFile
• type WriterFile

Functions

func ChangeExtension

func ChangeExtension(fileName string, ext string) string

ChangeExtension helps datasets maintain the same file name stem while replacing the extension.

// Example
changeExtension("foo.jpg", "txt")  // "foo.txt"
changeExtension("foo.bar.png", "jpg")  // "foo.bar.jpg"
changeExtension("foo", "txt")  // "foo.txt"

Types

type DiskFS

type DiskFS struct {
	// contains filtered or unexported fields
}

DiskFS is a file store whose operations interact w/ the local file system.

func Disk

func Disk(basePath string) *DiskFS

Disk creates a new file store that reads and writes files to/from the local file system. All operations will be rooted in the given directory.

Example:

files := Disk("./data")

// Open ./data/input.txt for reading
input, err := files.Read("input.txt")
if err != nil {
    // handle your error nicely
}
defer input.Close()

inputBytes, err := io.ReadAll(input)

func (DiskFS) ChangeDirectory

func (d DiskFS) ChangeDirectory(dir string) FS

ChangeDirectory returns a new FS that is rooted in the given subdirectory of this FS.

func (DiskFS) Exists

func (d DiskFS) Exists(filePath string) bool

Exists returns true when the file/directory already exits in the file system.

func (DiskFS) List

func (d DiskFS) List(dirPath string, filters ...FileFilter) ([]FileInfo, error)

List performs the equivalent of the "ls" command. It returns a slice of all files and directories found in the target dirPath.

You can optionally provide a set of filters to limit which files/directories are included in the final set.

func (DiskFS) Move

func (d DiskFS) Move(fromPath string, toPath string) error

Move takes an existing file at the fromPath location and moves it to another spot in this file system; the toPath location.

func (DiskFS) Read

func (d DiskFS) Read(filePath string) (ReaderFile, error)

Read opens the given file at the given path, providing you with an io.Reader that you can use to stream bytes from it.

func (DiskFS) Remove

func (d DiskFS) Remove(fileOrDirPath string) error

Remove deletes the given file/directory and any of its children.

func (DiskFS) Stat

func (d DiskFS) Stat(filePath string) (FileInfo, error)

Stat fetches metadata about the file w/o actually opening it for reading/writing.

func (DiskFS) WorkingDirectory

func (d DiskFS) WorkingDirectory() string

WorkingDirectory returns the current FS context's path/directory.

func (DiskFS) Write

func (d DiskFS) Write(filePath string) (WriterFile, error)

Write opens the given file at the given path for writing. The resulting file behaves like a standard io.Writer/At.

This operation will attempt to lazy-create the parent directory(s) if it does not exist. Should the file already exist, this will overwrite its entire contents so that it only contains what you write this time.

type FS

type FS interface {
	// WorkingDirectory returns the current FS context's path/directory.
	WorkingDirectory() string
	// Stat fetches metadata about the file w/o actually opening it for reading/writing.
	Stat(path string) (FileInfo, error)
	// Read opens the given file for reading.
	Read(path string) (ReaderFile, error)
	// Write opens the given file for writing
	Write(path string) (WriterFile, error)
	// Exists returns true when the file/directory already exits in the file system.
	Exists(path string) bool
	// List performs a UNIX style "ls" operation, giving you the names of each file
	// in the given directory. The filters offer a way to limit which files/dirs are included
	// in the final slice.
	//
	// Example:
	//
	//    filesAndDirs, err := myFS.List("./conf")
	//    jsonFiles, err := myFS.List("./conf", filestore.WithExt("json"))
	List(path string, filters ...FileFilter) ([]FileInfo, error)
	// ChangeDirectory creates a new FS in the given subdirectory. All operations on this new
	// instance will be rooted in the given directory.
	//
	// It should NOT matter if the directory exists or not. You still always get a valid FS
	// pointing to that location and only get an error when you attempt to perform some other operation.
	//
	// Example:
	//
	//    usrFS := Disk("/usr")
	//    usrLocalBinFS := usrFS.ChangeDirectory("local/bin")
	ChangeDirectory(path string) FS
	// Remove deletes the given file/directory within the file system. If the given path
	// is a directory, it should recursively delete it and its children. Additionally,
	// if you attempt to remove a file/directory that does not exist, this should behave
	// quietly as a nop, returning a nil error but not changing the store's state.
	//
	// Example:
	//
	//    myDocumentsFS := Disk("/Users/rob/Documents")
	//    err = myDocumentsFS.Remove("foo.txt")
	//    if err != nil {
	//        // could not delete file "foo.txt"
	//    }
	//    err = myDocumentsFS.Remove("Pictures")
	//    if err != nil {
	//        // could not delete directory "Pictures/"
	//    }
	Remove(path string) error
	// Move takes an existing file at the fromPath location and moves it to another
	// spot in this file system; the toPath location.
	Move(fromPath string, toPath string) error
}

FS represents a file system that you can interact with its directories and files.

type FileFilter

type FileFilter func(info FileInfo) bool

FileFilter provides a way to exclude files/directories from a list/search.

func WithEverything

func WithEverything() FileFilter

WithEverything is a dummy non-nil file filter you can use to act as though there are no filters. Basically it behaves such that all files match.

func WithExt

func WithExt(extension string) FileFilter

WithExt creates a file filter that only accepts files that have a specific extension.

func WithExts

func WithExts(extensions ...string) FileFilter

WithExts creates a file filter that only accepts files that have one of the given extensions.

func WithPattern

func WithPattern(pattern string) FileFilter

WithPattern only allows files to pass through that match the given glob pattern.

type FileInfo

type FileInfo fs.FileInfo

FileInfo contains 'stat' info about a file or directory.

type ReaderFile

type ReaderFile interface {
	io.ReadCloser
	io.ReaderAt
	io.Seeker
}

ReaderFile encapsulates a file within a file system that you can read from.

type WriterFile

type WriterFile interface {
	io.WriteCloser
	io.WriterAt
	io.Seeker
}

WriterFile encapsulates a file within a file system that you can write to.