fs

package module
v0.22.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 14, 2025 License: MIT Imports: 14 Imported by: 20

README

fs

The package fs offers a more complete altenative to the io/fs package of the Go standard library.

The idea was to find common abstractions for paths and filesystems, no matter, if they are local, remote, backed by memory, drives of cloud servers.

API docs

see https://pkg.go.dev/gitlab.com/golang-utils/fs

design decisions

If became clear soon, that such a general fs library can't be done without rethinking and integrating the concept of a path.

Paths should work in a similar way, no matter, if they are local windows paths, windows UNC paths or unix paths. No matter if the path is part of a URL or local, if it has a drive letter or a network share.

The fs/path package makes Path a proper type and has relative and absolute paths as different types. Every path can have a relative part and every path can be represented by a string.

Therefor the common path.Path interface is as follows:

type Path interface {
	String() string
	Relative() Relative
}

where path.Relative is just a string:

type Relative string

However, since a path.Relative is also a path.Path, it implements that interface:

func (r Relative) String() string {
	return string(r)
}

func (r Relative) Relative() Relative {
	return r
}

Very simple. It became clear, that an absolute path is always related to a filesystem, while a relative path is independant of the filesystem. Therefor path.Absolute became an interface, since the differences between local os paths and also remote paths show up in their absolute part:

type Absolute interface {
	Path
	Head() string
}

So a path.Absolute is just a path.Path with a Head. The easiest implementation of this path.Absolute is a local path:

type Local [2]string

func (a Local) Head() string {
	return a[0]
}

func (a Local) String() string {
	return a[0] + a[1]
}

func (a Local) Relative() Relative {
	return Relative(a[1])
}

Here some design decision come into place:

  1. the relative part of a path always uses the slash / as a separator.
  2. directories always end with a /. this makes it easy to check, if a path refering to a directory without the the need of a filesystem.
  3. the head of an absolute path is always a directory and therefor it always ends with a slash /
  4. parts of paths are joined together simply by glueing them together. Since a directory must end in a slash / this naturally leads to correct paths.
  5. the head of an absolute path is depending on the filesystem that it refers to: e.g.
    • a local windows paths starts with a drive letter, e.g. c:/
    • a windows share in UNC starts with the host followed by the share name, e.g. \\example.com/share/
    • a url starts with a schema, followed by a host http://example.com/
  6. absolute paths can be written differently, e.g.
    • c:/ can also be written as C:\
    • \\example.com/share/ can also be written as \\example.com\share\ therefor we need one unique internal representation, while allowing the path to be generated by parsing also the alternative ways of writing. this leads to parsers for converting an absolute path string to a path.Absolute.
  7. While having a unified syntax behind the scenes, path.Local can be converted to the most typical string notation that is used on the running system, by calling the ToSystem method, so that it can easily be integrated with external tools
  8. The only time where a local absolute path is being created via solo a relative path, is when the relative path is relative to the current working directory of the system. This case is handled like every other way to convert a string to a path.Local by the path.ParseLocal() function (see below)

Local and Remote

Not only because of the different ways the local and remote absolute paths are written, but also because of the very different performance characteristics and optimization opportunities, it makes sense to be able to distinguish between local and remote paths, while still being able to handle them both as absolute paths.

This is taken into account via having path.Local as well as path.Remote implement the path.Absolute interface. This way we end up with two parsers:

  • ParseLocal(string) handling local windows, UNC and unix paths, also paths relativ to the working directory, e.g. ./a/
  • ParseRemote(string) handling URLs

A path.Remote is basically just a wrapper around an url.URL that is implementing the path.Absolute interface.

type Remote struct {
	*url.URL
}

func (u *Remote) Relative() Relative {...}
func (u *Remote) Head() string {...}

There are some helpers, to get the common string notation for windows, UNC and so on back, so that everything does integrate well.

filesystems

Since it became clear that absolute paths are associated with a filesystem, this lead to filesystems being initiated via an path.Absolute.

For good integration with the existing io/fs.FS interface which provides only a solution for reading access, we started with the fs.ReadOnly filesystem interface:

type ReadOnly interface {
	Reader(p path.Relative) (io.ReadCloser, error)
	Exists(p path.Relative) bool
	ModTime(p path.Relative) (time.Time, error)
	Abs(p path.Relative) path.Absolute
	Size(p path.Relative) int64
}

It is very easy to convert an existing io/fs.FS implementation to the fs.ReadOnly interface via the wrapfs package:

dir := "/etc"
fsys := os.DirFS(dir)
ro, err := wrapfs.New(fsys, path.MustLocal(dir+"/")) 
size := ro.Size("fstab")

For os.DirFS there is an easier way via the localfs package:

fs, err := localfs.New(path.MustLocal("/etc/"))
size := fs.Size("fstab")

The real power comes with the more general fs.FS interface:

type FS interface {
	ReadOnly
	ExtWriteable
	ExtDeleteable
}

It adds to the ReadOnly interface the ability to write and delete files and folders. This is also implemented by the localfs package:

fs, err := localfs.New(path.MustLocal(`C:\`))
recursive := true
err = fs.Delete(path.Relative("Windows/"), recursive)

But the same powerful interface is also implemented by the httpsfs package which accesses a filesystem via http:

fs, err := httpsfs.New(path.MustRemote(`http://localhost:3030/data/`))
createDirs := true
err = fs.Write(path.Relative("myblog/january/something-new.txt"), fs.ReadCloser(strings.NewReader("some text")), createDirs)

Finally we have some properties specifically for local filesystems that we don't have for remote filesystems and vice versa:

fs, err := localfs.New(path.MustLocal(`/`))
err = fs.SetMode(path.Relative("etc/"), 0750)

fs, err := httpsfs.New(path.MustRemote(`http://localhost:3030/data/`))
meta := map[string][]byte{"Content-Type": []byte("application/json")}
data := fs.ReadCloser(strings.NewReader(`{key: "val"}`))
err = fs.WriteWithMeta(path.Relative("sth.json"), data, meta, true)

So we have this hierarchy of FS interfaces where the last ones a more specific but also more powerfull and the first ones are more general and easier to implement:

type ReadOnly interface {
	Reader(p path.Relative) (io.ReadCloser, error)
	Exists(p path.Relative) bool
	ModTime(p path.Relative) (time.Time, error)
	Abs(p path.Relative) path.Absolute
	Size(p path.Relative) int64 
}

type FS interface {
	ReadOnly
	ExtWriteable
	ExtDeleteable
}

type Local interface {
	FS
	ExtMoveable
	ExtModeable
	ExtRenameable
	ExtSpaceReporter
}

type Remote interface {
	FS
	ExtMeta
	ExtURL
}

Finally we have TestFS interface that is covering everything, so that can easily test our packages against all features:

type TestFS interface {
	Local
	Remote
}

The mockfs package offers an implementation of the fs.TestFS interface that is backed by a map for easy testing.

For implementors

For implementors there is a large test suite that can be easily integrated into your package testing to ensure that your filesystem behaves correctly according to the specifications. Here an example how to use it, based on the mockfs package:

package mockfs

import (
	"testing"

	"gitlab.com/golang-utils/fs"
	"gitlab.com/golang-utils/fs/path"
	"gitlab.com/golang-utils/fs/spec"
)

func mustNew(loc path.Absolute) fs.TestFS {
	f, err := New(loc)
	if err != nil {
		panic(err.Error())
	}
	return f
}

func TestSpec(t *testing.T) {
	var c spec.Config
	s := spec.TestFS(c, mustNew)
	s.Run("", t)
}

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrNotSupported      = errors.Error("%s not supported by %T")
	ErrNotFound          = errors.Error("%s not found")
	ErrNotEmpty          = errors.Error("%s not empty")
	ErrAlreadyExists     = errors.Error("%s already exists")
	ErrWhileReading      = errors.Error("could not read %s: %w")
	ErrExpectedFile      = errors.Error("%s is not a file")
	ErrExpectedDir       = errors.Error("%s is not a directory")
	ErrExpectedAbsPath   = errors.Error("%s is no absolute path")
	ErrExpectedWinDrive  = errors.Error("%s is no windows drive letter")
	ErrWhileChecksumming = errors.Error("could not make checksum for file %s: %w")
	ErrChecksumNotMatch  = errors.Error("checksums did not match %v vs %v")
)
View Source 
var SkipAll = iofs.SkipAll
View Source 
var SkipDir = iofs.SkipDir

Functions

func Base 

func Base(fsys ReadOnly) path.Absolute

func Checksum 

func Checksum(data io.Reader) (checksum string, err error)

see https://stackoverflow.com/questions/60328216/how-to-calculate-sha256-of-a-very-large-file-in-go

func ClearDir added in v0.5.0 

func ClearDir(fsys FS, dir path.Relative) error

func CopyDir 

func CopyDir(srcfs FS, srcDir path.Relative, trgtfs FS, trgtDir path.Relative) error

TODO implement

func CopyFile 

func CopyFile(srcfs FS, srcFile path.Relative, trgtfs FS, trgtFile path.Relative) error

TODO implement TODO: first check, if there is enough space left CopyFile copies the given file of the given FS to the given target in the given target fs. srcfs and trgtfs may be the same. In that case, and if FS also implements the Mover interface and if srcFile and trgtFile are on the same Mountpoint the more efficient Mover.Move method is used

func CopyFileWithCheck 

func CopyFileWithCheck(srcfs FS, srcFile path.Relative, trgtfs FS, trgtFile path.Relative, checksumFN func(io.Reader) (string, error)) error

CopyFileWithCheck is like CopyFile but doing another read to verify, that the checksum of both the src and target file matches. if the checksum does not match, the targetfile is removed

func CreateFile added in v0.6.0 

func CreateFile(fs ExtWriteable, f path.Relative, bt []byte) error

CreateFile does return an error, if the file already exists or if the parent directory does not exist

func CreateFileFrom added in v0.6.0 

func CreateFileFrom(fs ExtWriteable, f path.Relative, rd io.Reader) error

CreateFileFrom does return an error, if the file already exists or if the parent directory does not exist

func DirSize 

func DirSize(fsys ReadOnly, dir path.Relative) (size int64)

DirSize returns the size of the given directory. If the given path is not a directory or any error occurs, -1 is returned. If the given dir is empty, 0 is returned. TODO test

func Drive added in v0.17.0 

func Drive(loc path.Local) (string, error)

func FreeSpace added in v0.17.0 

func FreeSpace(drive string, freeBytesAvailable *uint64) error

see https://stackoverflow.com/questions/20108520/get-amount-of-free-disk-space-using-go

func IsEmpty 

func IsEmpty(fsys ReadOnly, dir path.Relative) bool

IsEmpty returns, if the given dir is empty or does not exist or is not a directory, but a file.

func MkDir 

func MkDir(fs ExtWriteable, rel path.Relative) error

func MkDirAll 

func MkDirAll(fs ExtWriteable, rel path.Relative) error

func MoveDir 

func MoveDir(srcfs FS, srcDir path.Relative, trgtfs FS, trgDir path.Relative) error

TODO: implement

func MoveFile 

func MoveFile(srcfs FS, srcFile path.Relative, trgtfs FS, trgDir path.Relative) error

MoveFile moves a file. If source and target filesystems are the same, the mountpoints of the source and target paths are the same and the filesystem is supporting the Moveable interface, the optimized Move method of the filesystem is used. Otherwise a copy of the file is made and the source file is only deleted, if the copying process was successfull. TODO check

func OnUnix added in v0.17.0 

func OnUnix() bool

func OnWindows added in v0.17.0 

func OnWindows() bool

func ReadCloser 

func ReadCloser(rd io.Reader) io.ReadCloser

func ReadDirNames added in v0.5.0 

func ReadDirNames(fs ReadOnly, dir path.Relative) ([]string, error)

ReadDirNames returns the names of the dirs and files in a directory and returns them

func ReadDirPaths added in v0.5.0 

func ReadDirPaths(fs ReadOnly, dir path.Relative) ([]path.Relative, error)

ReadDirPaths returns the full paths (including dir) of the files and folders inside dir

func ReadFile 

func ReadFile(fs ReadOnly, f path.Relative) ([]byte, error)

func WriteFile added in v0.6.0 

func WriteFile(fs ExtWriteable, f path.Relative, bt []byte, recursive bool) error

func WriteFileFrom added in v0.6.0 

func WriteFileFrom(fs ExtWriteable, f path.Relative, rd io.Reader, recursive bool) error

Types

type ExtClose added in v0.9.0 

type ExtClose interface {
	ReadOnly

	Close() error
}

type ExtDeleteable 

type ExtDeleteable interface {
	ReadOnly

	// Delete deletes either a file or a folder.
	// The deletion of folders may not be supported. In this case an error has to be returned, if the given path
	// is a folder.
	// If recursive is set to true, directories will be deleted, even if they are not empty.
	// If the given path does not exist, no error is returned.
	// If the given path can't be deleted, either because it is not empty and recursive was not set, or
	// because of missing permissions, an error is returned.
	Delete(p path.Relative, recursive bool) error
}

ExtDeleteable is an extension interface for a filesystem that adds the ability to delete files and folders. The deletion of folders may not be supported. In this case an error has to be returned, if the given path is a folder.

type ExtGlob added in v0.4.0 

type ExtGlob interface {
	ReadOnly

	Glob(pattern string) (matches []path.Relative, err error)
}

type ExtLock added in v0.4.0 

type ExtLock interface {
	ReadOnly

	Lock(p path.Relative) error
	Unlock(p path.Relative) error
}

type ExtMeta 

type ExtMeta interface {
	ReadOnly

	WriteWithMeta(p path.Relative, data io.ReadCloser, meta map[string][]byte, recursive bool) error
	SetMeta(p path.Relative, meta map[string][]byte) error
	GetMeta(p path.Relative) (map[string][]byte, error)
}

ExtMeta is an interface for a filesystem, that adds ability to set and get meta data

type ExtModeable 

type ExtModeable interface {
	ReadOnly

	// WriteWithMode behaves like Writeable.Write and only differs that the given filemod is used within the creation
	// (instead of the default filemods).
	WriteWithMode(p path.Relative, data io.ReadCloser, mode FileMode, recursive bool) error

	// GetMode returns the FileMode of the given path, or an error if the path does not exist.
	GetMode(p path.Relative) (FileMode, error)

	// SetMode set the given FileMode for the given path. If the path does not exist, an error is returned.
	SetMode(p path.Relative, m FileMode) error
}

ExtModeable is an interface for a filesystem, that adds the ability to read and set FileMode (permissions) for files and folders. To allow the creation of files with restricted permissions, this interface includes a method to write with a given mode, which is not supported by the FS interface. ExtModeable should be implemented mainly by local filesystems.

type ExtMoveable 

type ExtMoveable interface {
	ReadOnly

	// Drive returns the drive (drive letter or UNC on windows, mountpoint of unix) of the given relative path
	// for moving between mountpoints or filesystem, just copy all files via CopyFile or CopyDir and
	// remove afterwards (better to check the copy by comparing checksums or the original and the new file)
	Drive(p path.Relative) (path.Local, error)

	// Move should return an error, if
	// - the src does not exist
	// - src is a directory and moving of directories is not supported by the fs
	// - the resulting path trgDir.Join(path.Name(src)) does already exist
	// - src and trgdir have different mountpoints
	Move(src path.Relative, trgDir path.Relative) error
}

ExtMoveable is an interface for a filesystem, that adds the native / optimized (e.g. zero copy) ability to move files and folders.

type ExtReadSeekable added in v0.21.0 

type ExtReadSeekable interface {
	ReadOnly

	// ReadSeeker returns an io.ReadSeekCloser for the content of a file.
	// The io.ReadSeekCloser should be closed after reading all.
	// An error is only returned, if p is a directory or if p does not exist.
	ReadSeeker(p path.Relative) (ReadSeekCloser, error)
}

ExtReadSeekable is an extention interface for a filesystem that adds the ability to seek in files.

type ExtRenameable 

type ExtRenameable interface {
	ReadOnly

	// Rename deletes either a file or a folder.
	// The given name must not be a path, so it must not contain path seperators.
	// If the given path does not exist, an error is returned.
	// If the renaming procedure was not successfull, an error is returned.
	// Please note that Rename behaves differently from the traditional unix operation with the same name.
	// Rename can't be used to move a file or folder to a different folder, but only to change the name of the
	// file or folder while staying in the same folder.
	// To move files, you need the Move function which would be more efficient, if the filesystem
	// supports the Moveable interface.
	Rename(p path.Relative, name string) error
}

ExtRenameable is an interface for a filesystem, that adds the ability to rename files and folders. The renaming of folders may not be supported. In this case an error has to be returned, if the given path is a folder.

type ExtSpaceReporter 

type ExtSpaceReporter interface {
	ReadOnly

	// FreeSpace returns how many bytes of space are left on the mountpoint of the given path
	FreeSpace(p path.Relative) int64
}

type ExtURL 

type ExtURL interface {
	ReadOnly

	HostName() string
	Port() int
	UserName() string
	Password() string
	Scheme() string
}

ExtURL is an interface for a filesystem, that adds the ability to get url based information

type ExtWriteSeekable added in v0.21.0 

type ExtWriteSeekable interface {
	ExtWriteable

	// WriteSeeker returns a WriteSeekCloser where Write and Seek can be called. Close must be called when the writing is finished
	WriteSeeker(p path.Relative) (WriteSeekCloser, error)
}

ExtWriteSeekable is an extention interface for a filesystem that adds the ability to seek in files.

type ExtWriteable 

type ExtWriteable interface {
	ReadOnly

	// Write writes either a file or a folder to the given path with default permissions.
	// If p is a directory, data is ignored.
	// If recursive is set to true, all intermediate directories will be created if necessary.
	// Write always overwrites an existing file. If you need to make sure that a file at the given
	// path should not be overwritten, check with Exists(p) first.
	// If p is a directory that already exists, nothing will be done and no error will be returned.
	Write(p path.Relative, data io.ReadCloser, recursive bool) error
}

ExtWriteable is an extention interface for a filesystem that adds the ability to write files and folders.

type FS 

type FS interface {
	ReadOnly
	ExtWriteable
	ExtDeleteable
}

FS is an interface for a complete filesystem, that covers the ability to read, write, delete and rename files and folders.

type FileMode 

type FileMode = iofs.FileMode
const (
	DefaultDirMode  FileMode = 0750
	DefaultFileMode FileMode = 0640
)

type Local 

type Local interface {
	FS
	ExtReadSeekable
	ExtWriteSeekable
	ExtMoveable
	ExtModeable
	ExtRenameable
	ExtSpaceReporter
	ExtGlob
}

Local is the interface for a local filesystem

type ReadOnly 

type ReadOnly interface {
	// Reader returns an io.ReadCloser that reads in the whole file or in case of a directory
	// the relative paths related to the directory path, separated by a unix linefeed (\n).
	// The io.ReadCloser should be closed after reading all.
	// An error is only returned, if p does not exist or in case of directories, if reading of directories is not supported.
	Reader(p path.Relative) (io.ReadCloser, error)

	// Exists does return, if a file or directory with the given path exists.
	Exists(p path.Relative) bool

	// ModTime return the time of the last modification of p.
	// It returns an error, if p does not exist.
	ModTime(p path.Relative) (time.Time, error) // wenn nicht unterstützt: immer fehler zurückgeben

	// Abs converts the given relative path to an absolute path, based on the base of the filesystem.
	Abs(p path.Relative) path.Absolute

	// Size returns the size of the given p in bytes
	// If the given p does not exist or the size could not be determined, -1 is returned.
	// The size of a folder is always 0. In order to calculate the real size of a folder,
	// use DirSize()
	Size(p path.Relative) int64 // int64 is large enough for any filesystem
}

ReadOnly is a minimal readonly interface for a filesystem. Any "io/fs.FS" can be converted to the ReadOnly interface by calling wrapfs.New()

type ReadSeekCloser added in v0.11.0 

type ReadSeekCloser interface {
	io.Reader
	//in contrast to io.Seeker, here Seek always Seeks from the beginning of the file (whence=0)
	// other behaviors can be simply implemented by tracking (whence=1, starting from last position)
	// or via calculation by file size( whence=2, from the end of the file)
	Seek(offset int64) error
	io.Closer
}

func NewReadSeekCloser added in v0.21.0 

func NewReadSeekCloser(rd io.ReadSeeker) ReadSeekCloser

type Remote 

type Remote interface {
	FS
	ExtMeta
	ExtURL
}

Remote is the interface for a remote filesystem

type TestFS 

type TestFS interface {
	Local
	Remote
}

TestFS is an interface that is only interesting for testing. It includes all interfaces and extension.

type WalkFunc added in v0.4.0 

type WalkFunc func(p path.Relative, err error) error

type WriteSeekCloser added in v0.21.0 

type WriteSeekCloser interface {
	io.Writer
	//in contrast to io.Seeker, here Seek always Seeks from the beginning of the file (whence=0)
	// other behaviors can be simply implemented by tracking (whence=1, starting from last position)
	// or via calculation by file size( whence=2, from the end of the file)
	Seek(offset int64) error

	// when writing is finished Close must be called
	io.Closer
}

Source Files

View all Source files

Directories

Path Synopsis
filesystems
dirfs
mapfs
mockfs
rootfs
tempfs
wrapfs
internal
original/iofs
specs/core_fs
specs/core_local
specs/core_readonly
specs/core_remote
specs/core_testfs
specs/ext_deleteable
specs/ext_glob
specs/ext_lock
specs/ext_meta
specs/ext_modeable
specs/ext_moveable
specs/ext_renameable
specs/ext_seekable
specs/ext_spacereporter
specs/ext_url
specs/ext_writeable
specs/ext_writeseekable

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.