pfs

package
v0.4.140 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 14, 2023 License: ISC Imports: 21 Imported by: 3

Documentation

Overview

Package pfs provides a symlink-following file-systemtraverser and other file-system functions.

Index

Constants

View Source 
const (
	// AllModeBits is all known github.com/fsnotify/fsevents.EventFlags
	AllModeBits fs.FileMode = fs.ModeDir | fs.ModeAppend | fs.ModeExclusive |
		fs.ModeTemporary | fs.ModeSymlink | fs.ModeDevice | fs.ModeNamedPipe |
		fs.ModeSocket | fs.ModeSetuid | fs.ModeSetgid | fs.ModeCharDevice |
		fs.ModeSticky | fs.ModeIrregular
)
View Source 
const (
	DropPathsNotValues = true
)
View Source 
const (
	RetainSymlinks = false
)

Variables

View Source 
var ErrEndListing = errors.New("endListing")
View Source 
var St = parl.Debug

Functions

func Abs added in v0.4.23 

func Abs(dir string) (out string)

Abs ensures a file system path is fully qualified. Abs is single-return-value and panics on troubles

func AbsEval 

func AbsEval(path string, retainSymlinks ...bool) (absPath string, err error)

AbsEval returns an absolute path with resolved symlinks

  • use of current directory to make path absolute
  • “.” is returned for empty string
  • if retainSymlinks is RetainSymlinks, symlinks are returned
  • correct multiple Separators, eliminate “.”, eliminate inner and inappropriate “..”, ensure platform Separator

func AddDirEntry added in v0.4.138 

func AddDirEntry(abs string) (dirEntry fs.DirEntry, err error)

AddDirEntry returns an fs.DirEntry with fs.FileInfo available

func AddStatDirEntry added in v0.4.138 

func AddStatDirEntry(abs string) (dirEntry fs.DirEntry, err error)

AddDirEntry returns an fs.DirEntry for the target of a symbolic link

func Dirs 

func Dirs(dir string, callback ...func(dir string) (err error)) (dirs []string, err error)

Dirs retrieves absolute paths to all directories, while following symlinks, from initial dir argument. callback: cb is 6–58% faster than slice, results are found faster, and it can be canceled midway. if callback blocks, not good…

func Elems added in v0.4.26 

func Elems(path string) (dirs []string, file string)

Elems splits a path into a list of directory names and base filename part. if path is absolute, dirs[0] is "/". if there is no separator in path, dirs is empty. if path is empty string, dirs is empty and file is empty string.

func EnsureDirectory 

func EnsureDirectory(directory string, dirMode fs.FileMode)

use 0 for default file mode owner rwx

func Exists 

func Exists(path string) (fileInfo fs.FileInfo)

Exists determines if a path exists If the path exists, fileInfo is non-nil if the path does not exist, fileInfo is nil panic on troubles

func Exists2 added in v0.4.127 

func Exists2(path string) (fileInfo fs.FileInfo, isNotExist bool, err error)

Exists2 determines if a path exists

  • fileInfo non-nil: does exist
  • isNotExists true, fileInfo nil, err non-nil: does not exist
  • isNotExist false, fileInfo nil, err non-nil: some error

func GetModeBitValues 

func GetModeBitValues() (s string)

func IsDirectory 

func IsDirectory(path string, flags IsDirectoryArg) (isDirectory bool, err error)

IsDirectory determines if path exists, is a directory or other entry

  • flags: IsDirectoryNonExistentIsError
  • flags: IsDirectoryNotDirIsError

func IsEmptyDirectory 

func IsEmptyDirectory(path string, ignoreENOENT ...bool) (isEmpty bool)

IsEmptyDirectory checks if a directory is empty. ignoreENOENT if true, a non-exiting directory is ignored. pamnic on troubles

func MoveOrMerge 

func MoveOrMerge(src, dest string, outConsumer func(string)) (err error)

func Mv 

func Mv(src, dest string, outConsumer func(string)) (err error)

Mv moves a file or directory structure using system command. Mv uses -n for --n-clobber. Mv does not indicate if moive was aborted due to no-clobber. outConsumer can be nil but receiver command output if any. mv typically has no output.

func NewContextReader 

func NewContextReader(ctx context.Context, reader io.Reader) io.Reader

NewContextReader instantiates ContextReader

func NewIterator added in v0.4.138 

func NewIterator(path string) (iterator iters.Iterator[ResultEntry])
  • rootPath is the initial path for the file-system walk. it may be relative or absolute, contain symlinks and point to a file, directory or special file

func Stat 

func Stat(path string) (fileInfo fs.FileInfo, err error)

func TestElemsX added in v0.4.26 

func TestElemsX(t *testing.T)

func Walk 

func Walk(rootPath string, walkFn filepath.WalkFunc) (err error)

Walk pfs.Walk traverses a directory hierarchy following any symlinks

  • every entry in the hierarchy is provided exactly once to walkFn
  • to identify circular symlinks, the hierarchy is first completely scanned
  • — this builds an in-memory representation of all files and directories
  • — a complete scan is required toresolve nesting among symlinks
  • the Go standard library filepath.Walk does not follow symlinks
  • walkFn receives each entry with paths beginning similar to the path provided to Walk
  • — WalkFn is: func(path string, info fs.FileInfo, err error) error
  • — path may be implicitly relative to current directory: “subdir/file.txt”
  • — may have no directory part: “README.css”
  • — may be relative: “../file.txt”
  • — may contain symlinks, unnecessary “.” and “..”, and other problems
  • — pfs.AbsEval returns an absolute, clean, symlink or non-symlink path
  • — walkFn may receive an error if encountered while scanning a path
  • — if walkFn receives an error, info may be nil walkFn can choose to ignore, skip or return the error
  • walkFn may return filepath.SkipDir to skip entering a certain directory
  • walkFn may return filepath.SkipAll to end file system scanning
  • Walk does not return filepath.SkipDir or filepath.SkipAll errors

Types

type ContextReader 

type ContextReader struct {
	io.Reader
	// contains filtered or unexported fields
}

ContextReader reader terminated by context

func (*ContextReader) Read 

func (cr *ContextReader) Read(p []byte) (n int, err error)

type DLEntry 

type DLEntry struct {
	RelDir      string // directory name that may begin with '.'
	AbsDir      string // absolute directory name
	FqPath      string // fully qualified path to entry
	fs.DirEntry        // .Name() .IsDir() .Type() .Info()
	Info        fs.FileInfo
	Stat        *syscall.Stat_t
}

func GetEntry 

func GetEntry(rel, abs string, entry fs.DirEntry, info fs.FileInfo, stat *syscall.Stat_t) (e *DLEntry)

type DeferringDirEntry added in v0.4.138 

type DeferringDirEntry struct {
	// contains filtered or unexported fields
}

func NewDeferringDirEntry added in v0.4.138 

func NewDeferringDirEntry(abs string) (entry *DeferringDirEntry)

NewDeferringDirEntry returns fs.DirEntry with fs.FileInfo deferred

  • abs is absolute path that maybe does not exist

func (*DeferringDirEntry) Info added in v0.4.138 

func (e *DeferringDirEntry) Info() (fileInfo fs.FileInfo, err error)

Info returns the FileInfo for the file or subdirectory described by the entry

func (*DeferringDirEntry) IsDir added in v0.4.138 

func (e *DeferringDirEntry) IsDir() (isDir bool)

IsDir reports whether the entry is a directory, ie. ModeDir bit being set

  • may panic unless Info has successfully completed

func (*DeferringDirEntry) Name added in v0.4.138 

func (e *DeferringDirEntry) Name() (base string)

base name of the file

func (*DeferringDirEntry) Type added in v0.4.138 

func (e *DeferringDirEntry) Type() (modeType fs.FileMode)

Type returns ModeType bits, 0 for regular file

  • ModeDir ModeSymlink ModeNamedPipe ModeSocket ModeDevice ModeCharDevice ModeIrregular

type Directory 

type Directory struct {
	Entry
	Children []FSEntry
}

Directory is a file system entry with children

func NewDirectory 

func NewDirectory(entryImpl *Entry) (d *Directory)

NewDirectory instantiates FSEntry that can have children

func (*Directory) FetchChildren added in v0.4.108 

func (d *Directory) FetchChildren(path string) (children []FSEntry, paths []string, err error)

FetchChildren reads the directory and populates the Children field

  • errors are stored using SetError

type DirectoryLister 

type DirectoryLister struct {
	Path    string // may begin with ., may be .
	Abs     string
	Results chan *EntryResult
	// contains filtered or unexported fields
}

func NewDirStream 

func NewDirStream(path string, chanSize int) (dir *DirectoryLister)

func (*DirectoryLister) Shutdown 

func (dir *DirectoryLister) Shutdown()

type Entry 

type Entry struct {

	// info.Name() has basename, os.FileInfo: interface
	//	- may be nil
	//	- Name() Size() Mode() ModTime() IsDir() Sys()
	os.FileInfo
	// contains filtered or unexported fields
}

Entry is a file system entry that is not a directory

  • can be part of directory, symlink or other

func NewEntry 

func NewEntry(base string) (entry *Entry)

NewEntry returns Entry or Directory

  • path is absolute path to this file

func (*Entry) FetchFileInfo added in v0.4.108 

func (e *Entry) FetchFileInfo(path string) (err error)
func (e *Entry) IsSymlink() (isSymlink bool)

func (*Entry) Name added in v0.4.108 

func (e *Entry) Name() (base string)

Name gets filepath.Base() in a safe way

  • Name() fails if FileInfo si not available

func (*Entry) SetError added in v0.4.108 

func (e *Entry) SetError(err error)

SetError stores an error encountered during scan for symlinks

  • it is provided to filepath.WalkFunc during the walk phase

func (*Entry) Walks added in v0.4.108 

func (e *Entry) Walks() (info fs.FileInfo, err error)

Walk traverses which for a file is only the file itself

type EntryResult 

type EntryResult struct {
	*DLEntry
	Err error
}

func GetErrorResult 

func GetErrorResult(err error) (result *EntryResult)

type EntryScanner added in v0.4.108 

type EntryScanner struct {
	// contains filtered or unexported fields
}

EntryScanner scans file-system entries and their children

func NewEntryScanner added in v0.4.108 

func NewEntryScanner(
	rootEntry FSEntry, abs, path string,
	symlinkCb func(abs string),
	pFSEntryCount *atomic.Uint64,
) (scanner *EntryScanner)

func (*EntryScanner) Scan added in v0.4.108 

func (s *EntryScanner) Scan() (err error)

Scan scans the file system for this root

  • files and directories that are symlinks are stored via symlinkCb
  • sub-directories have deferred processing using a linked list

type FSEntry 

type FSEntry interface {
	// Name is basename from fs.DirEntry “file.txt”
	Name() (base string)
	// IsDir returns whether the file-system entry is a directory containing files
	// as opposed to a file
	IsDir() (isDir bool)
	// IsSymlink returns whether the file-system entry is a symbolic link
	// that may create a new root
	//	- a symlink is not a directory, but may point to a directory that
	//		may become a new root
	IsSymlink() (isSymlink bool)
	// Walks returns data required for invoking filepath.WalkFunc
	Walks() (info fs.FileInfo, err error)
	// SetError stores an error encountered during scan for symlinks
	//	- it is provided to filepath.WalkFunc during the walk phase
	SetError(err error)
}

FSEntry represents a branch in a file system hierarchy. May be:

  • an Entry, ie. a file and not a directory
  • a Directory containing additional file-system entries
  • a Root representing a file-system traversal entry point

func GetFSEntry added in v0.4.108 

func GetFSEntry(path string, base string) (entry FSEntry, err error)

GetFSEntry returns an FSentry corresponding to path

  • the returned FSEntry may be a directory or a file-type entry
  • path is provided path, possibly relative
  • base is basename in case fileInfo cannot be retrieved
  • sym receives a callback if the FSEntry is a symbolic link

type FileMode 

type FileMode struct {
	fs.FileMode
}

func (FileMode) Ch 

func (fm FileMode) Ch() (s string)

func (FileMode) Check 

func (fm FileMode) Check() (str string)

Check returns hex string of unknown bits, empty string if no unknown bits

func (FileMode) Hex 

func (fm FileMode) Hex() (s string)

func (FileMode) Hex8 

func (fm FileMode) Hex8() (s string)

func (FileMode) Or 

func (fm FileMode) Or() (s string)

func (FileMode) String 

func (fm FileMode) String() (s string)

type IsDirectoryArg 

type IsDirectoryArg byte
const (
	IsDirectoryNonExistentIsError IsDirectoryArg = 1 << iota
	IsDirectoryNotDirIsError
)

type Iterator added in v0.4.138 

type Iterator struct {
	iters.BaseIterator[ResultEntry]
	// contains filtered or unexported fields
}

func (*Iterator) Init added in v0.4.138 

func (i *Iterator) Init() (result ResultEntry, iterator iters.Iterator[ResultEntry])

type PathRegistry added in v0.4.108 

type PathRegistry[V any] struct {
	// contains filtered or unexported fields
}

PathRegistry stores values that are accessible by index or by absolute path

func NewPathRegistry added in v0.4.108 

func NewPathRegistry[V any]() (registry *PathRegistry[V])

NewPathRegistry returns a registry of values by absolute path

func (*PathRegistry[V]) Add added in v0.4.108 

func (r *PathRegistry[V]) Add(abs string, value *V)

Add adds a value to the registry

func (*PathRegistry[V]) DeleteByIndex added in v0.4.108 

func (r *PathRegistry[V]) DeleteByIndex(index int)

func (*PathRegistry[V]) Drop added in v0.4.108 

func (r *PathRegistry[V]) Drop(onlyPaths ...bool)

func (*PathRegistry[V]) GetNext added in v0.4.108 

func (r *PathRegistry[V]) GetNext() (value *V)

GetNext gets the next value and removes it from the registry

func (*PathRegistry[V]) GetValue added in v0.4.108 

func (r *PathRegistry[V]) GetValue(index int) (value *V)

GetValue retrieves value by index

  • if index is less than 0 or too large or for a removed value, nil is returned

func (*PathRegistry[V]) HasAbs added in v0.4.108 

func (r *PathRegistry[V]) HasAbs(abs string) (hasAbs bool)

HasAbs check whether an absolute path is stored in the registry as a key to a value

func (*PathRegistry[V]) ListLength added in v0.4.108 

func (r *PathRegistry[V]) ListLength() (length int)

ListLength returns the length of the value slice

  • a value can still be nil for a discarded root

func (*PathRegistry[V]) MapLength added in v0.4.108 

func (r *PathRegistry[V]) MapLength() (length int)

MapLength returns the number of stored values

func (*PathRegistry[V]) ObsoleteIndex added in v0.4.138 

func (r *PathRegistry[V]) ObsoleteIndex(index int)

type PendingEntry added in v0.4.108 

type PendingEntry struct {
	Next      *PendingEntry
	Abs, Path string
	Entry     FSEntry
}

func NewPendingEntry added in v0.4.108 

func NewPendingEntry(abs, path string, entry FSEntry) (pending *PendingEntry)

type Registry added in v0.4.138 

type Registry[V any] struct {
	// contains filtered or unexported fields
}

Registry stores values that are accessible by index or by absolute path

func NewRegistry added in v0.4.138 

func NewRegistry[V any]() (registry *Registry[V])

NewRegistry returns a registry of values by absolute path

func (*Registry[V]) Add added in v0.4.138 

func (r *Registry[V]) Add(abs string, value *V)

Add adds a value to the registry

func (*Registry[V]) GetValue added in v0.4.138 

func (r *Registry[V]) GetValue(index int) (value *V)

GetValue retrieves value by index

  • if index is less than 0 or too large or for a removed value, nil is returned

func (*Registry[V]) HasAbs added in v0.4.138 

func (r *Registry[V]) HasAbs(abs string) (hasAbs bool)

HasAbs check whether an absolute path is stored in the registry as a key to a value

func (*Registry[V]) ListLength added in v0.4.138 

func (r *Registry[V]) ListLength() (length int)

ListLength returns the length of the value slice

  • a value can still be nil for a discarded root

func (*Registry[V]) ObsoleteIndex added in v0.4.138 

func (r *Registry[V]) ObsoleteIndex(index int)

type ResultEntry added in v0.4.138 

type ResultEntry struct {
	//	- always non-nil
	//	- may be deferred-info executing [os.Lstat] every time, ie. only invoke Info once
	//	- if Err is non-nil, Info may return error
	fs.DirEntry
	// path as provided that may be easier to read
	//   - may be implicitly relative to current directory: “subdir/file.txt”
	//   - may have no directory or extension part: “README.css” “z”
	//   - may be relative: “../file.txt”
	//   - may contain symlinks, unnecessary “.” and “..” or
	//		multiple separators in sequence
	//	- may be empty string for current working directory
	ProvidedPath string
	//	- equivalent of Path: absolute, symlink-free, clean
	//	- if Err non-nil, may be empty
	//	- if the entry is symbolic link:
	//	- — ProvidedPath is the symbolic link location
	//	- — Abs is what the symbolic link points to
	Abs string
	// function to skip descending into directory or following symlink
	SkipEntry func(no uint64)
	// skippable serial number
	No uint64
	// any error associated with this entry
	Err error
	// why this entry was provided
	Reason ResultReason
}

ResultEntry is an existing file-system entry, traversal-end marker or an error-entry during file-system traversal

  • non-error non-end file-system entries have been proven to exist
  • ResultEntry.IsEnd or Reason == REnd indicates end
  • Reson indicates why the ResultEntry was returned:
  • REnd REntry RSkippable RDirBad RSymlinkBad RError
  • —
  • ResultEntry is a value-container that as a local variable, function argument or result is a tuple not causing allocation by using temporary stack storage
  • taking the address of a &ResultEntry causes allocation

func (ResultEntry) IsEnd added in v0.4.138 

func (e ResultEntry) IsEnd() (isEnd bool)

IsEnd indicates that this ResultEntry is an end of entries marker

func (ResultEntry) Skip added in v0.4.138 

func (e ResultEntry) Skip()

Skip marks the returned entry to be skipped

  • the entry is directory or symbolic link
  • can only be invoked when [ResultEntry.Reason] is RSkippable

type ResultReason added in v0.4.138 

type ResultReason uint8

Why a directory entry was provided by Traverser

  • REnd REntry RSkippable RDirBad RSymlinkBad RError
const (
	// the filesystem traversal completed all entries
	REnd ResultReason = iota
	// a non-symlink non-directory non-error entry
	REntry
	// a directory or symlink about to be traversed
	RSkippable
	// a directory whose listing failed [os.Open] [os.File.ReadDir]
	RDirBad
	// a broken symlink [os.Stat]
	RSymlinkBad
	// failure in [os.Lstat] [os.Readlink] [os.Getwd]
	RError
)

func (ResultReason) String added in v0.4.138 

func (r ResultReason) String() (s string)

type Root 

type Root struct {
	// path as provided that may be easier to read
	//   - may be implicitly relative to current directory: “subdir/file.txt”
	//   - may have no directory or extension part: “README.css” “z”
	//   - may be relative: “../file.txt”
	//   - may contain symlinks, unnecessary “.” and “..” or
	//		multiple separators in sequence
	//	- may be empty string for current working directory
	ProvidedPath string

	// FSEntry represents the file-system location of this root
	//	- Directory or Entry
	//	- SafeName() IsDir()
	FSEntry
	// contains filtered or unexported fields
}

Root is a file system hierarchy

func NewRoot 

func NewRoot(path string) (root *Root)

NewRoot returns a file-system starting point for traversal

  • path may be absolute or relative path

func (*Root) Abs added in v0.4.108 

func (r *Root) Abs() (abs string)

func (*Root) Init added in v0.4.108 

func (r *Root) Init() (absPath string, rootEntry FSEntry, err error)

type Root2 added in v0.4.138 

type Root2 struct {
	// path as provided that may be easier to read
	//   - may be implicitly relative to current directory: “subdir/file.txt”
	//   - may have no directory or extension part: “README.css” “z”
	//   - may be relative: “../file.txt”
	//   - may contain symlinks, unnecessary “.” and “..” or
	//		multiple separators in sequence
	//	- may be empty string for current working directory
	ProvidedPath string
	// Abs is required for relating roots
	//	- equivalent of Path: absolute, symlink-free, clean
	//	- may not exist
	Abs string
}

Root2 is a file system hierarchy

func NewAbsRoot2 added in v0.4.138 

func NewAbsRoot2(abs string) (root2 *Root2)

func NewRoot2 added in v0.4.138 

func NewRoot2(path string) (root2 *Root2)

NewRoot returns a file-system starting point for traversal

  • path is as-provided path that may be:
  • — absolute or relative
  • — unclean
  • — contain symlinks

func (*Root2) Load added in v0.4.138 

func (r *Root2) Load() (err error)

Load obtain the absolute, symlink-resolved path for the root

type Sha256 

type Sha256 []byte

Sha256 contains sha-256 hash

func Sha256Context 

func Sha256Context(ctx context.Context, filename string) (s2 Sha256, err error)

Sha256Context get sha256 of a file with context

func (Sha256) String 

func (s2 Sha256) String() string

func (*Sha256) Valid 

func (s2 *Sha256) Valid() bool

Valid determines if hash is present 

type SymlinkList struct {
	// contains filtered or unexported fields
}
func NewSymlinkList() (list *SymlinkList)

type Traverser added in v0.4.138 

type Traverser struct {
	// contains filtered or unexported fields
}

Traverser represents a file system that is scanned following symlinks

  • each file system entry is returned exactly once except:
  • — error reading a directory returns the directory a second time
  • directories and symlinks are returned before they are read so that they can be more efficiently skipped by invoking ResultEntry.Skip
  • directory entries are returned in 8-bit character order
  • returned entries may not exist, such entries have [ResultEntry.Err] non-nil
  • result.ProvidedPath is based on the initial path and may be relative, unclean and contain symlinks
  • if [ResultEntry.Err] is nil, Abs is absolute, symlink-free clean path
  • —
  • ResultEntry.DirEntry.Info typically invokes os.Lstat every time, so this value should be cached
  • because symlinks may point to parents or separate trees, the file system scan may involve multiple roots which may affect the order of return entries
  • symlinks are followed and not returned. Therefore, a symlink pointing to a scanned location is effectively ignored
  • the returned struct is by value. If its address is not taken, no allocation will occur

func NewTraverser added in v0.4.138 

func NewTraverser(path string) (traverser *Traverser)

NewTraverser returns a file-system traverser

  • path is the initial path. Path may be relative or absolute, contain symlinks and be unclean. Path may be of any modeType: file, directory or special file. Empty string means process’ current directory
  • the Next method is used to obtain file-system entries and errors

Usage: 

var traverser = pfs.NewTraverser(path)
for {
  var result = traverser.Next()
  if result.IsEnd() || result.Err != nil {
    break
  }
  println(result.Abs)
}

func (*Traverser) Next added in v0.4.138 

func (t *Traverser) Next() (result ResultEntry)

Next returns the next file-system entry

  • Next ends with [ResultEntry.DirEntry] nil, ie. ResultEntry.IsEnd returns true or ResultEntry.Reason == REnd
  • symlinks and directories can be skipped by invoking ResultEntry.Skip. Those have ResultEntry.Reason == RSkippable
  • symlinks have information based on the symlink source but [ResultEntry.Abs] is the fully resolved symlink target
  • [ResultEntry.ProvidedPath] is a path to the entry based upon the initially provided path. May be empty string
  • [ResultEntry.Abs] is an absolute symlink-free clean path but only available when [ResultEntry.Err] is nil
  • [ResultEntry.Err] holds any error associated with the returned entry
  • —
  • result.Err is from:
  • — process working directory cannot be read
  • — directory read error or os.Readlink or os.Lstat failed

type Tree 

type Tree struct {
	FSEntryCount *atomic.Uint64
	// contains filtered or unexported fields
}

Tree represents a file system scan originating at a single absolute or relative path

  • additional roots may appear if the original path contains symlinks that point outside the original directory tree
  • each such root is a separate starting path in the overall file system

func NewTree 

func NewTree(walkFunc filepath.WalkFunc) (tree *Tree)

NewTree returns a file-system scan object

func (*Tree) ScanRoots added in v0.4.108 

func (t *Tree) ScanRoots(rootPath string) (err error)

ScanRoots scans rootpath and all additional encountered roots

  • rootPath is as provided to the Walk function, ie. may be relative and unclean

func (*Tree) SetCounter added in v0.4.109 

func (t *Tree) SetCounter(FSEntryCount *atomic.Uint64)

func (*Tree) Walk 

func (t *Tree) Walk() (err error)

Walk traverses the built tree

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.