file

package
v0.24.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 10, 2024 License: AGPL-3.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrZipFileNotExist = errors.New("zip file does not exist")

Functions

func Destroy added in v0.17.0 

func Destroy(ctx context.Context, destroyer models.FileDestroyer, f models.File, fileDeleter *Deleter, deleteFile bool) error

func GetOrCreateFolderHierarchy added in v0.20.0 

func GetOrCreateFolderHierarchy(ctx context.Context, fc models.FolderFinderCreator, path string) (*models.Folder, error)

GetOrCreateFolderHierarchy gets the folder for the given path, or creates a folder hierarchy for the given path if one if no existing folder is found. Does not create any folders in the file system

func TransferZipFolderHierarchy added in v0.21.0 

func TransferZipFolderHierarchy(ctx context.Context, folderStore models.FolderReaderWriter, zipFileID models.FileID, oldPath string, newPath string) error

TransferZipFolderHierarchy creates the folder hierarchy for zipFileID under newPath, and removes ZipFileID from folders under oldPath.

Types

type CleanHandler added in v0.17.0 

type CleanHandler interface {
	HandleFile(ctx context.Context, fileDeleter *Deleter, fileID models.FileID) error
	HandleFolder(ctx context.Context, fileDeleter *Deleter, folderID models.FolderID) error
}

CleanHandler provides a handler for cleaning Files and Folders.

type CleanOptions added in v0.17.0 

type CleanOptions struct {
	Paths []string

	// Do a dry run. Don't delete any files
	DryRun bool

	// PathFilter are used to determine if a file should be included.
	// Excluded files are marked for cleaning.
	PathFilter PathFilter
}

ScanOptions provides options for scanning files.

type Cleaner added in v0.17.0 

type Cleaner struct {
	FS         models.FS
	Repository Repository

	Handlers []CleanHandler
}

Cleaner scans through stored file and folder instances and removes those that are no longer present on disk.

func (*Cleaner) Clean added in v0.17.0 

func (s *Cleaner) Clean(ctx context.Context, options CleanOptions, progress *job.Progress)

Clean starts the clean process.

type Decorator added in v0.17.0 

type Decorator interface {
	Decorate(ctx context.Context, fs models.FS, f models.File) (models.File, error)
	IsMissingMetadata(ctx context.Context, fs models.FS, f models.File) bool
}

Decorator wraps the Decorate method to add additional functionality while scanning files.

type Deleter added in v0.12.0 

type Deleter struct {
	RenamerRemover RenamerRemover
	// contains filtered or unexported fields
}

Deleter is used to safely delete files and directories from the filesystem. During a transaction, files and directories are marked for deletion using the Files and Dirs methods. This will rename the files/directories to be deleted. If the transaction is rolled back, then the files/directories can be restored to their original state with the Abort method. If the transaction is committed, the marked files are then deleted from the filesystem using the Complete method.

func NewDeleter added in v0.12.0 

func NewDeleter() *Deleter

func (*Deleter) Commit added in v0.12.0 

func (d *Deleter) Commit()

Commit deletes all files marked for deletion and clears the marked list. Any errors encountered are logged. All files will be attempted, regardless of the errors encountered.

func (*Deleter) Dirs added in v0.12.0 

func (d *Deleter) Dirs(paths []string) error

Dirs designates directories to be deleted. Each directory marked will be renamed to add a `.delete` suffix. An error is returned if a directory could not be renamed. Note that if an error is returned, then some directories may be left renamed. Abort should be called to restore marked files/directories if this function returns an error.

func (*Deleter) Files added in v0.12.0 

func (d *Deleter) Files(paths []string) error

Files designates files to be deleted. Each file marked will be renamed to add a `.delete` suffix. An error is returned if a file could not be renamed. Note that if an error is returned, then some files may be left renamed. Abort should be called to restore marked files if this function returns an error.

func (*Deleter) RegisterHooks added in v0.17.0 

func (d *Deleter) RegisterHooks(ctx context.Context)

RegisterHooks registers post-commit and post-rollback hooks.

func (*Deleter) Rollback added in v0.12.0 

func (d *Deleter) Rollback()

Rollback tries to rename all marked files and directories back to their original names and clears the marked list. Any errors encountered are logged. All files will be attempted regardless of any errors occurred.

type DirMakerStatRenamer added in v0.20.0 

type DirMakerStatRenamer interface {
	Statter
	Renamer
	Mkdir(name string, perm os.FileMode) error
	Remove(name string) error
}

type Filter added in v0.17.0 

type Filter interface {
	Accept(ctx context.Context, f models.File) bool
}

Filter provides a filter function for Files.

type FilterFunc added in v0.17.0 

type FilterFunc func(ctx context.Context, f models.File) bool

func (FilterFunc) Accept added in v0.17.0 

func (ff FilterFunc) Accept(ctx context.Context, f models.File) bool

type FilteredDecorator added in v0.17.0 

type FilteredDecorator struct {
	Decorator
	Filter
}

func (*FilteredDecorator) Decorate added in v0.17.0 

func (d *FilteredDecorator) Decorate(ctx context.Context, fs models.FS, f models.File) (models.File, error)

Decorate runs the decorator if the filter accepts the file.

func (*FilteredDecorator) IsMissingMetadata added in v0.17.0 

func (d *FilteredDecorator) IsMissingMetadata(ctx context.Context, fs models.FS, f models.File) bool

type FilteredHandler added in v0.17.0 

type FilteredHandler struct {
	Handler
	Filter
}

FilteredHandler is a Handler runs only if the filter accepts the file.

func (*FilteredHandler) Handle added in v0.17.0 

func (h *FilteredHandler) Handle(ctx context.Context, f models.File, oldFile models.File) error

Handle runs the handler if the filter accepts the file.

type FingerprintCalculator added in v0.17.0 

type FingerprintCalculator interface {
	CalculateFingerprints(f *models.BaseFile, o Opener, useExisting bool) ([]models.Fingerprint, error)
}

FingerprintCalculator calculates a fingerprint for the provided file.

type Handler added in v0.17.0 

type Handler interface {
	Handle(ctx context.Context, f models.File, oldFile models.File) error
}

Handler provides a handler for Files.

type Importer added in v0.23.0 

type Importer struct {
	ReaderWriter models.FileFinderCreator
	FolderStore  models.FolderFinderCreator
	Input        jsonschema.DirEntry
	// contains filtered or unexported fields
}

func (*Importer) Create added in v0.23.0 

func (i *Importer) Create(ctx context.Context) (*int, error)

func (*Importer) FindExistingID added in v0.23.0 

func (i *Importer) FindExistingID(ctx context.Context) (*int, error)

func (*Importer) Name added in v0.23.0 

func (i *Importer) Name() string

func (*Importer) PostImport added in v0.23.0 

func (i *Importer) PostImport(ctx context.Context, id int) error

func (*Importer) PreImport added in v0.23.0 

func (i *Importer) PreImport(ctx context.Context) error

func (*Importer) Update added in v0.23.0 

func (i *Importer) Update(ctx context.Context, id int) error

type Mover added in v0.20.0 

type Mover struct {
	Renamer DirMakerStatRenamer
	Files   models.FileFinderUpdater
	Folders models.FolderReaderWriter
	// contains filtered or unexported fields
}

func NewMover added in v0.20.0 

func NewMover(fileStore models.FileFinderUpdater, folderStore models.FolderReaderWriter) *Mover

func (*Mover) CreateFolderHierarchy added in v0.20.0 

func (m *Mover) CreateFolderHierarchy(path string) error

func (*Mover) Move added in v0.20.0 

func (m *Mover) Move(ctx context.Context, f models.File, folder *models.Folder, basename string) error

Move moves the file to the given folder and basename. If basename is empty, then the existing basename is used. Assumes that the parent folder exists in the filesystem.

func (*Mover) RegisterHooks added in v0.20.0 

func (m *Mover) RegisterHooks(ctx context.Context)

type Opener added in v0.17.0 

type Opener interface {
	Open() (io.ReadCloser, error)
}

Opener provides an interface to open a file.

type OsFS added in v0.17.0 

type OsFS struct{}

OsFS is a file system backed by the OS.

func (*OsFS) Create added in v0.20.0 

func (f *OsFS) Create(name string) (*os.File, error)

func (*OsFS) IsPathCaseSensitive added in v0.17.2 

func (f *OsFS) IsPathCaseSensitive(path string) (bool, error)

func (*OsFS) Lstat added in v0.17.0 

func (f *OsFS) Lstat(name string) (fs.FileInfo, error)

func (*OsFS) MkdirAll added in v0.20.0 

func (f *OsFS) MkdirAll(path string, perm fs.FileMode) error

func (*OsFS) Open added in v0.17.0 

func (f *OsFS) Open(name string) (fs.ReadDirFile, error)

func (*OsFS) OpenZip added in v0.17.0 

func (f *OsFS) OpenZip(name string) (models.ZipFS, error)

func (*OsFS) Remove added in v0.20.0 

func (f *OsFS) Remove(name string) error

func (*OsFS) RemoveAll added in v0.20.0 

func (f *OsFS) RemoveAll(path string) error

func (*OsFS) Rename added in v0.20.0 

func (f *OsFS) Rename(oldpath, newpath string) error

func (*OsFS) Stat added in v0.17.2 

func (f *OsFS) Stat(name string) (fs.FileInfo, error)

type PathFilter added in v0.17.0 

type PathFilter interface {
	Accept(ctx context.Context, path string, info fs.FileInfo) bool
}

PathFilter provides a filter function for paths.

type PathFilterFunc added in v0.17.0 

type PathFilterFunc func(path string) bool

func (PathFilterFunc) Accept added in v0.17.0 

func (pff PathFilterFunc) Accept(path string) bool

type ProgressReporter added in v0.17.0 

type ProgressReporter interface {
	AddTotal(total int)
	Increment()
	Definite()
	ExecuteTask(description string, fn func())
}

ProgressReporter is used to report progress of the scan.

type Renamer added in v0.20.0 

type Renamer interface {
	Rename(oldpath, newpath string) error
}

type RenamerRemover added in v0.12.0 

type RenamerRemover interface {
	Renamer
	Remove(name string) error
	RemoveAll(path string) error
	Statter
}

RenamerRemover provides access to the Rename and Remove functions.

type Repository added in v0.17.0 

type Repository struct {
	TxnManager models.TxnManager

	File   models.FileReaderWriter
	Folder models.FolderReaderWriter
}

Repository provides access to storage methods for files and folders.

func NewRepository added in v0.24.0 

func NewRepository(repo models.Repository) Repository

func (*Repository) WithDB added in v0.24.0 

func (r *Repository) WithDB(ctx context.Context, fn txn.TxnFunc) error

func (*Repository) WithReadTxn added in v0.24.0 

func (r *Repository) WithReadTxn(ctx context.Context, fn txn.TxnFunc) error

func (*Repository) WithTxn added in v0.24.0 

func (r *Repository) WithTxn(ctx context.Context, fn txn.TxnFunc) error

type ScanOptions added in v0.17.0 

type ScanOptions struct {
	Paths []string

	// ZipFileExtensions is a list of file extensions that are considered zip files.
	// Extension does not include the . character.
	ZipFileExtensions []string

	// ScanFilters are used to determine if a file should be scanned.
	ScanFilters []PathFilter

	// HandlerRequiredFilters are used to determine if an unchanged file needs to be handled
	HandlerRequiredFilters []Filter

	ParallelTasks int
}

ScanOptions provides options for scanning files.

type Scanner 

type Scanner struct {
	FS                    models.FS
	Repository            Repository
	FingerprintCalculator FingerprintCalculator

	// FileDecorators are applied to files as they are scanned.
	FileDecorators []Decorator
}

Scanner scans files into the database.

The scan process works using two goroutines. The first walks through the provided paths in the filesystem. It runs each directory entry through the provided ScanFilters. If none of the filter Accept methods return true, then the file/directory is ignored. Any folders found are handled immediately. Files inside zip files are also handled immediately. All other files encountered are sent to the second goroutine queue.

Folders are handled by checking if the folder exists in the database, by its full path. If a folder entry already exists, then its mod time is updated (if applicable). If the folder does not exist in the database, then a new folder entry its created.

Files are handled by first querying for the file by its path. If the file entry exists in the database, then the mod time is compared to the value in the database. If the mod time is different then file is marked as updated - it recalculates any fingerprints and fires decorators, then the file entry is updated and any applicable handlers are fired.

If the file entry does not exist in the database, then fingerprints are calculated for the file. It then determines if the file is a rename of an existing file by querying for file entries with the same fingerprint. If any are found, it checks each to see if any are missing in the file system. If one is, then the file is treated as renamed and its path is updated. If none are missing, or many are, then the file is treated as a new file.

If the file is not a renamed file, then the decorators are fired and the file is created, then the applicable handlers are fired.

func (*Scanner) Scan added in v0.17.0 

func (s *Scanner) Scan(ctx context.Context, handlers []Handler, options ScanOptions, progressReporter ProgressReporter)

Scan starts the scanning process.

type Statter added in v0.20.0 

type Statter interface {
	Stat(name string) (fs.FileInfo, error)
}

type ZipDestroyer added in v0.17.0 

type ZipDestroyer struct {
	FileDestroyer   models.FileFinderDestroyer
	FolderDestroyer models.FolderFinderDestroyer
}

func (*ZipDestroyer) DestroyZip added in v0.17.0 

func (d *ZipDestroyer) DestroyZip(ctx context.Context, f models.File, fileDeleter *Deleter, deleteFile bool) error

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.