local

package
v1.30.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 7, 2024 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefaultDirectoryPermissions Octal representation of default folder permissions
	DefaultDirectoryPermissions = 0o040777
	ClientMtimeMetadataKey      = apiutil.LakeFSMetadataPrefix + "client-mtime"
)
View Source 
const (
	IndexFileName = ".lakefs_ref.yaml"
	IgnoreMarker  = "ignored by lakectl local"
	IndexFileMode = 0o644
)
View Source 
const (
	// DefaultFilePermissions Octal representation of default file permissions
	DefaultFilePermissions      = 0o100666
	POSIXPermissionsMetadataKey = apiutil.LakeFSMetadataPrefix + "posix-permissions"
)

Variables

View Source 
var (
	ErrConflict        = errors.New("conflict")
	ErrDownloadingFile = errors.New("error downloading file")
	ErrRemoteFailure   = errors.New("remote failure")
	ErrNotFound        = errors.New("not found")
)
View Source 
var (
	ErrUnsupportedFS = errors.New("unsupported filesystem")
)

Functions

func ChangeSourceString added in v0.106.0 

func ChangeSourceString(changeSource ChangeSource) string

func ChangeTypeString added in v0.106.0 

func ChangeTypeString(changeType ChangeType) string

func FindIndices 

func FindIndices(root string) ([]string, error)

FindIndices searches the specified root directory for index files, returning their relative directory paths while skipping hidden folders.

func IndexExists 

func IndexExists(baseAbs string) (bool, error)

func ListRemote added in v0.106.0 

func ListRemote(ctx context.Context, client apigen.ClientWithResponsesInterface, loc *uri.URI, objects chan<- apigen.ObjectStats, includeDirs bool) error

ListRemote - Lists objects from a remote uri and inserts them into the objects channel

func WalkS3 added in v1.26.0 

func WalkS3(root string, callbackFunc func(p string, info fs.FileInfo, err error) error) error

WalkS3 - walk like an Egyptian... ¯\_(ツ)¯\_ This walker function simulates the way object listing is performed by S3. Contrary to how a standard FS walk function behaves, S3 does not take into consideration the directory hierarchy. Instead, object paths include the entire path relative to the root and as a result the directory or "path separator" is also taken into account when providing the listing in a lexicographical order.

Types

type Change added in v0.106.0 

type Change struct {
	Source ChangeSource
	Path   string
	Type   ChangeType
}

func (*Change) String added in v0.106.0 

func (c *Change) String() string

type ChangeSource added in v0.106.0 

type ChangeSource int
const (
	ChangeSourceRemote ChangeSource = iota
	ChangeSourceLocal
)

type ChangeType added in v0.106.0 

type ChangeType int
const (
	ChangeTypeAdded ChangeType = iota
	ChangeTypeModified
	ChangeTypeRemoved
	ChangeTypeConflict
)

func ChangeTypeFromString added in v0.106.0 

func ChangeTypeFromString(changeType string) ChangeType

type Changes added in v0.106.0 

type Changes []*Change

func DiffLocalWithHead added in v0.106.0 

func DiffLocalWithHead(left <-chan apigen.ObjectStats, rightPath string, cfg Config) (Changes, error)

DiffLocalWithHead Checks changes between a local directory and the head it is pointing to. The diff check assumes the remote is an immutable set so any changes found resulted from changes in the local directory left is an object channel which contains results from a remote source. rightPath is the local directory to diff with

func Undo added in v0.106.0 

func Undo(c Changes) Changes

Undo Creates a new list of changes that reverses the given changes list.

func (Changes) MergeWith added in v0.106.0 

func (c Changes) MergeWith(other Changes, strategy MergeStrategy) Changes

MergeWith combines changes from two diffs, sorting by lexicographic order. If the same path appears in both diffs, it's marked as a conflict.

func (Changes) String added in v0.106.0 

func (c Changes) String() string

type Config added in v1.30.1 

type Config struct {
	SyncFlags
	// SkipNonRegularFiles - By default lakectl local fails if local directory contains irregular files. When set, lakectl will skip these files instead.
	SkipNonRegularFiles bool
	// IncludePerm - Experimental: preserve Unix file permissions
	IncludePerm bool
}

type Index 

type Index struct {
	PathURI         string `yaml:"src"`
	AtHead          string `yaml:"at_head"`
	ActiveOperation string `yaml:"active_operation"`
	// contains filtered or unexported fields
}

Index defines the structure of the lakefs local reference file consisting of the information linking local directory with lakefs path

func ReadIndex 

func ReadIndex(path string) (*Index, error)

func WriteIndex 

func WriteIndex(path string, remote *uri.URI, atHead string, operation string) (*Index, error)

func (*Index) GetCurrentURI 

func (l *Index) GetCurrentURI() (*uri.URI, error)

func (*Index) LocalPath 

func (l *Index) LocalPath() string

type MergeStrategy added in v0.106.0 

type MergeStrategy int
const (
	MergeStrategyNone MergeStrategy = iota
	MergeStrategyThis
	MergeStrategyOther
)

type POSIXOwnership added in v1.28.1 

type POSIXOwnership struct {
	UID int
	GID int
}

type POSIXPermissions added in v1.28.1 

type POSIXPermissions struct {
	POSIXOwnership
	Mode os.FileMode
}

func GetDefaultPermissions added in v1.28.1 

func GetDefaultPermissions(isDir bool) POSIXPermissions

GetDefaultPermissions - returns default permissions as defined by file system. Public for testing purposes

type ProgressPool added in v0.106.0 

type ProgressPool struct {
	// contains filtered or unexported fields
}

func NewProgressPool added in v0.106.0 

func NewProgressPool() *ProgressPool

func (*ProgressPool) AddReader added in v0.106.0 

func (p *ProgressPool) AddReader(name string, sizeBytes int64) *ProgressUpdater

func (*ProgressPool) AddSpinner added in v0.106.0 

func (p *ProgressPool) AddSpinner(name string) *ProgressSpinner

func (*ProgressPool) Start added in v0.106.0 

func (p *ProgressPool) Start()

func (*ProgressPool) Stop added in v0.106.0 

func (p *ProgressPool) Stop()

type ProgressSpinner added in v0.106.0 

type ProgressSpinner struct {
	// contains filtered or unexported fields
}

func (*ProgressSpinner) Done added in v0.106.0 

func (p *ProgressSpinner) Done()

func (*ProgressSpinner) Error added in v0.106.0 

func (p *ProgressSpinner) Error()

type ProgressUpdater added in v0.106.0 

type ProgressUpdater struct {
	// contains filtered or unexported fields
}

func (*ProgressUpdater) Done added in v0.106.0 

func (p *ProgressUpdater) Done()

func (*ProgressUpdater) Error added in v0.106.0 

func (p *ProgressUpdater) Error()

func (*ProgressUpdater) Reader added in v0.106.0 

func (p *ProgressUpdater) Reader(reader io.Reader) io.Reader

type ProgressUpdaterReader added in v0.106.0 

type ProgressUpdaterReader struct {
	// contains filtered or unexported fields
}

func (*ProgressUpdaterReader) Read added in v0.106.0 

func (pu *ProgressUpdaterReader) Read(p []byte) (n int, err error)

type StringHeap added in v1.26.0 

type StringHeap []string

A StringHeap is a min-heap of strings

func (*StringHeap) Len added in v1.26.0 

func (pq *StringHeap) Len() int

func (*StringHeap) Less added in v1.26.0 

func (pq *StringHeap) Less(i, j int) bool

func (*StringHeap) Peek added in v1.26.0 

func (pq *StringHeap) Peek() any

Peek - according to heap documentation (https://pkg.go.dev/container/heap) "The minimum element in the tree is the root, at index 0."

func (*StringHeap) Pop added in v1.26.0 

func (pq *StringHeap) Pop() any

func (*StringHeap) Push added in v1.26.0 

func (pq *StringHeap) Push(x any)

func (*StringHeap) Swap added in v1.26.0 

func (pq *StringHeap) Swap(i, j int)

type SyncFlags added in v0.113.0 

type SyncFlags struct {
	Parallelism      int
	Presign          bool
	PresignMultipart bool
}

type SyncManager added in v0.106.0 

type SyncManager struct {
	// contains filtered or unexported fields
}

func NewSyncManager added in v0.106.0 

func NewSyncManager(ctx context.Context, client *apigen.ClientWithResponses, httpClient *http.Client, cfg Config) *SyncManager

func (*SyncManager) Summary added in v0.106.0 

func (s *SyncManager) Summary() Tasks

func (*SyncManager) Sync added in v0.106.0 

func (s *SyncManager) Sync(rootPath string, remote *uri.URI, changeSet <-chan *Change) error

Sync - sync changes between remote and local directory given the Changes channel. For each change, will apply download, upload or delete according to the change type and change source

type Tasks added in v0.106.0 

type Tasks struct {
	Downloaded uint64
	Uploaded   uint64
	Removed    uint64
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.