repository

package
v0.0.0-...-83b708d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 3, 2024 License: Apache-2.0 Imports: 35 Imported by: 0

Documentation

Index

Constants

View Source 
const SpecPath = "repository.json"
View Source 
const Version = 1

Variables

This section is empty.

Functions

func CopyToTempDir 

func CopyToTempDir(localPath string, includePath string) (tempDir string, err error)

func CreateS3Bucket 

func CreateS3Bucket(region, bucket string) (err error)

func DeleteS3Bucket 

func DeleteS3Bucket(region, bucket string) (err error)

func NeedsCaching 

func NeedsCaching(repositoryURL string) (bool, error)

NeedsCaching returns true if the repository URL is slow and needs caching

func Sync 

func Sync(sourceRepository Repository, sourcePath string, destRepository Repository, destPath string) error

Sync destRepository/destPath to match sourceRepository/sourcePath

- If file exists in source, but not in dest, it will copy from source to dest - If file exists in both but different content, it will copy from source to dest - If file exists in dest but not in source, it will delete in dest

func WriteSpec 

func WriteSpec(r Repository) error

Types

type CachedRepository 

type CachedRepository struct {
	// contains filtered or unexported fields
}

CachedRepository wraps another repository, caching a prefix in a local directory.

SyncCache() syncs cachePrefix locally, which you must call before doing any reads. It is not done automatically so you can control output to the user about syncing.

If a read hits a path starting with cachePrefix, it will use the local cached version.

func NewCachedMetadataRepository 

func NewCachedMetadataRepository(projectDir string, repo Repository) (*CachedRepository, error)

NewCachedMetadataRepository returns a CachedRepository that caches the metadata/ path in .keepsake/metadata-cache in a source dir

func NewCachedRepository 

func NewCachedRepository(repo Repository, cachePrefix string, projectDir string, cacheDir string) (*CachedRepository, error)

func (*CachedRepository) Delete 

func (s *CachedRepository) Delete(p string) error

func (*CachedRepository) Get 

func (s *CachedRepository) Get(p string) ([]byte, error)

func (*CachedRepository) GetPath 

func (s *CachedRepository) GetPath(repoPath string, localPath string) error

func (*CachedRepository) GetPathItemTar 

func (s *CachedRepository) GetPathItemTar(tarPath, itemPath, localPath string) error

func (*CachedRepository) GetPathTar 

func (s *CachedRepository) GetPathTar(tarPath, localPath string) error

func (*CachedRepository) List 

func (s *CachedRepository) List(p string) ([]string, error)

func (*CachedRepository) ListRecursive 

func (s *CachedRepository) ListRecursive(results chan<- ListResult, path string)

func (*CachedRepository) ListTarFile 

func (s *CachedRepository) ListTarFile(p string) ([]string, error)

func (*CachedRepository) MatchFilenamesRecursive 

func (s *CachedRepository) MatchFilenamesRecursive(results chan<- ListResult, path string, filename string)

func (*CachedRepository) Put 

func (s *CachedRepository) Put(p string, data []byte) error

func (*CachedRepository) PutPath 

func (s *CachedRepository) PutPath(localPath string, repoPath string) error

func (*CachedRepository) PutPathTar 

func (s *CachedRepository) PutPathTar(localPath, tarPath, includePath string) error

func (*CachedRepository) RootURL 

func (s *CachedRepository) RootURL() string

func (*CachedRepository) SyncCache 

func (s *CachedRepository) SyncCache() error

type DiskRepository 

type DiskRepository struct {
	// contains filtered or unexported fields
}

func NewDiskRepository 

func NewDiskRepository(rootDir string) (*DiskRepository, error)

func (*DiskRepository) Delete 

func (s *DiskRepository) Delete(pathToDelete string) error

Delete deletes path. If path is a directory, it recursively deletes all everything under path

func (*DiskRepository) Get 

func (s *DiskRepository) Get(path string) ([]byte, error)

Get data at path

func (*DiskRepository) GetPath 

func (s *DiskRepository) GetPath(repoDir string, localDir string) error

GetPath recursively copies repoDir to localDir

func (*DiskRepository) GetPathItemTar 

func (s *DiskRepository) GetPathItemTar(tarPath, itemPath, localPath string) error

func (*DiskRepository) GetPathTar 

func (s *DiskRepository) GetPathTar(tarPath, localPath string) error

GetPathTar extracts tarball `tarPath` to `localPath`

See repository.go for full documentation.

func (*DiskRepository) List 

func (s *DiskRepository) List(path string) ([]string, error)

List files in a path non-recursively

Returns a list of paths, prefixed with the given path, that can be passed straight to Get(). Directories are not listed. If path does not exist, an empty list will be returned.

func (*DiskRepository) ListRecursive 

func (s *DiskRepository) ListRecursive(results chan<- ListResult, folder string)

func (*DiskRepository) ListTarFile 

func (s *DiskRepository) ListTarFile(tarPath string) ([]string, error)

func (*DiskRepository) MatchFilenamesRecursive 

func (s *DiskRepository) MatchFilenamesRecursive(results chan<- ListResult, folder string, filename string)

func (*DiskRepository) Put 

func (s *DiskRepository) Put(path string, data []byte) error

Put data at path

func (*DiskRepository) PutPath 

func (s *DiskRepository) PutPath(localPath string, repoPath string) error

PutPath recursively puts the local `localPath` directory into path `repoPath` in the repository

func (*DiskRepository) PutPathTar 

func (s *DiskRepository) PutPathTar(localPath, tarPath, includePath string) error

PutPathTar recursively puts the local `localPath` directory into a tar.gz file `tarPath` in the repository If `includePath` is set, only that will be included.

See repository.go for full documentation.

func (*DiskRepository) RootURL 

func (s *DiskRepository) RootURL() string

type GCSRepository 

type GCSRepository struct {
	// contains filtered or unexported fields
}

func NewGCSRepository 

func NewGCSRepository(bucket, root string) (*GCSRepository, error)

func (*GCSRepository) CreateBucket 

func (s *GCSRepository) CreateBucket() error

func (*GCSRepository) Delete 

func (s *GCSRepository) Delete(path string) error

Delete deletes path. If path is a directory, it recursively deletes all everything under path

func (*GCSRepository) Get 

func (s *GCSRepository) Get(path string) ([]byte, error)

func (*GCSRepository) GetPath 

func (s *GCSRepository) GetPath(repoDir string, localDir string) error

GetPath recursively copies repoDir to localDir

func (*GCSRepository) GetPathItemTar 

func (s *GCSRepository) GetPathItemTar(tarPath, itemPath, localPath string) error

func (*GCSRepository) GetPathTar 

func (s *GCSRepository) GetPathTar(tarPath, localPath string) error

func (*GCSRepository) List 

func (s *GCSRepository) List(dir string) ([]string, error)

List files in a path non-recursively

func (*GCSRepository) ListRecursive 

func (s *GCSRepository) ListRecursive(results chan<- ListResult, dir string)

List files in a path recursively

func (*GCSRepository) ListTarFile 

func (s *GCSRepository) ListTarFile(tarPath string) ([]string, error)

func (*GCSRepository) MatchFilenamesRecursive 

func (s *GCSRepository) MatchFilenamesRecursive(results chan<- ListResult, folder string, filename string)

func (*GCSRepository) Put 

func (s *GCSRepository) Put(path string, data []byte) error

Put data at path

func (*GCSRepository) PutPath 

func (s *GCSRepository) PutPath(localPath string, repoPath string) error

func (*GCSRepository) PutPathTar 

func (s *GCSRepository) PutPathTar(localPath, tarPath, includePath string) error

func (*GCSRepository) RootURL 

func (s *GCSRepository) RootURL() string

type ListResult 

type ListResult struct {
	Path  string
	MD5   []byte
	Error error
}

type Repository 

type Repository interface {
	// A human-readable representation of this repository location. For example: s3://my-bucket/root
	RootURL() string

	// Get data at path
	Get(path string) ([]byte, error)

	// GetPath recursively copies repoDir to localDir
	GetPath(repoPath, localPath string) error

	// GetPathTar extracts tarball `tarPath` to `localPath`
	//
	// The first component of the tarball is stripped. E.g. Extracting a tarball with `abc123/weights` in it to `/code` would create `/code/weights`.
	GetPathTar(tarPath, localPath string) error

	// GetPathItemTar extracts `itemPath` from tarball `tarPath` to `localPath`
	//
	// itemPath can be a single file or a directory.
	GetPathItemTar(tarPath, itemPath, localPath string) error

	// Put data at path
	Put(path string, data []byte) error

	// PutPath recursively puts the local `localPath` directory into path `repoPath` in the repository
	PutPath(localPath, repoPath string) error

	// PutPathTar recursively puts the local `localPath` directory into a tar.gz file `tarPath` in the repository.
	// If `includePath` is set, only that will be included
	//
	// For example, PutPathTar("/code", "/tmp/abc123.tar.gz", "data") on these files:
	// - /code/train.py
	// - /code/data/weights
	// will result in a tarball containing:
	// - `abc123/data/weights`
	PutPathTar(localPath, tarPath, basePath string) error

	// Delete deletes path. If path is a directory, it recursively deletes
	// all everything under path
	Delete(path string) error

	// List files in a path non-recursively
	//
	// Returns a list of paths, prefixed with the given path, that can be passed straight to Get().
	// Directories are not listed.
	// If path does not exist, an empty list will be returned.
	List(path string) ([]string, error)

	// List files in a tar-file
	//
	// Returns a list of paths, present inside the give tarfile, that can be passed straight to GetPathItemTar()
	// Directories are not listed.
	ListTarFile(path string) ([]string, error)

	// List files in a path recursively
	ListRecursive(results chan<- ListResult, folder string)

	MatchFilenamesRecursive(results chan<- ListResult, folder string, filename string)
}

Repository represents a blob store

TODO: this interface needs trimming. A lot of things exist on this interface for the shared library with Python, but perhaps we could detatch that API from this. For example, this API could provide a GetPath with reader, then the shared API could add extracting from tarball on top of that.

func ForURL 

func ForURL(repositoryURL string, projectDir string) (Repository, error)

type S3Repository 

type S3Repository struct {
	// contains filtered or unexported fields
}

func NewS3Repository 

func NewS3Repository(bucket, root string) (*S3Repository, error)

func (*S3Repository) Delete 

func (s *S3Repository) Delete(path string) error

func (*S3Repository) Get 

func (s *S3Repository) Get(path string) ([]byte, error)

Get data at path

func (*S3Repository) GetPath 

func (s *S3Repository) GetPath(remoteDir string, localDir string) error

GetPath recursively copies repoDir to localDir

func (*S3Repository) GetPathItemTar 

func (s *S3Repository) GetPathItemTar(tarPath, itemPath, localPath string) error

func (*S3Repository) GetPathTar 

func (s *S3Repository) GetPathTar(tarPath, localPath string) error

func (*S3Repository) List 

func (s *S3Repository) List(dir string) ([]string, error)

List files in a path non-recursively

func (*S3Repository) ListRecursive 

func (s *S3Repository) ListRecursive(results chan<- ListResult, dir string)

func (*S3Repository) ListTarFile 

func (s *S3Repository) ListTarFile(tarPath string) ([]string, error)

func (*S3Repository) MatchFilenamesRecursive 

func (s *S3Repository) MatchFilenamesRecursive(results chan<- ListResult, folder string, filename string)

func (*S3Repository) Put 

func (s *S3Repository) Put(path string, data []byte) error

Put data at path

func (*S3Repository) PutPath 

func (s *S3Repository) PutPath(localPath string, destPath string) error

func (*S3Repository) PutPathTar 

func (s *S3Repository) PutPathTar(localPath, tarPath, includePath string) error

func (*S3Repository) RootURL 

func (s *S3Repository) RootURL() string

type Scheme 

type Scheme string
const (
	SchemeDisk Scheme = "file"
	SchemeS3   Scheme = "s3"
	SchemeGCS  Scheme = "gs"
)

func SplitURL 

func SplitURL(repositoryURL string) (scheme Scheme, bucket string, root string, err error)

SplitURL splits a repository URL into <scheme>://<path>

type Spec 

type Spec struct {
	Version int `json:"version"`
}

func LoadSpec 

func LoadSpec(r Repository) (*Spec, error)

LoadSpec returns the repository spec, or nil if the repository doesn't have a spec file

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.