repo

package
v0.18.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 20, 2024 License: Apache-2.0 Imports: 58 Imported by: 20

Documentation

Overview

Package repo implements content-addressable Repository on top of BLOB storage.

Index

Constants

View Source 
const DefaultServerRepoCacheKeyDerivationAlgorithm = crypto.ScryptAlgorithm

DefaultServerRepoCacheKeyDerivationAlgorithm is the default algorithm used to derive an encryption key for the local cache when connecting to a repository through the kopia API server.

View Source 
const MaxGRPCMessageSize = 20 << 20

MaxGRPCMessageSize is the maximum size of a message sent or received over GRPC API when talking to Kopia repository server. This is bigger than the size of any possible content, which is defined by supported splitters.

Variables

View Source 
var (
	BuildInfo       = "unknown"
	BuildVersion    = "v0-unofficial"
	BuildGitHubRepo = ""
)

BuildInfo is the build information of Kopia.

View Source 
var ErrAlreadyInitialized = format.ErrAlreadyInitialized

ErrAlreadyInitialized is returned when repository is already initialized in the provided storage.

View Source 
var ErrCannotWriteToRepoConnectionWithPermissiveCacheLoading = errors.New("cannot write to repo connection with permissive cache loading")

ErrCannotWriteToRepoConnectionWithPermissiveCacheLoading error to indicate.

View Source 
var ErrInvalidPassword = format.ErrInvalidPassword

ErrInvalidPassword is returned when repository password is invalid.

View Source 
var ErrRepositoryNotInitialized = errors.New("repository not initialized in the provided storage")

ErrRepositoryNotInitialized is returned when attempting to connect to repository that has not been initialized.

View Source 
var ErrRepositoryUnavailableDueToUpgradeInProgress = errors.New("repository upgrade in progress")

ErrRepositoryUnavailableDueToUpgradeInProgress is returned when repository is undergoing upgrade that requires exclusive access.

Functions

func Connect 

func Connect(ctx context.Context, configFile string, st blob.Storage, password string, opt *ConnectOptions) error

Connect connects to the repository in the specified storage and persists the configuration and credentials in the file provided.

func ConnectAPIServer added in v0.6.0 

func ConnectAPIServer(ctx context.Context, configFile string, si *APIServerInfo, password string, opt *ConnectOptions) error

ConnectAPIServer sets up repository connection to a particular API server.

func DecodeToken 

func DecodeToken(token string) (blob.ConnectionInfo, string, error)

DecodeToken decodes the provided token and returns connection info and password if persisted.

func DirectWriteSession added in v0.8.0 

func DirectWriteSession(ctx context.Context, r DirectRepository, opt WriteSessionOptions, cb func(ctx context.Context, dw DirectRepositoryWriter) error) error

DirectWriteSession executes the provided callback in a DirectRepositoryWriter created for the purpose and flushes writes.

func Disconnect 

func Disconnect(ctx context.Context, configFile string) error

Disconnect removes the specified configuration file and any local cache directories.

func EncodeToken added in v0.14.0 

func EncodeToken(password string, ci blob.ConnectionInfo) (string, error)

EncodeToken returns an opaque token that contains the given connection information and optionally the provided password.

func GetCachingOptions added in v0.8.0 

func GetCachingOptions(ctx context.Context, configFile string) (*content.CachingOptions, error)

GetCachingOptions reads caching configuration for a given repository.

func GetDefaultHostName added in v0.7.0 

func GetDefaultHostName(ctx context.Context) string

GetDefaultHostName returns default hostname.

func GetDefaultUserName added in v0.7.0 

func GetDefaultUserName(ctx context.Context) string

GetDefaultUserName returns default username.

func GetLockingStoragePrefixes added in v0.14.0 

func GetLockingStoragePrefixes() []string

GetLockingStoragePrefixes Return all prefixes that may be maintained by Object Locking.

func Initialize 

func Initialize(ctx context.Context, st blob.Storage, opt *NewRepositoryOptions, password string) error

Initialize creates initial repository data structures in the specified storage with given credentials.

func SetCachingOptions added in v0.8.0 

func SetCachingOptions(ctx context.Context, configFile string, opt *content.CachingOptions) error

SetCachingOptions changes caching configuration for a given repository.

func SetClientOptions added in v0.7.0 

func SetClientOptions(ctx context.Context, configFile string, cliOpt ClientOptions) error

SetClientOptions updates client options stored in the provided configuration file.

func SupportedLocalCacheKeyDerivationAlgorithms added in v0.18.0 

func SupportedLocalCacheKeyDerivationAlgorithms() []string

SupportedLocalCacheKeyDerivationAlgorithms returns the supported algorithms for deriving the local cache encryption key when connecting to a repository via the kopia API server.

func WriteSession added in v0.8.0 

func WriteSession(ctx context.Context, r Repository, opt WriteSessionOptions, cb func(ctx context.Context, w RepositoryWriter) error) error

WriteSession executes the provided callback in a repository writer created for the purpose and flushes writes.

Types

type APIServerInfo added in v0.6.0 

type APIServerInfo struct {
	BaseURL                             string `json:"url"`
	TrustedServerCertificateFingerprint string `json:"serverCertFingerprint"`
	LocalCacheKeyDerivationAlgorithm    string `json:"localCacheKeyDerivationAlgorithm,omitempty"`
}

APIServerInfo is remote repository configuration stored in local configuration.

NOTE: this structure is persistent on disk may be read/written using different versions of Kopia, so it must be backwards-compatible.

Apply appropriate defaults when reading.

type ClientOptions added in v0.7.0 

type ClientOptions struct {
	Hostname string `json:"hostname"`
	Username string `json:"username"`

	ReadOnly               bool `json:"readonly,omitempty"`
	PermissiveCacheLoading bool `json:"permissiveCacheLoading,omitempty"`

	// Description is human-readable description of the repository to use in the UI.
	Description string `json:"description,omitempty"`

	EnableActions bool `json:"enableActions"`

	FormatBlobCacheDuration time.Duration `json:"formatBlobCacheDuration,omitempty"`

	Throttling *throttling.Limits `json:"throttlingLimits,omitempty"`
}

ClientOptions contains client-specific options that are persisted in local configuration file.

func (ClientOptions) ApplyDefaults added in v0.7.0 

func (o ClientOptions) ApplyDefaults(ctx context.Context, defaultDesc string) ClientOptions

ApplyDefaults returns a copy of ClientOptions with defaults filled out.

func (ClientOptions) Override added in v0.7.0 

func (o ClientOptions) Override(other ClientOptions) ClientOptions

Override returns ClientOptions that overrides fields present in the provided ClientOptions.

func (ClientOptions) UsernameAtHost added in v0.8.0 

func (o ClientOptions) UsernameAtHost() string

UsernameAtHost returns 'username@hostname' string.

type ConcatenateOptions added in v0.18.0 

type ConcatenateOptions struct {
	Compressor compression.Name
}

ConcatenateOptions describes options for concatenating objects.

type ConnectOptions 

type ConnectOptions struct {
	ClientOptions

	content.CachingOptions
}

ConnectOptions specifies options when persisting configuration to connect to a repository.

type DirectRepository added in v0.6.0 

type DirectRepository interface {
	Repository

	ObjectFormat() format.ObjectFormat
	FormatManager() *format.Manager
	BlobReader() blob.Reader
	BlobVolume() blob.Volume
	ContentReader() content.Reader
	IndexBlobs(ctx context.Context, includeInactive bool) ([]indexblob.Metadata, error)
	NewDirectWriter(ctx context.Context, opt WriteSessionOptions) (context.Context, DirectRepositoryWriter, error)
	AlsoLogToContentLog(ctx context.Context) context.Context
	UniqueID() []byte
	ConfigFilename() string
	DeriveKey(purpose []byte, keyLength int) []byte
	Token(password string) (string, error)
	Throttler() throttling.SettableThrottler
	DisableIndexRefresh()
}

DirectRepository provides additional low-level repository functionality.

type DirectRepositoryWriter added in v0.8.0 

type DirectRepositoryWriter interface {
	RepositoryWriter
	DirectRepository
	BlobStorage() blob.Storage
	ContentManager() *content.WriteManager
}

DirectRepositoryWriter provides low-level write access to the repository.

type LocalConfig 

type LocalConfig struct {
	// APIServer is only provided for remote repository.
	APIServer *APIServerInfo `json:"apiServer,omitempty"`

	// Storage is only provided for direct repository access.
	Storage *blob.ConnectionInfo `json:"storage,omitempty"`

	Caching *content.CachingOptions `json:"caching,omitempty"`

	ClientOptions
}

LocalConfig is a configuration of Kopia stored in a configuration file.

func LoadConfigFromFile added in v0.8.0 

func LoadConfigFromFile(fileName string) (*LocalConfig, error)

LoadConfigFromFile reads the local configuration from the specified file.

type NewRepositoryOptions 

type NewRepositoryOptions struct {
	UniqueID                          []byte               `json:"uniqueID"` // force the use of particular unique ID
	BlockFormat                       format.ContentFormat `json:"blockFormat"`
	DisableHMAC                       bool                 `json:"disableHMAC"`
	ObjectFormat                      format.ObjectFormat  `json:"objectFormat"` // object format
	RetentionMode                     blob.RetentionMode   `json:"retentionMode,omitempty"`
	RetentionPeriod                   time.Duration        `json:"retentionPeriod,omitempty"`
	FormatBlockKeyDerivationAlgorithm string               `json:"formatBlockKeyDerivationAlgorithm,omitempty"`
}

NewRepositoryOptions specifies options that apply to newly created repositories. All fields are optional, when not provided, reasonable defaults will be used.

type Options 

type Options struct {
	TraceStorage        bool                       // Logs all storage access using provided Printf-style function
	TimeNowFunc         func() time.Time           // Time provider
	DisableInternalLog  bool                       // Disable internal log
	UpgradeOwnerID      string                     // Owner-ID of any upgrade in progress, when this is not set the access may be restricted
	DoNotWaitForUpgrade bool                       // Disable the exponential forever backoff on an upgrade lock.
	BeforeFlush         []RepositoryWriterCallback // list of callbacks to invoke before every flush

	OnFatalError func(err error) // function to invoke when repository encounters a fatal error, usually invokes os.Exit

	// test-only flags
	TestOnlyIgnoreMissingRequiredFeatures bool // ignore missing features
}

Options provides configuration parameters for connection to a repository.

type RemoteNotifications added in v0.18.0 

type RemoteNotifications interface {
	SendNotification(ctx context.Context, templateName string, templateDataJSON []byte, severity int32) error
}

RemoteNotifications is an interface implemented by repository clients that support remote notifications.

type RemoteRetentionPolicy added in v0.14.0 

type RemoteRetentionPolicy interface {
	ApplyRetentionPolicy(ctx context.Context, sourcePath string, reallyDelete bool) ([]manifest.ID, error)
}

RemoteRetentionPolicy is an interface implemented by repository clients that support remote retention policy. when implemented, the repository server will invoke ApplyRetentionPolicy() server-side.

type Repository 

type Repository interface {
	OpenObject(ctx context.Context, id object.ID) (object.Reader, error)
	VerifyObject(ctx context.Context, id object.ID) ([]content.ID, error)
	GetManifest(ctx context.Context, id manifest.ID, data interface{}) (*manifest.EntryMetadata, error)
	FindManifests(ctx context.Context, labels map[string]string) ([]*manifest.EntryMetadata, error)
	ContentInfo(ctx context.Context, contentID content.ID) (content.Info, error)
	PrefetchContents(ctx context.Context, contentIDs []content.ID, hint string) []content.ID
	PrefetchObjects(ctx context.Context, objectIDs []object.ID, hint string) ([]content.ID, error)
	Time() time.Time
	ClientOptions() ClientOptions
	NewWriter(ctx context.Context, opt WriteSessionOptions) (context.Context, RepositoryWriter, error)
	UpdateDescription(d string)
	Refresh(ctx context.Context) error
	Close(ctx context.Context) error
}

Repository exposes public API of Kopia repository, including objects and manifests.

func Open 

func Open(ctx context.Context, configFile, password string, options *Options) (rep Repository, err error)

Open opens a Repository specified in the configuration file.

type RepositoryWriter added in v0.8.0 

type RepositoryWriter interface {
	Repository

	NewObjectWriter(ctx context.Context, opt object.WriterOptions) object.Writer
	ConcatenateObjects(ctx context.Context, objectIDs []object.ID, opt ConcatenateOptions) (object.ID, error)
	PutManifest(ctx context.Context, labels map[string]string, payload interface{}) (manifest.ID, error)
	ReplaceManifests(ctx context.Context, labels map[string]string, payload interface{}) (manifest.ID, error)
	DeleteManifest(ctx context.Context, id manifest.ID) error
	OnSuccessfulFlush(callback RepositoryWriterCallback)
	Flush(ctx context.Context) error
}

RepositoryWriter provides methods to write to a repository.

type RepositoryWriterCallback added in v0.13.0 

type RepositoryWriterCallback func(ctx context.Context, w RepositoryWriter) error

RepositoryWriterCallback is a hook function invoked before and after each flush.

type WriteSessionOptions added in v0.8.0 

type WriteSessionOptions struct {
	Purpose        string
	FlushOnFailure bool        // whether to flush regardless of write session result.
	OnUpload       func(int64) // function to invoke after completing each upload in the session.
}

WriteSessionOptions describes options for a write session.

Source Files

View all Source files

Directories

Path Synopsis
Package blob implements simple storage of immutable, unstructured binary large objects (BLOBs).
 Package blob implements simple storage of immutable, unstructured binary large objects (BLOBs).
azure
Package azure implements Azure Blob Storage.
 Package azure implements Azure Blob Storage.
b2
Package b2 implements Storage based on an Backblaze B2 bucket.
 Package b2 implements Storage based on an Backblaze B2 bucket.
beforeop
Package beforeop implements wrapper around blob.Storage that run a given callback before all operations.
 Package beforeop implements wrapper around blob.Storage that run a given callback before all operations.
filesystem
Package filesystem implements filesystem-based Storage.
 Package filesystem implements filesystem-based Storage.
gcs
Package gcs implements Storage based on Google Cloud Storage bucket.
 Package gcs implements Storage based on Google Cloud Storage bucket.
gdrive
Package gdrive implements Storage based on Google Drive.
 Package gdrive implements Storage based on Google Drive.
logging
Package logging implements wrapper around Storage that logs all activity.
 Package logging implements wrapper around Storage that logs all activity.
rclone
Package rclone implements blob storage provider proxied by rclone (http://rclone.org)
 Package rclone implements blob storage provider proxied by rclone (http://rclone.org)
readonly
Package readonly implements wrapper around readonlyStorage that prevents all mutations.
 Package readonly implements wrapper around readonlyStorage that prevents all mutations.
retrying
Package retrying implements wrapper around blob.Storage that adds retry loop around all operations in case they return unexpected errors.
 Package retrying implements wrapper around blob.Storage that adds retry loop around all operations in case they return unexpected errors.
s3
Package s3 implements Storage based on an S3 bucket.
 Package s3 implements Storage based on an S3 bucket.
sftp
Package sftp implements blob storage provided for SFTP/SSH.
 Package sftp implements blob storage provided for SFTP/SSH.
sharded
Package sharded implements common support for sharded blob providers, such as filesystem or webdav.
 Package sharded implements common support for sharded blob providers, such as filesystem or webdav.
storagemetrics
Package storagemetrics implements wrapper around Storage that adds metrics around all activity.
 Package storagemetrics implements wrapper around Storage that adds metrics around all activity.
throttling
Package throttling implements wrapper around blob.Storage that adds throttling to all calls.
 Package throttling implements wrapper around blob.Storage that adds throttling to all calls.
webdav
Package webdav implements WebDAV-based Storage.
 Package webdav implements WebDAV-based Storage.
Package compression manages compression algorithm implementations.
 Package compression manages compression algorithm implementations.
Package content implements repository support for content-addressable storage.
 Package content implements repository support for content-addressable storage.
index
Package index manages content indices.
 Package index manages content indices.
indexblob
Package indexblob manages sets of active index blobs.
 Package indexblob manages sets of active index blobs.
Package ecc implements common support for error correction in sharded blob providers
 Package ecc implements common support for error correction in sharded blob providers
Package encryption manages content encryption algorithms.
 Package encryption manages content encryption algorithms.
Package format manages kopia.repository and other central format blobs.
 Package format manages kopia.repository and other central format blobs.
Package hashing encapsulates all keyed hashing algorithms.
 Package hashing encapsulates all keyed hashing algorithms.
Package logging provides loggers for Kopia.
 Package logging provides loggers for Kopia.
Package maintenance manages automatic repository maintenance.
 Package maintenance manages automatic repository maintenance.
Package manifest implements support for managing JSON-based manifests in repository.
 Package manifest implements support for managing JSON-based manifests in repository.
Package object implements repository support for content-addressable objects of arbitrary size.
 Package object implements repository support for content-addressable objects of arbitrary size.
Package splitter manages splitting of object data into chunks.
 Package splitter manages splitting of object data into chunks.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.