backup

package
v1.7.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 13, 2025 License: AGPL-3.0 Imports: 18 Imported by: 0

Documentation

Overview

Package backup is providing commands to inspect a file system (hard-drive, USB, Android, S3) and backup medias to a remote DPhoto storage.

Index

Constants

View Source 
const (
	MediaTypeImage MediaType = "IMAGE"
	MediaTypeVideo MediaType = "VIDEO"
	MediaTypeOther MediaType = "OTHER"

	OrientationUpperLeft  ImageOrientation = "UPPER_LEFT"
	OrientationLowerRight ImageOrientation = "LOWER_RIGHT"
	OrientationUpperRight ImageOrientation = "UPPER_RIGHT"
	OrientationLowerLeft  ImageOrientation = "LOWER_LEFT"
)

Variables

View Source 
var (
	ErrAnalyserNoDateTime   = errors.New("media must have a date time included in the metadata")
	ErrAnalyserNotSupported = errors.New("media format is not supported")
)
View Source 
var (
	ErrCatalogerFilterMustBeInAlbum        = errors.New("media must be in album")
	ErrCatalogerFilterMustNotAlreadyExists = errors.New("media must not already exists")
	ErrMediaMustNotBeDuplicated            = errors.New("media is present twice in the volume")
)
View Source 
var (
	MediaCounterZero = MediaCounter{}
)
View Source 
var SupportedExtensions = map[string]MediaType{
	"jpg":  MediaTypeImage,
	"jpeg": MediaTypeImage,
	"png":  MediaTypeImage,
	"gif":  MediaTypeImage,
	"webp": MediaTypeImage,
	"raw":  MediaTypeImage,

	"svg": MediaTypeImage,
	"eps": MediaTypeImage,

	"mkv":  MediaTypeVideo,
	"mts":  MediaTypeVideo,
	"avi":  MediaTypeVideo,
	"mp4":  MediaTypeVideo,
	"mpeg": MediaTypeVideo,
	"mov":  MediaTypeVideo,
	"wmv":  MediaTypeVideo,
	"webm": MediaTypeVideo,
}

SupportedExtensions is used by SourceVolume adapters to filter files they find

Functions

This section is empty.

Types

type AlbumLookupPort added in v1.5.54 

type AlbumLookupPort interface {
	FindOrCreateAlbum(owner ownermodel.Owner, mediaTime time.Time) (folderName string, created bool, err error)
}

type AlbumReport added in v1.6.35 

type AlbumReport struct {
	// contains filtered or unexported fields
}

func NewAlbumReport added in v1.7.0 

func NewAlbumReport(mediaType MediaType, count int, size int, isNew bool) *AlbumReport

NewAlbumReport is a convenience method for testing or mocking 'backup' domain

func (*AlbumReport) IsNew added in v1.7.0 

func (c *AlbumReport) IsNew() bool

func (*AlbumReport) OfType added in v1.6.35 

func (c *AlbumReport) OfType(mediaType MediaType) MediaCounter

func (*AlbumReport) Total added in v1.6.35 

func (c *AlbumReport) Total() MediaCounter

type AnalysedMedia 

type AnalysedMedia struct {
	FoundMedia FoundMedia    // FoundMedia is the reference of the file, implementation depends on the VolumeType
	Type       MediaType     // Type is 'photo' or 'video'
	Sha256Hash string        // Sha256Hash sha256 of the file
	Details    *MediaDetails // Details are data found within the file (location, date, ...)
}

AnalysedMedia is a FoundMedia to which has been attached its type (photo / video) and other details usually found within the file.

type AnalysedMediaObserver added in v1.6.26 

type AnalysedMediaObserver interface {
	OnAnalysedMedia(ctx context.Context, media *AnalysedMedia) error
}

type AnalysedMediaObserverFunc added in v1.6.35 

type AnalysedMediaObserverFunc func(ctx context.Context, media *AnalysedMedia) error

func (AnalysedMediaObserverFunc) OnAnalysedMedia added in v1.6.35 

func (a AnalysedMediaObserverFunc) OnAnalysedMedia(ctx context.Context, media *AnalysedMedia) error

type AnalysedMediaObservers added in v1.7.0 

type AnalysedMediaObservers []AnalysedMediaObserver

func (AnalysedMediaObservers) OnAnalysedMedia added in v1.7.0 

func (a AnalysedMediaObservers) OnAnalysedMedia(ctx context.Context, media *AnalysedMedia) error

type Analyser added in v1.6.26 

type Analyser interface {
	Analyse(ctx context.Context, found FoundMedia) (*AnalysedMedia, error)
}

Analyser is a method to get the details of the media from the content of the file.

type AnalyserDecorator added in v1.5.0 

type AnalyserDecorator interface {
	Decorate(analyse Analyser, observers ...AnalyserDecoratorObserver) Analyser
}

AnalyserDecorator allows to customize the Analyser behaviour, like adding a cache.

type AnalyserDecoratorObserver added in v1.6.26 

type AnalyserDecoratorObserver interface {
	OnSkipDelegateAnalyser(ctx context.Context, found FoundMedia) error
}

AnalyserDecoratorObserver is used to observe the decorator (if the cache hits, it will call this observer).

type AnalyserFromMediaDetails added in v1.7.0 

type AnalyserFromMediaDetails struct {
	// contains filtered or unexported fields
}

AnalyserFromMediaDetails is using DetailsReader to extract data from the file (EXIF, MP4, ...).

func (*AnalyserFromMediaDetails) Analyse added in v1.7.0 

func (a *AnalyserFromMediaDetails) Analyse(ctx context.Context, found FoundMedia) (*AnalysedMedia, error)

type ArchiveMediaPort added in v1.7.0 

type ArchiveMediaPort interface {
	// ArchiveMedia uploads the file in the right folder but might change the name to avoid clash with other existing files. Use files name is always returned.
	ArchiveMedia(owner string, media *BackingUpMediaRequest) (string, error)
}

type BackingUpMediaRequest 

type BackingUpMediaRequest struct {
	AnalysedMedia    *AnalysedMedia
	CatalogReference CatalogReference
}

BackingUpMediaRequest is the requests that must be executed to back up the media

type BatchBackup added in v1.7.0 

type BatchBackup struct {
	CataloguerFactory CataloguerFactory
	DetailsReaders    []DetailsReader
	InsertMediaPort   InsertMediaPort
	ArchivePort       ArchiveMediaPort
}

func (*BatchBackup) Backup added in v1.7.0 

func (b *BatchBackup) Backup(ctx context.Context, owner ownermodel.Owner, volume SourceVolume, optionsSlice ...Options) (Report, error)

Backup is analysing each media and is backing it up if not already in the catalog.

type BatchScanner added in v1.6.35 

type BatchScanner struct {
	CataloguerFactory CataloguerFactory
	DetailsReaders    []DetailsReader
}

func (*BatchScanner) Scan added in v1.6.35 

func (s *BatchScanner) Scan(ctx context.Context, owner ownermodel.Owner, volume SourceVolume, optionSlice ...Options) ([]*ScannedFolder, error)

type CatalogMediaRequest 

type CatalogMediaRequest struct {
	BackingUpMediaRequest *BackingUpMediaRequest
	ArchiveFilename       string // ArchiveFilename is a normalised named generated and used in archive.
}

CatalogMediaRequest is the request passed to Archive domain

type CatalogReference added in v1.5.54 

type CatalogReference interface {
	// Exists returns true if the media exists in the catalog
	Exists() bool

	// AlbumCreated returns true if the album was created during the cataloger process
	AlbumCreated() bool

	// AlbumFolderName returns the name of the album where the media would be stored
	AlbumFolderName() string

	// UniqueIdentifier is identifying the media no matter its filename, its id in the catalog (if it's in it or not), its album, ... It's its signature.
	UniqueIdentifier() string

	// MediaId is the id of the media in the catalog and in the archive
	MediaId() string
}

CatalogReference is used to project where a media will fit in the catalog: its ID and its album.

type CatalogReferencerObserver added in v1.6.32 

type CatalogReferencerObserver interface {
	OnMediaCatalogued(ctx context.Context, requests []BackingUpMediaRequest) error
}

type CatalogReferencerObserverFunc added in v1.6.35 

type CatalogReferencerObserverFunc func(ctx context.Context, requests []BackingUpMediaRequest) error

func (CatalogReferencerObserverFunc) OnMediaCatalogued added in v1.6.35 

func (c CatalogReferencerObserverFunc) OnMediaCatalogued(ctx context.Context, requests []BackingUpMediaRequest) error

type CatalogReferencerObservers added in v1.7.0 

type CatalogReferencerObservers []CatalogReferencerObserver

func (CatalogReferencerObservers) OnMediaCatalogued added in v1.7.0 

func (c CatalogReferencerObservers) OnMediaCatalogued(ctx context.Context, requests []BackingUpMediaRequest) error

type Cataloguer added in v1.6.32 

type Cataloguer interface {
	Reference(ctx context.Context, medias []*AnalysedMedia, observer CatalogReferencerObserver) error
}

type CataloguerFactory added in v1.6.35 

type CataloguerFactory interface {
	NewOwnerScopedCataloguer(ctx context.Context, owner ownermodel.Owner) (Cataloguer, error)
}

CataloguerFactory returns a Cataloguer scoped to a single owner. Implementations can be read-only, or have the behaviour of creating missing albums.

type CataloguerFilterObserver added in v1.6.32 

type CataloguerFilterObserver interface {
	OnFilteredOut(ctx context.Context, media AnalysedMedia, reference CatalogReference, cause error) error
}

type CataloguerFilterObservers added in v1.7.0 

type CataloguerFilterObservers []CataloguerFilterObserver

func (CataloguerFilterObservers) OnFilteredOut added in v1.7.0 

func (c CataloguerFilterObservers) OnFilteredOut(ctx context.Context, media AnalysedMedia, reference CatalogReference, cause error) error

type ClosableFoundMedia 

type ClosableFoundMedia interface {
	Close() error
}

ClosableFoundMedia can be implemented alongside FoundMedia if the implementation requires to release resources once the media has been handled.

type ConcurrencyParameters added in v1.6.35 

type ConcurrencyParameters struct {
	ConcurrentAnalyserRoutines   int // ConcurrentAnalyserRoutines is the number of concurrent analyser (read files, compute hash, filter out duplicates, ...)
	ConcurrentCataloguerRoutines int // ConcurrentCataloguerRoutines is the number of concurrent cataloguer (find album, create new albums)
	ConcurrentUploaderRoutines   int // ConcurrentUploaderRoutines is the number of concurrent uploader (upload files)
}

func (ConcurrencyParameters) NumberOfConcurrentAnalyserRoutines added in v1.6.35 

func (c ConcurrencyParameters) NumberOfConcurrentAnalyserRoutines() int

func (ConcurrencyParameters) NumberOfConcurrentCataloguerRoutines added in v1.6.35 

func (c ConcurrencyParameters) NumberOfConcurrentCataloguerRoutines() int

func (ConcurrencyParameters) NumberOfConcurrentUploaderRoutines added in v1.6.35 

func (c ConcurrencyParameters) NumberOfConcurrentUploaderRoutines() int

type DetailsReader added in v1.7.0 

type DetailsReader interface {
	// Supports returns true if the file can be parsed with this reader. False otherwise.
	Supports(media FoundMedia, mediaType MediaType) bool
	// ReadDetails extracts metadata from the content of the file.
	ReadDetails(reader io.Reader, options DetailsReaderOptions) (*MediaDetails, error)
}

type DetailsReaderOptions 

type DetailsReaderOptions struct {
	Fast bool // Fast true indicate the parser should focus at extracting the date, nothing else TODO can be retired
}

type ExtraCounts added in v1.6.27 

type ExtraCounts struct {
	Cached   MediaCounter
	Rejected MediaCounter
}

func (ExtraCounts) String added in v1.6.27 

func (c ExtraCounts) String() interface{}

type FoundMedia 

type FoundMedia interface {
	// MediaPath return breakdown of the absolute path of the media.
	MediaPath() MediaPath
	// ReadMedia reads content of the file ; it might not be optimised to call it several times (see VolumeToBackup)
	ReadMedia() (io.ReadCloser, error)
	// Size returns the size of the file
	Size() int

	LastModification() time.Time

	String() string
}

FoundMedia represents files found on the scanned volume

func NewInMemoryMedia added in v1.5.0 

func NewInMemoryMedia(name string, date time.Time, content []byte) FoundMedia

NewInMemoryMedia creates a new FoundMedia for TESTING PURPOSE ONLY

type FullMediaSignature 

type FullMediaSignature struct {
	Sha256 string
	Size   uint
}

FullMediaSignature is the business key of the media, unique per user

func (*FullMediaSignature) String 

func (s *FullMediaSignature) String() string

type ImageOrientation 

type ImageOrientation string

ImageOrientation is teh start point of stored data

type InMemoryMedia added in v1.5.0 

type InMemoryMedia struct {
	// contains filtered or unexported fields
}

func (*InMemoryMedia) LastModification added in v1.5.0 

func (i *InMemoryMedia) LastModification() time.Time

func (*InMemoryMedia) MediaPath added in v1.5.0 

func (i *InMemoryMedia) MediaPath() MediaPath

func (*InMemoryMedia) ReadMedia added in v1.5.0 

func (i *InMemoryMedia) ReadMedia() (io.ReadCloser, error)

func (*InMemoryMedia) Size added in v1.5.0 

func (i *InMemoryMedia) Size() int

func (*InMemoryMedia) String added in v1.5.0 

func (i *InMemoryMedia) String() string

type IndexMediaPort added in v1.7.0 

type IndexMediaPort interface {
	// IndexMedias add to the catalog following medias
	IndexMedias(owner string, requests []*CatalogMediaRequest) error
}

type InsertMediaPort added in v1.5.57 

type InsertMediaPort interface {
	IndexMedias(ctx context.Context, owner ownermodel.Owner, requests []*CatalogMediaRequest) error
}

type MediaCounter 

type MediaCounter struct {
	Count int // Count is the number of medias
	Size  int // Size is the sum of the size of the medias
}

func NewMediaCounter 

func NewMediaCounter(count int, size int) MediaCounter

func (MediaCounter) Add 

func (c MediaCounter) Add(count int, size int) MediaCounter

Add creates a new MediaCounter with the delta applied ; initial MediaCounter is not updated.

func (MediaCounter) AddCounter 

func (c MediaCounter) AddCounter(counter MediaCounter) MediaCounter

AddCounter creates a new MediaCounter which is the sum of the 2 counters provided.

func (MediaCounter) IsZero 

func (c MediaCounter) IsZero() bool

IsZero returns true if it's the default value

type MediaDetails 

type MediaDetails struct {
	Width, Height             int
	DateTime                  time.Time
	Orientation               ImageOrientation
	Make                      string
	Model                     string
	GPSLatitude, GPSLongitude float64
	Duration                  int64  // Duration is the length, in milliseconds, of a video
	VideoEncoding             string // VideoEncoding is the codec used to encode the video (ex: 'H264')
}

func (*MediaDetails) String 

func (s *MediaDetails) String() string

type MediaPath 

type MediaPath struct {
	ParentFullPath string // ParentFullPath is the absolute path of the media folder (URL = ParentFullPath + Filename)
	Root           string // Root is the path or URL representing the volume in which the media has been found. (URL = Root + Path + Filename)
	Path           string // Path is the directory path relative to Root: URL = Root + Path + Filename
	Filename       string // Filename does not contain any slash, and contains the extension
	ParentDir      string // ParentDir is the name of the directory: dirname(Root + Path)
}

MediaPath is a breakdown of an absolute path, or URL, agnostic of its origin.

func (MediaPath) Absolute added in v1.5.0 

func (m MediaPath) Absolute() string

func (MediaPath) String added in v1.5.0 

func (m MediaPath) String() string

type MediaType 

type MediaType string

MediaType is photo or video

type NopeAnalyserDecorator added in v1.5.0 

type NopeAnalyserDecorator struct {
}

NopeAnalyserDecorator is a default implementation for AnalyserDecorator which doesn't decorate the AnalyseMediaFunc.

func (*NopeAnalyserDecorator) Decorate added in v1.5.0 

func (n *NopeAnalyserDecorator) Decorate(analyseFunc Analyser, observers ...AnalyserDecoratorObserver) Analyser

type Options 

type Options struct {
	RestrictedAlbumFolderName map[string]interface{} // RestrictedAlbumFolderName will restrict the media to only back up medias that are in one of these albums
	Listener                  interface{}            // Listener will receive progress events.
	SkipRejects               bool                   // SkipRejects mode will report any analysis error, or missing timestamp, and continue.
	AnalyserDecorator         AnalyserDecorator      // AnalyserDecorator is an optional decorator to add concept like caching (might be nil)
	ConcurrencyParameters     ConcurrencyParameters
	BatchSize                 int    // BatchSize is the number of items to read from the database at once (used by analyser) ; default to the maximum DynamoDB can handle
	RejectDir                 string // RejectDir is the directory where rejected files will be copied
	ChannelSize               int    // ChannelSize is a hint of the size of the channels to use. Default is set in the `chain` package (255).
}

func OptionsAnalyserDecorator added in v1.7.0 

func OptionsAnalyserDecorator(analyserDecorator AnalyserDecorator) Options

OptionsAnalyserDecorator adds a decorator on analysis function ; argument can be nil. Used to add a cache.

func OptionsBatchSize added in v1.6.35 

func OptionsBatchSize(batchSize int) Options

func OptionsChannelSize added in v1.7.1 

func OptionsChannelSize(i int) Options

func OptionsConcurrentAnalyserRoutines added in v1.6.35 

func OptionsConcurrentAnalyserRoutines(concurrent int) Options

func OptionsConcurrentCataloguerRoutines added in v1.6.35 

func OptionsConcurrentCataloguerRoutines(concurrent int) Options

func OptionsConcurrentUploaderRoutines added in v1.6.35 

func OptionsConcurrentUploaderRoutines(concurrent int) Options

func OptionsOnlyAlbums added in v1.7.0 

func OptionsOnlyAlbums(albums ...string) Options

OptionsOnlyAlbums restricts backed up medias to those in these albums

func OptionsSkipRejects added in v1.6.35 

func OptionsSkipRejects(skip bool) Options

OptionsSkipRejects disables the strict mode and ignores invalid files (wrong / no date, ...)

func OptionsWithListener added in v1.7.0 

func OptionsWithListener(listener interface{}) Options

OptionsWithListener adds a listener tracking the progress of the scan/backup

func OptionsWithRejectDir added in v1.7.0 

func OptionsWithRejectDir(rejectDir string) Options

func ReduceOptions added in v1.6.27 

func ReduceOptions(requestedOptions ...Options) Options

func (Options) GetAnalyserDecorator added in v1.5.0 

func (o Options) GetAnalyserDecorator() AnalyserDecorator

GetAnalyserDecorator is returning the AnalyserDecorator or NopeAnalyserDecorator, never nil.

func (Options) GetBatchSize added in v1.6.35 

func (o Options) GetBatchSize() int

type PostAnalyseFilter 

type PostAnalyseFilter interface {
	// AcceptAnalysedMedia returns TRUE if the media should be backed-up.
	AcceptAnalysedMedia(media *AnalysedMedia, folderName string) bool
}

type RejectedMediaObserver added in v1.6.26 

type RejectedMediaObserver interface {
	// OnRejectedMedia is called when the media is invalid and cannot be used ; the error is returned only if there is a technical issue.
	OnRejectedMedia(ctx context.Context, found FoundMedia, cause error) error
}

type RejectedMediaObservers added in v1.7.0 

type RejectedMediaObservers []RejectedMediaObserver

func (RejectedMediaObservers) OnRejectedMedia added in v1.7.0 

func (a RejectedMediaObservers) OnRejectedMedia(ctx context.Context, found FoundMedia, cause error) error

type Report added in v1.7.0 

type Report interface {
	Skipped() MediaCounter
	CountPerAlbum() map[string]*AlbumReport
}

type ScannedFolder 

type ScannedFolder struct {
	Name         string
	RelativePath string                  // RelativePath can be used for display purpose
	FolderName   string                  // FolderName is the original folder name (Name with date prefix that have been removed)
	AbsolutePath string                  // AbsolutePath is used to create a new SourceVolume
	Start, End   time.Time               // Start and End are the beginning of the day of the first media, and the beginning of the day following the last media.
	Distribution map[string]MediaCounter // Distribution is the number of media found for each day (format YYYY-MM-DD)
	RejectsCount int                     // RejectsCount is the number of media that have been rejected or filtered out in the folder
}

ScannedFolder represents a (sub)folder in the scanned target

func (*ScannedFolder) PushBoundaries 

func (f *ScannedFolder) PushBoundaries(date time.Time, size int)

PushBoundaries is updating the ScannedFolder dates, and update the counter.

type SourceVolume 

type SourceVolume interface {
	String() string
	FindMedias(ctx context.Context) ([]FoundMedia, error)
}

type TimelinePort added in v1.7.0 

type TimelinePort interface {
	FindOrCreateAlbum(mediaTime time.Time) (folderName string, created bool, err error)
	FindAlbum(dateTime time.Time) (folderName string, exists bool, err error)
}

type TrackAnalysed 

type TrackAnalysed interface {
	OnAnalysed(done, total MediaCounter, others ExtraCounts)
}

type TrackEvents 

type TrackEvents interface {
	OnEvent(event progressEvent)
}

type TrackScanComplete 

type TrackScanComplete interface {
	OnScanComplete(total MediaCounter)
}

type TrackUploaded 

type TrackUploaded interface {
	OnUploaded(done, total MediaCounter)
}

TrackUploaded includes both uploaded and skipped

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.