Documentation
¶
Overview ¶
utils/filestorage provides types for abstracting and implementing a system that stores files, including their metadata.
Each file in the system is identified by a unique ID, determined by the system at the time the file is stored.
File metadata includes such information as the size of the file, its checksum, and when it was created. Regardless of how it is stored in the system, at the abstraction level it is represented as a document.
Metadata can exist in the system without an associated file. However, every file must have a corresponding metadata doc stored in the system. A file can be added for a metadata doc that does not have one already.
The main type is the FileStorage interface. It exposes the core functionality of such a system. This includes adding/removing files, retrieving them or their metadata, and listing all files in the system.
The package also provides a basic implementation of FileStorage, available through NewFileStorage(). This implementation simply wraps two more focused systems: doc storage and raw file storage. The wrapper uses the doc storage to store the metadata and raw file storage to store the files.
The two subsystems are exposed via corresponding interfaces: DocStorage (and its specialization MetadataStorage) and RawFileStorage. While a single type could implement both, in practice they will be separate. The doc storage is responsible to generating the unique IDs. The raw file storage defers to the doc storage for any information about the file, including the ID.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DocStorage ¶
type DocStorage interface {
io.Closer
// Doc returns the doc that matches the ID. If there is no match,
// an error is returned (see errors.IsNotFound). Any other problem
// also results in an error.
Doc(id string) (Document, error)
// ListDocs returns a list of all the docs in the storage.
ListDocs() ([]Document, error)
// AddDoc adds the doc to the storage. If successful, the storage-
// generated ID for the doc is returned. Otherwise an error is
// returned.
AddDoc(doc Document) (string, error)
// RemoveDoc removes the matching doc from the storage. If there
// is no match an error is returned (see errors.IsNotFound). Any
// other problem also results in an error.
RemoveDoc(id string) error
}
DocStorage is an abstraction for a system that can store docs (structs). The system is expected to generate its own unique ID for each doc.
type Document ¶
type Document interface {
// ID returns the unique ID of the document.
ID() string
// SetID sets the ID of the document. If the ID is already set,
// SetID() should return true (false otherwise).
SetID(id string) (alreadySet bool)
}
Document represents a document that can be identified uniquely by a string.
type FileMetadata ¶
type FileMetadata struct {
Doc
Raw RawFileMetadata
}
FileMetadata contains the metadata for a single stored file.
func NewMetadata ¶
func NewMetadata() *FileMetadata
NewMetadata returns a new Metadata for a stored file.
func (*FileMetadata) Checksum ¶
func (m *FileMetadata) Checksum() string
func (*FileMetadata) ChecksumFormat ¶
func (m *FileMetadata) ChecksumFormat() string
func (*FileMetadata) SetFileInfo ¶
func (m *FileMetadata) SetFileInfo(size int64, checksum, format string) error
func (*FileMetadata) SetStored ¶
func (m *FileMetadata) SetStored(timestamp *time.Time)
func (*FileMetadata) Size ¶
func (m *FileMetadata) Size() int64
func (*FileMetadata) Stored ¶
func (m *FileMetadata) Stored() *time.Time
type FileStorage ¶
type FileStorage interface {
io.Closer
// Metadata returns a file's metadata.
Metadata(id string) (Metadata, error)
// Get returns a file and its metadata.
Get(id string) (Metadata, io.ReadCloser, error)
// List returns the metadata for each stored file.
List() ([]Metadata, error)
// Add stores a file and its metadata.
Add(meta Metadata, archive io.Reader) (string, error)
// SetFile stores a file for an existing metadata entry.
SetFile(id string, file io.Reader) error
// Remove removes a file from storage.
Remove(id string) error
}
FileStorage is an abstraction that can be used for the storage of files.
func NewFileStorage ¶
func NewFileStorage(meta MetadataStorage, files RawFileStorage) FileStorage
NewFileStorage returns a new FileStorage value that wraps a MetadataStorage and a RawFileStorage. It coordinates the two even though they may not be designed to be compatible (or the two may be the same value).
A stored file will always have a metadata value stored. However, it is not required to have a raw file stored.
type Metadata ¶
type Metadata interface {
Document
// Size is the size of the file (in bytes).
Size() int64
// Checksum is the checksum for the file.
Checksum() string
// ChecksumFormat is the kind (and encoding) of checksum.
ChecksumFormat() string
// Stored returns when the file was last stored. If it has not been
// stored yet, nil is returned. If it has been stored but the
// timestamp is not available, a zero value is returned
// (see Time.IsZero).
Stored() *time.Time
// SetFileInfo sets the file info on the metadata.
SetFileInfo(size int64, checksum, checksumFormat string) error
// SetStored records when the file was last stored. If the previous
// value matters, be sure to call Stored() first.
SetStored(timestamp *time.Time)
}
Metadata is the meta information for a stored file.
type MetadataDocStorage ¶
type MetadataDocStorage struct {
DocStorage
}
MetadataDocStorage provides the MetadataStorage methods than can be derived from DocStorage methods. To fully implement MetadataStorage, this type must be embedded in a type that implements the remaining methods.
func (*MetadataDocStorage) AddMetadata ¶
func (s *MetadataDocStorage) AddMetadata(meta Metadata) (string, error)
ListMetadata implements MetadataStorage.ListMetadata.
func (*MetadataDocStorage) ListMetadata ¶
func (s *MetadataDocStorage) ListMetadata() ([]Metadata, error)
ListMetadata implements MetadataStorage.ListMetadata.
func (*MetadataDocStorage) Metadata ¶
func (s *MetadataDocStorage) Metadata(id string) (Metadata, error)
Metadata implements MetadataStorage.Metadata.
func (*MetadataDocStorage) RemoveMetadata ¶
func (s *MetadataDocStorage) RemoveMetadata(id string) error
ListMetadata implements MetadataStorage.ListMetadata.
type MetadataStorage ¶
type MetadataStorage interface {
io.Closer
// Metadata returns the matching Metadata. It fails if there is no
// match (see errors.IsNotFound). Any other problems likewise
// results in an error.
Metadata(id string) (Metadata, error)
// ListMetadata returns a list of all metadata in the storage.
ListMetadata() ([]Metadata, error)
// AddMetadata adds the metadata to the storage. If successful, the
// storage-generated ID for the metadata is returned. Otherwise an
// error is returned.
AddMetadata(meta Metadata) (string, error)
// RemoveMetadata removes the matching metadata from the storage.
// If there is no match an error is returned (see errors.IsNotFound).
// Any other problem also results in an error.
RemoveMetadata(id string) error
// SetStored updates the stored metadata to indicate that the
// associated file has been successfully stored in a RawFileStorage
// system. If it does not find a stored metadata with the matching
// ID, it will return an error (see errors.IsNotFound). It also
// returns an error if it fails to update the stored metadata.
SetStored(id string) error
}
MetadataStorage is an extension of DocStorage adapted to file metadata.
type RawDoc ¶
type RawDoc struct {
// ID is the unique identifier for the document.
ID string
}
RawDoc is a basic, uniquely identifiable document.
type RawFileMetadata ¶
type RawFileMetadata struct {
// Size is the size (in bytes) of the stored file.
Size int64
// Checksum is the checksum of the stored file.
Checksum string
// ChecksumFormat describes the kind of the checksum.
ChecksumFormat string
// Stored records the timestamp of when the file was last stored.
Stored *time.Time
}
RawFileMetadata holds info specific to stored files.
type RawFileStorage ¶
type RawFileStorage interface {
io.Closer
// File returns the matching file. If there is no match an error is
// returned (see errors.IsNotFound). Any other problem also results
// in an error.
File(id string) (io.ReadCloser, error)
// AddFile adds the file to the storage. If it fails to do so,
// it returns an error. If a file is already stored for the ID,
// AddFile() fails (see errors.IsAlreadyExists).
AddFile(id string, file io.Reader, size int64) error
// RemoveFile removes the matching file from the storage. It fails
// if there is no such file (see errors.IsNotFound). Any other problem
// also results in an error.
RemoveFile(id string) error
}
RawFileStorage is an abstraction around a system that can store files. The system is expected to rely on the user for unique IDs.
Source Files