interf

Documentation

Overview

Package interf provides all interfaces and constants around storage services.

Index

• Constants
• type Cache
• type File
• type Files
• type ReaderAt
• type ReaderService
• type Service

Constants

const CacheExpireSeconds = 2 * 24 * 60 * 60 // 2 days

CacheExpireSeconds is the default value n. The cache stores data for max. n seconds.

const MaxFileSize = 100 * 1024 * 1024 * 1024 // 100 GiB

MaxFileSize defines the maximum size in byte of the supported files.

const MaxReadersPerFile = 6

MaxReadersPerFile determines how many open readers can be kept for later use. This should reduce reader openings.

const MaxSectorJump = (50 * 1024 * 1024) / SectorSize // 3200 sectors (=50 MiB, ~1sec with 400 MBit/s)

MaxSectorJump determines how far you can jump backwards in an open reader. An open reader for google drive does not allow random read access. To reach a more distant sector, you either have to read up to this point or open a new reader. Opening a new reader often takes longer than reading unnecessary data.

const SectorSize = 16384 // 16 kiB

SectorSize is the size of a sector. A sector is a part of a file. It is comparable to sectors of a block device. The SectorSize is also the buffer size for the download.

Types

type Cache

type Cache interface {

	// Get returns the value or 'not found' error.
	// This method doesn't allocate memory when the capacity of buf is greater or equal to value.
	Get(fileId string, sector uint64, buf []byte) ([]byte, error)

	// Set stores the value in the cache.
	// Old data can be deleted if the cache is full.
	// The value expires after interf.CacheExpireSeconds.
	Set(fileId string, sector uint64, data []byte) error

	// Pool returns a byte pool. This means that the small byte buffers can be reused and the allocation is reduced.
	// The Pool contain 300 buffer with the size of interf.SectorSize.
	//
	// Example of use:
	//   buf := c.Pool().Get()
	//   defer c.Pool().Put(buf)
	Pool() *bpool.BytePool
}

Cache stores sectors (data blocks of a file) for a performant random read access. (@see interf.ReadAt) The cache is always at least 1024 * SectorSize big (~17 MB). If possible, there should only be one common large cache (reuse the object in your program).

type File

type File interface {

	// Id uniquely identifies a file.
	// This method is thread safe (File is an immutable object).
	// Example: 1pl-ijL8cnNcS2mBwN-ZKxHYUdL3DTl9C
	Id() string

	// Name of the file. The name is not unique and there can be multiple files with the same name.
	// This method is thread safe (File is an immutable object).
	// Example: test.dat
	Name() string

	// ModTime show the last change or update of the object (unix time; seconds).
	// If a file has never been changed, it's the time of creation.
	// This method is thread safe (File is an immutable object).
	// Example: 1584535538
	ModTime() int64

	// Size is the file size in bytes.
	// This method is thread safe (File is an immutable object).
	// Example 16317
	Size() int64

	// Md5 is the hash of the file content (hex string).
	// This method is thread safe (File is an immutable object).
	// Example: 098f6bcd4621d373c0de4e832627b4f6
	Md5() string
}

File stands for a single file in storage. File is an immutable object!

type Files

type Files interface {

	// All returns a list of all internally managed files.
	// The list is created with every call and can be changed safely.
	// There are no online connections to the storage (internal data are used).
	// This method is thread safe (Files is an immutable object).
	All() []File

	// ById returns the file with the requested file id.
	// If no file is found, the os.ErrNotExist error is returned.
	// There are no online connections to the storage (internal data are used).
	// This method is thread safe (Files is an immutable object).
	ById(id string) (File, error)

	// ByName returns the latest (File.ModTime) file found with the requested name.
	// If no file is found, the os.ErrNotExist error is returned.
	// There are no online connections to the storage (internal data are used).
	// This method is thread safe (Files is an immutable object).
	ByName(name string) (File, error)

	// ByAttr returns the first file found with the requested attributes.
	// If the parameter md5 is empty, this attribute is not considered in the search.
	// If no file is found, the os.ErrNotExist error is returned.
	// There are no online connections to the storage (internal data are used).
	// This method is thread safe (Files is an immutable object).
	ByAttr(name string, size int64, md5 string) (File, error)
}

Files maintains an internal list of files. Files and File are immutable objects!

type ReaderAt

type ReaderAt interface {
	io.ReaderAt // ReadAt(p []byte, off int64) (n int, err error)
	io.Closer   // Close() error

	// Stat returns the number of times internal processes have been run since initialization.
	// This method is relevant for testing and debugging purposes.
	// The KEY is the internal process, the VALUE is the count.
	Stat() map[string]uint64
}

ReaderAt allow random read access to a file identified by the file id. A cache must be used internally for random read access. It may also be necessary to open several internal connections to the storage.

ReaderAt extends io.ReaderAt with a Closer and is the interface that wraps the ReadAt method and the Close method.

ReadAt reads len(p) bytes into p starting at offset off in the underlying input source. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered.

When ReadAt returns n < len(p), it returns a non-nil error explaining why more bytes were not returned. In this respect, ReadAt is stricter than Read.

Even if ReadAt returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, ReadAt blocks until either all the data is available or an error occurs. In this respect ReadAt is different from Read.

If the n = len(p) bytes returned by ReadAt are at the end of the input source, ReadAt may return either err == EOF or err == nil.

If ReadAt is reading from an input source with a seek offset, ReadAt should not affect nor be affected by the underlying seek offset.

Clients of ReadAt can execute parallel ReadAt calls on the same input source.

Implementations must not retain p.

type ReaderService

type ReaderService interface {

	// Reader enables read access to a file identified by the file id.
	// The connection must be closed manually with Close() after use.
	// This method is thread-safe.
	Reader(file File, off int64) (io.ReadCloser, error)

	// LimitedReader enables read access to a file identified by the file id,
	// but stops with EOF after n bytes. This method behaves like io.LimitedReader.
	// The connection must be closed manually with Close() after use.
	// This method is thread-safe.
	LimitedReader(file File, off int64, n int64) (io.ReadCloser, error)
}

ReaderService is an part of the Service and offers all reader functions.

type Service

type Service interface {

	// Update the internal file index, which can be accessed with Files().
	// Only files in the configured root directory (see parentFolderId) are processed.
	// Folders and sub-folders are ignored. This method is very slow at the first call!
	// This method is thread-safe.
	Update() error

	// Files returns all available files.
	// This method is offline and does not trigger a connection to the storage.
	// The internal file index must be updated separately with Update().
	// The map key is the file id.
	// This method is thread-safe.
	Files() Files

	// Save reads bytes from the io.Reader r and saves them in the storage.
	// The file name can exist multiple times and existing files with the same name are not overwritten.
	// The param max limits the read bytes (see io.LimitedReader). max=0 means read until EOF.
	// Return the new file id of the saved file (if successful).
	// Don't forget to call Update().
	// This method is thread-safe.
	Save(name string, r io.Reader, max int64) (file File, err error)

	// Trash moves a file (identified via the file id) to the trash.
	// Don't forget to call Update().
	// This method is thread-safe.
	Trash(file File) error

	// ReaderService implements
	// - Reader
	// - LimitedReader
	ReaderService

	// ReaderAt allow random read access to a file identified by the file id.
	// A cache must be used internally for random read access.
	// It may also be necessary to open several internal connections to the storage.
	// The connection must be closed manually with Close() after use.
	// This method is thread-safe.
	ReaderAt(file File) (ReaderAt, error)

	// MultiReaderAt allow random read access to a series of files identified by the file ids.
	// All files except the last file must be the same size.
	// In addition, this method behaves like ReaderAt.
	//
	// A cache must be used internally for random read access.
	// It may also be necessary to open several internal connections to the storage.
	// The connection must be closed manually with Close() after use.
	// This method is thread-safe.
	MultiReaderAt(list []File) (ReaderAt, error)
}

Service is the central interface to access the storage.