Documentation
¶
Overview ¶
Package fragment is concerned with the mapping of journal offsets to protocol.Fragments, to corresponding local or remote journal content. It provides implementation for:
- Interacting with remote fragment stores.
- Indexing local and remote Fragments (see Index).
- The construction of new Fragments from a replication stream (see Spool).
- The persisting of constructed Fragments to remote stores (see Persister).
Index ¶
- Variables
- func List(ctx context.Context, store pb.FragmentStore, name pb.Journal, ...) error
- func Open(ctx context.Context, fragment pb.Fragment) (io.ReadCloser, error)
- func Persist(ctx context.Context, spool Spool) error
- func Remove(ctx context.Context, store pb.FragmentStore, fragment pb.Fragment) error
- func SignGetURL(fragment pb.Fragment, d time.Duration) (string, error)
- type CoverSet
- type File
- type Fragment
- type Index
- func (fi *Index) EndOffset() int64
- func (fi *Index) Inspect(callback func(CoverSet) error) error
- func (fi *Index) Query(ctx context.Context, req *pb.ReadRequest) (*pb.ReadResponse, File, error)
- func (fi *Index) ReplaceRemote(set CoverSet)
- func (fi *Index) SpoolCommit(frag Fragment)
- func (fi *Index) WaitForFirstRemoteRefresh(ctx context.Context) error
- type Persister
- type Spool
- type SpoolObserver
Constants ¶
This section is empty.
Variables ¶
var FileSystemStoreRoot = "/dev/null/invalid/example/path/to/store"
FileSystemStoreRoot is the filesystem path which roots fragment ContentPaths of a file:// fragment store. It must be set at program startup prior to use.
Functions ¶
func List ¶
func List(ctx context.Context, store pb.FragmentStore, name pb.Journal, callback func(pb.Fragment)) error
List Fragments of the FragmentStore for a given journal. |callback| is invoked with each listed Fragment, and any returned error aborts the listing.
func Open ¶
Open a Reader of the Fragment on the store. The returned ReadCloser does not perform any applicable client-side decompression, but does request server decompression in the case of GZIP_OFFLOAD_DECOMPRESSION.
func Persist ¶
Persist a Spool to its store. If the Spool Fragment is already present, this is a no-op. If the Spool has not been compressed incrementally, it will be compressed before being persisted.
Types ¶
type CoverSet ¶
type CoverSet []Fragment
CoverSet maintains Fragments ordered on |Begin| and |End|, with the invariant that no Fragment is fully overlapped by another Fragment in the set (though it may be overlapped by a combination of other Fragments). Intuitively, CoverSet represents the set of offsets which are "covered" by a collection of Fragments, and is able to map, for each byte offset, a "best" covering Fragment. It employs a heuristic of preferring larger fragments (and will replace spans of overlapped smaller fragments). An implication of its invariant is that no two Fragments have the same |Begin| or |End| (as that would imply an overlap). Both are monotonically increasing in the set: set[0].Begin represents the minimum offset, and set[len(set)-1].End represents the maximum offset.
func CoverSetDifference ¶
CoverSetDifference returns the subset of Fragments in |a| which cover byte offsets not also covered by Fragments in |b|.
func WalkAllStores ¶
func WalkAllStores(ctx context.Context, name pb.Journal, stores []pb.FragmentStore) (CoverSet, error)
WalkAllStores enumerates Fragments from each of |stores| into the returned CoverSet, or returns an encountered error.
func (CoverSet) Add ¶
Add the Fragment to the CoverSet. The CoverSet is returned, along with an indication of whether an offset span was updated to reflect Fragment. All updates occur in-place.
func (CoverSet) BeginOffset ¶
BeginOffset returns the first (lowest) Begin offset of any Fragment in the CoverSet.
func (CoverSet) EndOffset ¶
EndOffset returns the last (largest) End offset of any Fragment in the set.
func (CoverSet) LongestOverlappingFragment ¶
LongestOverlappingFragment finds and returns the index |ind| of the Fragment covering |offset| which also has the most content following |offset|. If no fragment covers |offset|, the index of the next Fragment beginning after |offset| is returned (which may be beyond the current CoverSet range). |found| indicates whether an overlapping Fragment was found.
type Fragment ¶
type Fragment struct {
protocol.Fragment
// Local uncompressed file of the Fragment, or nil iff the Fragment is remote.
File File
}
Fragment wraps the protocol.Fragment type with a nil-able backing local File.
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index maintains a queryable index of local and remote journal Fragments.
func (*Index) Inspect ¶
Inspect will call |callback| with a CoverSet represeting a snapshot of all the fragments in the index. While |callback| is executing there will be no changes to the fragment set of the index.
func (*Index) Query ¶
func (fi *Index) Query(ctx context.Context, req *pb.ReadRequest) (*pb.ReadResponse, File, error)
Query the Index for a Fragment matching the ReadRequest.
func (*Index) ReplaceRemote ¶
ReplaceRemote replaces all remote Fragments in the index with |set|.
func (*Index) SpoolCommit ¶
SpoolCommit adds local Spool Fragment |frag| to the index.
type Persister ¶
type Persister struct {
// contains filtered or unexported fields
}
func NewPersister ¶
NewPersister returns an empty, initialized Persister.
func (*Persister) SpoolComplete ¶
type Spool ¶
type Spool struct {
// Fragment at time of last commit.
Fragment
// FirstAppendTime is the timestamp of the first append of the current fragment.
FirstAppendTime time.Time
// contains filtered or unexported fields
}
Spool is a Fragment which is in the process of being created, backed by a local *os.File. As commits occur and the file extent is updated, the Spool Fragment is also updated to reflect the new committed extent. At all times, the Spool Fragment is a consistent, valid Fragment.
func NewSpool ¶
func NewSpool(journal pb.Journal, observer SpoolObserver) Spool
NewSpool returns an empty Spool of |journal|.
func (*Spool) Apply ¶
func (s *Spool) Apply(r *pb.ReplicateRequest, primary bool) (pb.ReplicateResponse, error)
Apply the ReplicateRequest to the Spool, returning any encountered error.
func (*Spool) MustApply ¶
func (s *Spool) MustApply(r *pb.ReplicateRequest)
MustApply applies the ReplicateRequest, and panics if a !OK status is returned or error occurs. MustApply is a convenience for cases such as rollbacks, where the request is derived from the Spool itself and cannot reasonably fail.
type SpoolObserver ¶
type SpoolObserver interface {
// SpoolCommit is called when the Spool Fragment is extended.
SpoolCommit(Fragment)
// SpoolComplete is called when the Spool has been completed.
SpoolComplete(_ Spool, primary bool)
}
SpoolObserver is notified of important events in the Spool lifecycle.
Source Files