cephprovider

package
v4.15.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 17, 2018 License: GPL-3.0 Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source 
const ADDR_LOCK_SIZE = 0x1000000000

4096 blocks per addr lock

View Source 
const ADDR_OBJ_SIZE = 0x0001000000
View Source 
const CJournalProviderNamespace = "journalprovider"
View Source 
const INITIAL_COLD_BASE_ADDRESS = 0
View Source 
const INITIAL_HOT_BASE_ADDRESS = 0x8000000000000000
View Source 
const MAX_EXPECTED_OBJECT_SIZE = 20485

Just over the DBSIZE

View Source 
const METADATA_BASE = 0xFF00000000000000

We know we won't get any addresses here, because this is the relocation base as well

View Source 
const MaxDistinctRanges = 64
View Source 
const MaxObjectSize = 16 * 1024 * 1024

Keeping RADOS objects small is a good idea

View Source 
const NUM_COLD_HANDLES = 128
View Source 
const NUM_HOT_HANDLES = 128
View Source 
const OFFSET_MASK = 0xFFFFFF
View Source 
const RADOS_CACHE_SIZE = 512

The number of RADOS blocks to cache (up to 16MB each, probably only 1.6MB each)

View Source 
const R_ADDRMASK = ^(uint64(R_CHUNKSIZE) - 1)
View Source 
const R_CHUNKSIZE = 1 << 17

This is the size of the readahead/behind for caching. It rounds down so it is sometimes readahead sometimes readbehind

View Source 
const R_OFFSETMASK = (uint64(R_CHUNKSIZE) - 1)
View Source 
const SBLOCKS_PER_CHUNK = 1 << SBLOCK_CHUNK_SHIFT
View Source 
const SBLOCK_CHUNK_MASK = 0xFFFFF
View Source 
const SBLOCK_CHUNK_SHIFT = 20

Makes 16MB for 16B sblocks

View Source 
const SBLOCK_SIZE = 16
View Source 
const SEGCACHE_SIZE = 100 * 1024
View Source 
const WCACHE_SIZE = 1 << 20

1MB for write cache, I doubt we will ever hit this tbh

View Source 
const WORTH_CACHING = OFFSET_MASK - MAX_EXPECTED_OBJECT_SIZE

This is how many uuid/address pairs we will keep to facilitate appending to segments instead of creating new ones.

Variables

This section is empty.

Functions

func ForgetAboutNode 

func ForgetAboutNode(ctx context.Context, ioctx *rados.IOContext, nodename string) bte.BTE

Delete all journals associated with a node and also remove the tombstone, allowing the same node name to be used again

func IsAddressHot 

func IsAddressHot(addr uint64) bool

func NewJournalProvider 

func NewJournalProvider(cfg configprovider.Configuration, ccfg configprovider.ClusterConfiguration) (jprovider.JournalProvider, bte.BTE)

func ParseObjectName 

func ParseObjectName(s string) *objName

func UUIDSliceToArr 

func UUIDSliceToArr(id []byte) [16]byte

Types

type CJournalProvider 

type CJournalProvider struct {
	// contains filtered or unexported fields
}

We also need some other metadata that we can use to learn what checkpoint number to start from

func (*CJournalProvider) Barrier 

func (jp *CJournalProvider) Barrier(ctx context.Context, cp jprovider.Checkpoint) bte.BTE

func (*CJournalProvider) ForgetAboutNode 

func (jp *CJournalProvider) ForgetAboutNode(ctx context.Context, nodename string) bte.BTE

func (*CJournalProvider) GetLatestCheckpoint 

func (jp *CJournalProvider) GetLatestCheckpoint() jprovider.Checkpoint

func (*CJournalProvider) Insert 

func (jp *CJournalProvider) Insert(ctx context.Context, rng *configprovider.MashRange, jr *jprovider.JournalRecord) (checkpoint jprovider.Checkpoint, err bte.BTE)

func (*CJournalProvider) Nodename 

func (jp *CJournalProvider) Nodename() string

func (*CJournalProvider) ObtainNodeJournals 

func (jp *CJournalProvider) ObtainNodeJournals(ctx context.Context, nodename string) (jprovider.JournalIterator, bte.BTE)

Used by a node taking control of a range, returns an iterator over all unreleased journals

func (*CJournalProvider) ReleaseAllOurJournals 

func (jp *CJournalProvider) ReleaseAllOurJournals(ctx context.Context) bte.BTE

A bit like forget about node, but still keep the node name tombstone

func (*CJournalProvider) ReleaseDisjointCheckpoint 

func (jp *CJournalProvider) ReleaseDisjointCheckpoint(ctx context.Context, cp jprovider.Checkpoint) bte.BTE

func (*CJournalProvider) ReleaseJournalEntries 

func (jp *CJournalProvider) ReleaseJournalEntries(ctx context.Context, nodename string, upto jprovider.Checkpoint, rng *configprovider.MashRange) bte.BTE

Used by both the recovering nodes and the generating nodes Given that the same journal can be processed by two different nodes across different ranges, it is important that the provider only frees resources associated with old checkpoints if they have been released across the entire range of the journal. The checkpoint is EXCLUSIVE.

func (*CJournalProvider) WaitForCheckpoint 

func (jp *CJournalProvider) WaitForCheckpoint(ctx context.Context, checkpoint jprovider.Checkpoint) bte.BTE

type CJrecord 

type CJrecord struct {
	R *jprovider.JournalRecord
	C uint64
}

func (*CJrecord) DecodeMsg 

func (z *CJrecord) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*CJrecord) EncodeMsg 

func (z *CJrecord) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*CJrecord) MarshalMsg 

func (z *CJrecord) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*CJrecord) Msgsize 

func (z *CJrecord) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*CJrecord) UnmarshalMsg 

func (z *CJrecord) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type CacheItem 

type CacheItem struct {
	// contains filtered or unexported fields
}

type CephCache 

type CephCache struct {
	// contains filtered or unexported fields
}

type CephSegment 

type CephSegment struct {
	// contains filtered or unexported fields
}

func (*CephSegment) BaseAddress 

func (seg *CephSegment) BaseAddress() uint64

Returns the address of the first free word in the segment when it was locked

func (*CephSegment) Flush 

func (seg *CephSegment) Flush()

Block until all writes are complete. Note this does not imply a flush of the underlying files.

func (*CephSegment) Unlock 

func (seg *CephSegment) Unlock()

Unlocks the segment for the StorageProvider to give to other consumers Implies a flush

func (*CephSegment) Write 

func (seg *CephSegment) Write(uuid []byte, address uint64, data []byte) (uint64, error)

Writes a slice to the segment, returns immediately Returns nil if op is OK, otherwise ErrNoSpace or ErrInvalidArgument It is up to the implementer to work out how to report no space immediately The uint64 is the address to be used for the next write

type CephStorageProvider 

type CephStorageProvider struct {
	// contains filtered or unexported fields
}

func (*CephStorageProvider) BackgroundCleanup 

func (sp *CephStorageProvider) BackgroundCleanup(uuids [][]byte) error

func (*CephStorageProvider) CreateDatabase 

func (sp *CephStorageProvider) CreateDatabase(cfg configprovider.Configuration, overwrite bool) error

Called to create the database for the first time This doesn't lock, but nobody else would be trying to do the same thing at the same time, so...

func (*CephStorageProvider) CreateJournalProvider 

func (sp *CephStorageProvider) CreateJournalProvider(ournodename string) (jprovider.JournalProvider, bte.BTE)

func (*CephStorageProvider) GetStreamVersion 

func (sp *CephStorageProvider) GetStreamVersion(ctx context.Context, uuid []byte) (uint64, error)

Gets the version of a stream. Returns 0 if none exists.

func (*CephStorageProvider) Initialize 

func (sp *CephStorageProvider) Initialize(cfg configprovider.Configuration, rm *rez.RezManager)

Called at startup of a normal run

func (*CephStorageProvider) LockCoreSegment 

func (sp *CephStorageProvider) LockCoreSegment(uuid []byte) bprovider.Segment

func (*CephStorageProvider) LockVectorSegment 

func (sp *CephStorageProvider) LockVectorSegment(uuid []byte) bprovider.Segment

func (*CephStorageProvider) ObliterateStreamMetadata 

func (sp *CephStorageProvider) ObliterateStreamMetadata(uuid []byte)

func (*CephStorageProvider) Read 

func (sp *CephStorageProvider) Read(ctx context.Context, uuid []byte, address uint64, buffer []byte) ([]byte, error)

Read the blob into the given buffer

func (*CephStorageProvider) ReadSuperBlock 

func (sp *CephStorageProvider) ReadSuperBlock(ctx context.Context, uuid []byte, version uint64, buffer []byte) ([]byte, error)

Read the given version of superblock into the buffer. mebbeh we want to cache this?

func (*CephStorageProvider) SetStreamVersion 

func (sp *CephStorageProvider) SetStreamVersion(uuid []byte, version uint64)

Sets the version of a stream. If it is in the past, it is essentially a rollback, and although no space is freed, the consecutive version numbers can be reused note to self: you must make sure not to call ReadSuperBlock on versions higher than you get from GetStreamVersion because they might succeed

func (*CephStorageProvider) WriteSuperBlock 

func (sp *CephStorageProvider) WriteSuperBlock(uuid []byte, version uint64, buffer []byte)

Writes a superblock of the given version

type CheckpointHeap 

type CheckpointHeap []uint64

func (CheckpointHeap) Len 

func (h CheckpointHeap) Len() int

func (CheckpointHeap) Less 

func (h CheckpointHeap) Less(i, j int) bool

func (*CheckpointHeap) Peek 

func (h *CheckpointHeap) Peek() uint64

func (*CheckpointHeap) Pop 

func (h *CheckpointHeap) Pop() interface{}

func (*CheckpointHeap) Push 

func (h *CheckpointHeap) Push(x interface{})

func (CheckpointHeap) Swap 

func (h CheckpointHeap) Swap(i, j int)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.