storehost

package
v1.20.0-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 9, 2017 License: MIT Imports: 31 Imported by: 12

Documentation

Index

Constants

View Source 
const (

	// JobTypeReReplication is the job type for re-replication. The source of re-replication is another store
	JobTypeReReplication

	// JobTypeRemoteReplication is the job type for remote replication. The source of remote replication is
	// replicator(which will receive message from remote replicator)
	JobTypeRemoteReplication
)
View Source 
const (
	// SMReadWrite allows both read and write
	SMReadWrite = iota
	// SMReadOnly allows read only
	SMReadOnly
)
View Source 
const (
	// SeqnumInvalid designates an invalid or unknown sequence number
	SeqnumInvalid = int64(math.MaxInt64)
	// TimestampInvalid designates an invalid or unknown timestamp
	TimestampInvalid = int64(math.MaxInt64)
	// AddressInvalid designates an invalid or unknown address
	AddressInvalid = int64(math.MaxInt64)
)

Variables

This section is empty.

Functions

func ExtStatsReporterPause added in v1.26.0 

func ExtStatsReporterPause()

ExtStatsReporterPause pauses the reporting (intended for tests)

func ExtStatsReporterResume added in v1.26.0 

func ExtStatsReporterResume()

ExtStatsReporterResume resumes the reporting (intended for tests)

func ExtStatsReporterSetReportInterval added in v1.26.0 

func ExtStatsReporterSetReportInterval(interval time.Duration)

ExtStatsReporterSetReportInterval updates the report interval (intended for tests)

Types

type CreditLine 

type CreditLine interface {
	Borrow() (credits int32)
	Return(credits int32)
	Close()
}

CreditLine defines the credit-line interface

type CreditMgr 

type CreditMgr interface {
	NewCreditLine() CreditLine
	Close() // close credit mgr
}

CreditMgr defines the credit-manager interface

func NewCreditMgr 

func NewCreditMgr(totalCredits int32) CreditMgr

NewCreditMgr returns an instance of CreditMgr

type ExtStatsReporter added in v1.26.0 

type ExtStatsReporter struct {
	// contains filtered or unexported fields
}

ExtStatsReporter reports extent stats to metadata

func NewExtStatsReporter added in v1.26.0 

func NewExtStatsReporter(hostID string, xMgr *ExtentManager, mClient metadata.TChanMetadataService, logger bark.Logger) *ExtStatsReporter

NewExtStatsReporter is the constructor for ExtStatsReporter

func (*ExtStatsReporter) ExtentCleanUp added in v1.26.0 

func (t *ExtStatsReporter) ExtentCleanUp(id uuid.UUID, ext *extentContext) bool

ExtentCleanUp is the callback from extent-manager when an extent is cleaned-up/torn down

func (*ExtStatsReporter) ExtentClose added in v1.26.0 

func (t *ExtStatsReporter) ExtentClose(id uuid.UUID, ext *extentContext, intent OpenIntent) (done bool)

ExtentClose is the callback from extent-manager when an extent is closed/dereferenced

func (*ExtStatsReporter) ExtentInit added in v1.26.0 

func (t *ExtStatsReporter) ExtentInit(id uuid.UUID, ext *extentContext)

ExtentInit is the callback from extent-manager when a new extent is initialized

func (*ExtStatsReporter) ExtentOpen added in v1.26.0 

func (t *ExtStatsReporter) ExtentOpen(id uuid.UUID, ext *extentContext, intent OpenIntent)

ExtentOpen is the callback from extent-manager when an extent is opened/referenced

func (*ExtStatsReporter) Start added in v1.26.0 

func (t *ExtStatsReporter) Start()

Start registers callbacks, and kicks off reporter pump and scheduler pump

func (*ExtStatsReporter) Stop added in v1.26.0 

func (t *ExtStatsReporter) Stop()

Stop stops periodic pumps for ExtStatsReporter

type ExtentCallbacks added in v1.26.0 

type ExtentCallbacks interface {

	// ExtentInit defines the callback function called when a new extent context
	// is initialized in memory (this could be for a new or existing extent);
	// the callback is called with extent-lock held exclusive.
	ExtentInit(id uuid.UUID, ext *extentContext)

	// ExtentOpen defines the callback function called when an extent is referenced;
	// this is called immediately after InitCallback and for every open that comes
	// in while the extent is 'active';
	// the callback is called with extent-lock held shared.
	ExtentOpen(id uuid.UUID, ext *extentContext, intent OpenIntent)

	// ExtentClose defines the callback function called when an extent is dereferenced;
	// the extent could still be 'active' due to other references;
	// called with extent-lock held shared.
	// the callback can return a 'false' to indicate to the extent-manager that it needs
	// the extent-context to be held on (not torn-down yet). when it is done and safe
	// for the extent-context to be torn down, it is should call 'CallbackDone'.
	ExtentClose(id uuid.UUID, ext *extentContext, intent OpenIntent) (done bool)

	// ExtentCleanUp defines the callback function called when an extent is cleaned-up,
	// ie, when all the references to the extent have been removed and the extent
	// context is about to be torn down;
	// called with no locks held; the extentContext is already in 'closed' state
	// ensuring it will not be racing with any activity.
	// the callback can return a 'false' to indicate to the extent-manager that it needs
	// the extent-context to be held on (not torn-down yet). when it is done and safe
	// for the extent-context to be torn down, it is should call 'CallbackDone'.
	ExtentCleanUp(id uuid.UUID, ext *extentContext) (done bool)
}

ExtentCallbacks is an interface for any objects want to subscribe to extent events

type ExtentInfo added in v1.26.0 

type ExtentInfo struct {
	Size     int64
	Modified int64
}

ExtentInfo contains basic information about the extent

type ExtentManager 

type ExtentManager struct {

	// lock, primarily used to synchronize the 'extents' map (below)
	sync.RWMutex
	// contains filtered or unexported fields
}

ExtentManager contains the map of all open extent-contexts

func NewExtentManager 

func NewExtentManager(storeMgr storage.StoreManager, m3Client metrics.Client, hostMetrics *load.HostMetrics, log bark.Logger) (xMgr *ExtentManager)

NewExtentManager initializes and returns a new 'extent manager'

func (*ExtentManager) CallbackDone added in v1.26.0 

func (xMgr *ExtentManager) CallbackDone(ext *extentContext)

CallbackDone decrement callback ref count

func (*ExtentManager) GetExtentInfo added in v1.26.0 

func (xMgr *ExtentManager) GetExtentInfo(extentID string) (info *ExtentInfo, err error)

GetExtentInfo returns info about an extent

func (*ExtentManager) IsExtentOpenedForReplication 

func (xMgr *ExtentManager) IsExtentOpenedForReplication(extentID string) bool

IsExtentOpenedForReplication checks whether an extent is already opened for replication

func (*ExtentManager) ListExtents added in v1.26.0 

func (xMgr *ExtentManager) ListExtents() (extentIDs []string, err error)

ListExtents returns list of extents on store

func (*ExtentManager) OpenExtent 

func (xMgr *ExtentManager) OpenExtent(id uuid.UUID, mode Mode, intent OpenIntent) (x *ExtentObj, err error)

OpenExtent opens DB and initializes state for a given extent and returns a 'handle' to it

func (*ExtentManager) RegisterCallbacks added in v1.26.0 

func (xMgr *ExtentManager) RegisterCallbacks(callbacks ExtentCallbacks)

RegisterCallbacks registers a subscriber

type ExtentObj 

type ExtentObj struct {
	// contains filtered or unexported fields
}

ExtentObj is allocated one per "open" to an extent

func (*ExtentObj) Close 

func (x *ExtentObj) Close()

Close closes this handle to the extent

func (*ExtentObj) Delete 

func (x *ExtentObj) Delete()

Delete marks the extent to be deleted when the last reference goes away

func (*ExtentObj) Mode 

func (x *ExtentObj) Mode() Mode

Mode returns this extent's mode (ie, AppendOnly or TimerQueue)

type Mode 

type Mode int

Mode of operation: timer-queue or append-only mode 

const (

	// AppendOnly mode (assumes all extents are in "append only" message queues)
	AppendOnly Mode

	// TimerQueue mode (assumes all extents are in "timer queues")
	TimerQueue

	// Log mode (similar to AppendOnly which is gated on watermark)
	Log
)

func (Mode) String 

func (t Mode) String() string

type OpenIntent 

type OpenIntent int

OpenIntent is used to indicate what the particular call to OpenExtent is intended for 

const (

	// OpenIntentAppendStream is used by OpenAppendStream
	OpenIntentAppendStream OpenIntent

	// OpenIntentReadStream is used by OpenReadStream
	OpenIntentReadStream

	// OpenIntentSealExtent is used by SealExtent
	OpenIntentSealExtent

	// OpenIntentGetAddressFromTimestamp is used by GetAddressFromTimestamp
	OpenIntentGetAddressFromTimestamp

	// OpenIntentGetExtentInfo is used by GetExtentInfo
	OpenIntentGetExtentInfo

	// OpenIntentPurgeMessages is used by PurgeMessages
	OpenIntentPurgeMessages

	// OpenIntentReplicateExtent is used by ReplicateExtent
	OpenIntentReplicateExtent
)

func (OpenIntent) String 

func (t OpenIntent) String() string

type Options 

type Options struct {
	Store   Store
	BaseDir string
}

Options are the arguments passed to the storehost

type ReplicationJob 

type ReplicationJob struct {
	// contains filtered or unexported fields
}

ReplicationJob holds the context for a replication job

func (*ReplicationJob) Done 

func (t *ReplicationJob) Done() <-chan struct{}

Done returns a channel that is signaled when the replication job completes

func (*ReplicationJob) Error 

func (t *ReplicationJob) Error() error

Error returns the error from the replication job

func (*ReplicationJob) Start 

func (t *ReplicationJob) Start() error

Start starts the replication job

func (*ReplicationJob) Stop 

func (t *ReplicationJob) Stop()

Stop stops the replication job

type ReplicationJobRunner 

type ReplicationJobRunner interface {
	common.Daemon
}

ReplicationJobRunner periodically queries metadata and start replication job for extents if there's no existing job running The runner is needed to handle error cases like store restart, etc Eventually it should be replaced with a more general purpose job framework

func NewReplicationJobRunner 

func NewReplicationJobRunner(mClient metadata.TChanMetadataService, store *StoreHost, logger bark.Logger, m3Client metrics.Client) ReplicationJobRunner

NewReplicationJobRunner returns an instance of ReplicationJobRunner

type ReplicationJobType 

type ReplicationJobType int

ReplicationJobType is the job type

type ReplicationManager 

type ReplicationManager struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

ReplicationManager holds the context for replication manager

func NewReplicationManager 

func NewReplicationManager(
	xMgr *ExtentManager,
	m3Client metrics.Client,
	mClient metadata.TChanMetadataService,
	log bark.Logger,
	hostID string,
	wsConn common.WSConnector,
) *ReplicationManager

NewReplicationManager initializes and returns a new 'replication manager'

func (*ReplicationManager) NewJob 

func (t *ReplicationManager) NewJob(jobType ReplicationJobType, args *replicationArgs, log bark.Logger) *ReplicationJob

NewJob schedules starts a new replication job

type StorageMode 

type StorageMode int32

StorageMode defines the read write mode of the storage host

type StorageMonitor 

type StorageMonitor interface {
	common.Daemon
	GetStorageMode() StorageMode
}

StorageMonitor keep monitoring disk usage, and log/alert/trigger necessary handling

func NewStorageMonitor 

func NewStorageMonitor(store *StoreHost, m3Client metrics.Client, hostMetrics *load.HostMetrics, logger bark.Logger, path string) StorageMonitor

NewStorageMonitor returns an instance of NewStorageMonitor.

type StorageStatus 

type StorageStatus int32

StorageStatus defines the different storage status

type Store 

type Store int

Store indicates the underlying storage to use 

const (

	// Rockstor for store
	Rockstor Store

	// Chunky for store
	Chunky

	// ManyRocks for store
	ManyRocks

	// RockCFstor for store
	RockCFstor
)

func (Store) String 

func (t Store) String() string

type StoreHost 

type StoreHost struct {
	common.SCommon
	// contains filtered or unexported fields
}

StoreHost is the main server class for StoreHosts

func NewStoreHost 

func NewStoreHost(serviceName string, sCommon common.SCommon, mClient metadata.TChanMetadataService, opts *Options) (*StoreHost, []thrift.TChanServer)

NewStoreHost is the constructor for store host

func (*StoreHost) DisableWrite 

func (t *StoreHost) DisableWrite()

DisableWrite disables all the write

func (*StoreHost) EnableWrite 

func (t *StoreHost) EnableWrite()

EnableWrite enables write mode

func (*StoreHost) GetAddressFromTimestamp 

func (t *StoreHost) GetAddressFromTimestamp(ctx thrift.Context, req *store.GetAddressFromTimestampRequest) (res *store.GetAddressFromTimestampResult_, err error)

GetAddressFromTimestamp is the implementation of the thrift handler for store host

func (*StoreHost) GetExtentInfo 

func (t *StoreHost) GetExtentInfo(ctx thrift.Context, extReq *store.GetExtentInfoRequest) (*store.ExtentInfo, error)

GetExtentInfo is the implementation of the thrift handler for the store host

func (*StoreHost) GetNodeStatus added in v1.26.0 

func (t *StoreHost) GetNodeStatus() controller.NodeStatus

GetNodeStatus is the current status of this host

func (*StoreHost) ListExtents added in v1.26.0 

func (t *StoreHost) ListExtents(tCtx thrift.Context) (res *store.ListExtentsResult_, err error)

ListExtents lists extents available on this storehost

func (*StoreHost) OpenAppendStream 

func (t *StoreHost) OpenAppendStream(ctx thrift.Context, call storeStream.BStoreOpenAppendStreamInCall) error

OpenAppendStream is the implementation of the thrift handler for the store host

func (*StoreHost) OpenAppendStreamHandler 

func (t *StoreHost) OpenAppendStreamHandler(w http.ResponseWriter, r *http.Request)

OpenAppendStreamHandler is websocket handler for opening write stream

func (*StoreHost) OpenReadStream 

func (t *StoreHost) OpenReadStream(ctx thrift.Context, call storeStream.BStoreOpenReadStreamInCall) error

OpenReadStream is the implementation of the thrift handler for the store host

func (*StoreHost) OpenReadStreamHandler 

func (t *StoreHost) OpenReadStreamHandler(w http.ResponseWriter, r *http.Request)

OpenReadStreamHandler is websocket handler for opening read stream

func (*StoreHost) PurgeMessages 

func (t *StoreHost) PurgeMessages(ctx thrift.Context, req *store.PurgeMessagesRequest) (res *store.PurgeMessagesResult_, err error)

PurgeMessages is the implementation of the thrift handler for the store host

func (*StoreHost) ReadMessages 

func (t *StoreHost) ReadMessages(
	ctx thrift.Context, req *store.ReadMessagesRequest) (result *store.ReadMessagesResult_, err error)

ReadMessages reads a set of messages start from the set StartAddress

func (*StoreHost) RegisterWSHandler 

func (t *StoreHost) RegisterWSHandler() *http.ServeMux

RegisterWSHandler is the implementation of WSService interface

func (*StoreHost) RemoteReplicateExtent 

func (t *StoreHost) RemoteReplicateExtent(tCtx thrift.Context, req *store.RemoteReplicateExtentRequest) (err error)

RemoteReplicateExtent replicates a remote extent from replicator

func (*StoreHost) ReplicateExtent 

func (t *StoreHost) ReplicateExtent(tCtx thrift.Context, req *store.ReplicateExtentRequest) (err error)

ReplicateExtent creates a replica of the extent in the local store

func (*StoreHost) Report 

func (t *StoreHost) Report(reporter common.LoadReporter)

Report is used for reporting Host specific load to controller

func (*StoreHost) SealExtent 

func (t *StoreHost) SealExtent(ctx thrift.Context, req *store.SealExtentRequest) (err error)

SealExtent is the implementation of the thrift handler for the store host

func (*StoreHost) SetNodeStatus added in v1.26.0 

func (t *StoreHost) SetNodeStatus(status controller.NodeStatus)

SetNodeStatus sets the status of this host

func (*StoreHost) Shutdown 

func (t *StoreHost) Shutdown()

Shutdown storehost

func (*StoreHost) Start 

func (t *StoreHost) Start(thriftService []thrift.TChanServer)

Start starts the storehost service

func (*StoreHost) Stop 

func (t *StoreHost) Stop()

Stop stops the service

func (*StoreHost) UpgradeHandler added in v1.26.0 

func (t *StoreHost) UpgradeHandler(w http.ResponseWriter, r *http.Request)

UpgradeHandler implements the upgrade end point

type Throttler 

type Throttler struct {
	// contains filtered or unexported fields
}

Throttler holds the context for a throttler instance.

func NewThrottler 

func NewThrottler(refillTokens uint64, refillInterval time.Duration) *Throttler

NewThrottler instantiates a new Throttler object - refillTokens: number of tokens to issue every refillInterval (below) - refillInterval: interval at which 'refillTokens' should be issued

func (*Throttler) Allow 

func (t *Throttler) Allow() bool

Allow checks to see if a particular request should be allowed; ie, it would returns true if we are under the throttling threshold, otherwise false.

func (*Throttler) AllowN 

func (t *Throttler) AllowN(n uint64) bool

AllowN checks to see if we will be allowed to do a batch of 'n' tokens; it would return true if we are under the throttling threshold, otherwise false.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.