server

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 17, 2024 License: MIT Imports: 29 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	ErrNoConnectionsNumber = gas.Error("nconnections must be provided")

	// EndPointAuthQueueStatus is the endpoint for getting queue status.
	EndPointAuthQueueStatus = gas.EndPointAuth + queueStatusPath

	// EndPointAuthQueueBuried is the endpoint for dealing with buried items in
	// the queue.
	EndPointAuthQueueBuried = gas.EndPointAuth + queueBuriedPath

	// EndPointAuthQueueUploading is the endpoint for getting items currently
	// uploading from the queue.
	EndPointAuthQueueUploading = gas.EndPointAuth + queueUploadingPath

	// EndPointAuthQueueAll is the endpoint for getting all items currently
	// in the global put queue.
	EndPointAuthQueueAll = gas.EndPointAuth + queueAllPath

	// EndPointAuthQueueCollCreation is the endpoint for telling the server
	// when you start and stop creating collections.
	EndPointAuthQueueCollCreation = gas.EndPointAuth + queueCollCreationPath

	// EndPointAuthIRODSConnection is the endpoint for telling the server
	// that a client created or closed its connections.
	EndPointAuthIRODSConnection = gas.EndPointAuth + iRODSConnectionPath
)
View Source 
const (

	// EndPointAuthSet is the endpoint for getting and setting sets.
	EndPointAuthSet = gas.EndPointAuth + setPath

	// EndPointAuthFiles is the endpoint for setting set file paths.
	EndPointAuthFiles = gas.EndPointAuth + filePath

	// EndPointAuthDirs is the endpoint for setting set directory paths.
	EndPointAuthDirs = gas.EndPointAuth + dirPath

	// EndPointAuthEntries is the endpoint for getting set entries.
	EndPointAuthEntries = gas.EndPointAuth + entryPath

	// EndPointAuthExampleEntry is the endpoint for getting set entries.
	EndPointAuthExampleEntry = gas.EndPointAuth + exampleEntryPath

	// EndPointAuthFailedEntries is the endpoint for getting some failed set
	// entries.
	EndPointAuthFailedEntries = gas.EndPointAuth + failedEntryPath

	// EndPointAuthDiscovery is the endpoint for triggering set discovery.
	EndPointAuthDiscovery = gas.EndPointAuth + discoveryPath

	// EndPointAuthRequests is the endpoint for getting file upload requests.
	EndPointAuthRequests = gas.EndPointAuth + requestsPath

	// EndPointAuthWorking is the endpoint for advising the server you're still
	// working on Requests retrieved from EndPointAuthRequests.
	EndPointAuthWorking = gas.EndPointAuth + workingPath

	// EndPointAuthFileStatus is the endpoint for updating file upload status.
	EndPointAuthFileStatus = gas.EndPointAuth + fileStatusPath

	// EndPointAuthRetryEntries is the endpoint for retrying file uploads.
	EndPointAuthRetryEntries = gas.EndPointAuth + fileRetryPath

	ErrNoAuth          = gas.Error("auth must be enabled")
	ErrNoSetDBDirFound = gas.Error("set database directory not found")
	ErrNoRequester     = gas.Error("requester not supplied")
	ErrBadRequester    = gas.Error("you are not the set requester")
	ErrNotAdmin        = gas.Error("you are not the server admin")
	ErrBadSet          = gas.Error("set with that id does not exist")
	ErrInvalidInput    = gas.Error("invalid input")
	ErrInternal        = gas.Error("internal server error")
)
View Source 
const (
	ErrKilledDueToStuck = gas.Error(put.ErrStuckTimeout)
)
View Source 
const (
	ErrNoLogger = gas.Error("a http logger must be configured")
)

Variables

This section is empty.

Functions

This section is empty.

Types

type BuriedFilter added in v0.3.0 

type BuriedFilter struct {
	User string
	Set  string
	Path string
}

BuriedFilter describes a filter to be used when retrying or deleting buried requests, to only act of requests that match the set properties of this.

func (*BuriedFilter) RequestPasses added in v0.3.0 

func (b *BuriedFilter) RequestPasses(r *put.Request) bool

RequestPasses returns true if the given Request matches our filter details. Also returns true if User isn't set.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is used to interact with the Server over the network, with authentication.

func NewClient 

func NewClient(url, cert, jwt string) *Client

NewClient returns a Client you can use to call methods on a Server listening at the given domain:port url.

Provide a non-blank path to a certificate to force us to trust that certificate, eg. if the server was started with a self-signed certificate.

You must first gas.GetJWT() to get a JWT that you must supply here.

func (*Client) AddOrUpdateSet 

func (c *Client) AddOrUpdateSet(set *set.Set) error

AddOrUpdateSet adds details about a backup set to the Server's database.

func (*Client) AllRequests added in v0.6.0 

func (c *Client) AllRequests() ([]*put.Request, error)

AllRequests gets the details of all put requests in the queue.

func (*Client) BuriedRequests added in v0.3.0 

func (c *Client) BuriedRequests() ([]*put.Request, error)

BuriedRequests gets the details of put requests that have failed multiple times and will no longer be retried.

func (*Client) ClosedIRODSConnections added in v1.5.0 

func (c *Client) ClosedIRODSConnections() error

ClosedIRODSConnections tells the server you've closed some iRODS connections following a MakingIRODSConnections call.

func (*Client) FinishedCreatingCollections added in v0.6.0 

func (c *Client) FinishedCreatingCollections() error

FinishedCreatingCollections tells the server you've finished creating collections following a StartingToCreateCollections call.

func (*Client) GetDirs 

func (c *Client) GetDirs(setID string) ([]*set.Entry, error)

GetDirs gets the directories for the given set that were supplied to SetDirs().

func (*Client) GetExampleFile added in v0.7.0 

func (c *Client) GetExampleFile(setID string) (*set.Entry, error)

GetExampleFile gets an example (not discovered) file for the given set that was supplied to SetFiles(). If there are no undiscovered files, returns a nil entry and error.

func (*Client) GetFailedFiles added in v0.6.0 

func (c *Client) GetFailedFiles(setID string) ([]*set.Entry, int, error)

GetFailedFiles gets up to 10 examples of GetFiles() Entries that have a failed status, along with a count of how many other failed Entries there were.

func (*Client) GetFiles 

func (c *Client) GetFiles(setID string) ([]*set.Entry, error)

GetFiles gets the defined and discovered file paths and their backup status for the given set.

func (*Client) GetQueueStatus 

func (c *Client) GetQueueStatus() (*QStatus, error)

GetQueueStatus gets information about the server's queue.

func (*Client) GetSetByID 

func (c *Client) GetSetByID(requester, setID string) (*set.Set, error)

GetSetByID gets details about a given requester's backup set from the Server's database. This is a convienience function that calls GetSets() and filters on the given set ID. Returns an error if the requester has no set with the given ID.

func (*Client) GetSetByName 

func (c *Client) GetSetByName(requester, setName string) (*set.Set, error)

GetSetByName gets details about a given requester's backup set from the Server's database. This is a convienience function that calls GetSets() and filters on the given set name. Returns an error if the requester has no set with the given name.

func (*Client) GetSets 

func (c *Client) GetSets(requester string) ([]*set.Set, error)

GetSets gets details about a given requester's backup sets from the Server's database. If you started the server, the user "all" will return all sets in the system.

func (*Client) GetSomeUploadRequests 

func (c *Client) GetSomeUploadRequests() ([]*put.Request, error)

GetSomeUploadRequests gets up to 100 Requests from the global put queue and returns them, moving from "ready" status in the queue to "running".

This automatically handles regularly telling the server knows we're still working on them, stopping when you UpdateFileStatus().

Only the user who started the server has permission to call this.

func (*Client) MakingIRODSConnections added in v1.5.0 

func (c *Client) MakingIRODSConnections(number int) error

MakingIRODSConnections tells the server you've started to create iRODS connections. Be sure to defer ClosedIRODSConnections().

func (*Client) RemoveBuried added in v0.3.0 

func (c *Client) RemoveBuried(bf *BuriedFilter) (int, error)

RemoveBuried removes put requests that have failed multiple times from the queue, optionally filtered according to the BuriedFilter. Returns the number of items removed.

func (*Client) RetryBuried added in v0.3.0 

func (c *Client) RetryBuried(bf *BuriedFilter) (int, error)

RetryBuried retries put requests that have failed multiple times, optionally filtered according to the BuriedFilter. Returns the number of items retried.

func (*Client) RetryFailedSetUploads added in v0.6.0 

func (c *Client) RetryFailedSetUploads(id string) (int, error)

RetryFailedSetUploads initiates the retry of any failed uploads in the given set.

Only the user who started the server can retry sets of other users.

Returns the number of failed uploads the retry was initiated for.

func (*Client) SendPutResultsToServer 

func (c *Client) SendPutResultsToServer(uploadStarts, uploadResults, skipResults chan *put.Request,
	minMBperSecondUploadSpeed float64, minTimeForUpload, maxStuckTime time.Duration, logger log15.Logger) error

SendPutResultsToServer reads from the given channels (as returned by put.Putter.Put()) and sends the results to the server, which will deal with any failures and update its database. Could return an error related to not being able to update the server with the results.

If an upload seems to take too long, based on the given minimum MB/s upload speed, tells the server the upload might be stuck, but continues to wait an additional maxStuckTime before giving up and returning an error. (Note that uploads may still actually be occurring; we only stop updating the server; you should exit your process to stop the uploads.)

Do not call this concurrently!

func (*Client) SetDirs 

func (c *Client) SetDirs(setID string, paths []string) error

SetDirs sets the given paths as the directory paths for the backup set with the given ID.

func (*Client) SetFiles 

func (c *Client) SetFiles(setID string, paths []string) error

SetFiles sets the given paths as the file paths for the backup set with the given ID.

func (*Client) StartingToCreateCollections added in v0.6.0 

func (c *Client) StartingToCreateCollections() error

StartingToCreateCollections tells the server you've started to create collections. Be sure to defer FinishedCreatingCollections().

func (*Client) TriggerDiscovery 

func (c *Client) TriggerDiscovery(setID string) error

TriggerDiscovery tells the server that you've called SetFiles() and SetDirs() for the given set, and now want it to discover the files that exist and discover the contents of the directories, and start the process of backing up the files.

func (*Client) UpdateFileStatus 

func (c *Client) UpdateFileStatus(r *put.Request) error

UpdateFileStatus updates a file's status in the DB based on the given Request's status.

If the status isn't "uploading", this also tells the server we're no longer working on the request. The db update will not be carried out if we're not currently touching a corresponding request due to a prior GetSomeUploadRequests().

Only the user who started the server has permission to call this.

func (*Client) UploadingRequests added in v0.4.0 

func (c *Client) UploadingRequests() ([]*put.Request, error)

UploadingRequests gets the details of put requests that are currently uploading.

type Config added in v1.4.0 

type Config struct {
	// HTTPLogger is used to log all HTTP requests. This is required.
	HTTPLogger io.Writer

	// StillRunningMsgFreq is the time between slack messages being sent about
	// the server still running.
	StillRunningMsgFreq time.Duration

	// Slacker is used to send messages to a slack channel.
	Slacker set.Slacker

	// SlackMessageDebounce is the minimum time between slack upload count
	// messages. Default value means send unlimited messages, which will likely
	// result in slack restricting messages itself.
	SlackMessageDebounce time.Duration
}

Config configures the server.

type FailedEntries added in v0.6.0 

type FailedEntries struct {
	Entries []*set.Entry
	Skipped int
}

FailedEntries holds failed Entries and the number of failed ones that were skipped because there were more than 10.

type Monitor added in v0.7.0 

type Monitor struct {
	// contains filtered or unexported fields
}

Monitor represents a time sort heap of sets.

func NewMonitor added in v0.7.0 

func NewMonitor(fn MonitorCallback) *Monitor

NewMonitor returns a Monitor which will call your callback every time a set you add to this monitor needs to be discovered.

func (*Monitor) Add added in v0.7.0 

func (m *Monitor) Add(s *set.Set)

Add pushes a set to the Monitor Heap.

func (*Monitor) NextDiscovery added in v0.7.0 

func (m *Monitor) NextDiscovery() time.Time

NextDiscovery retrieves the discovery time of the next set in the heap.

Returns an empty time.Time if the heap is empty.

func (*Monitor) NextSet added in v0.7.0 

func (m *Monitor) NextSet() *set.Set

NextSet returns the next set in the heap.

If the heap is empty, it returns nil.

type MonitorCallback added in v0.7.0 

type MonitorCallback func(*set.Set)

MonitorCallback receives a set when it is time for it to be discovered.

type MonitoredSet added in v0.7.0 

type MonitoredSet struct {
	// contains filtered or unexported fields
}

MonitorSet is a set with its next discovery time.

type QStatus 

type QStatus struct {
	Total               int
	Reserved            int
	IRODSConnections    int
	CreatingCollections int
	Uploading           int
	Failed              int
	Stuck               []*put.Request
}

QStatus describes the status of a Server's in-memory put request queue.

type Server 

type Server struct {
	gas.Server
	// contains filtered or unexported fields
}

Server is used to start a web server that provides a REST API to the setdb package's database, and a website that displays the information nicely.

func New 

func New(conf Config) (*Server, error)

New creates a Server which can serve a REST API and website.

It logs to the required configured io.Writer, which could for example be syslog using the log/syslog pkg with syslog.new(syslog.LOG_INFO, "tag").

func (*Server) AllRequests added in v0.6.0 

func (s *Server) AllRequests() []*put.Request

AllRequests returns all the put requests in the global put queue.

func (*Server) BuriedRequests added in v0.3.0 

func (s *Server) BuriedRequests() []*put.Request

BuriedRequests returns the put requests that are currently buried in the global put queue.

func (*Server) EnableJobSubmission 

func (s *Server) EnableJobSubmission(putCmd, deployment, cwd, queue string, numClients int, logger log15.Logger) error

EnableJobSubmission enables submission of `ibackup put` jobs to wr in response to there being put requests from client backup sets having their discovery completed.

Supply the `ibackup put` command (ie. including absolute path to the ibackup executable and the option to get put jobs from this server).

Deployment is the wr deployment you wish to use; either 'production' or 'development'.

Added jobs will have the given cwd, which matters. If cwd is blank, the current working dir is used. If queue is not blank, that queue will be forced.

Provide a hint as the the maximum number of put job clients you'll run at once, so that reservations can be balanced between them.

func (*Server) EnableRemoteDBBackups added in v0.7.0 

func (s *Server) EnableRemoteDBBackups(remotePath string, handler put.Handler)

EnableRemoteDBBackups causes the database backup file to also be backed up to the remote path.

func (*Server) LoadSetDB 

func (s *Server) LoadSetDB(path, backupPath string) error

LoadSetDB loads the given set.db or creates it if it doesn't exist. Optionally, also provide a path to backup the database to.

It adds a number of endpoints to the REST API for working with the set and its entries:

PUT /rest/v1/auth/set : takes a set.Set encoded as JSON in the body to add or update details about the given set.

GET /rest/v1/auth/set/[requester] : takes "requester" URL parameter to get the sets requested by this requester.

PUT /rest/v1/auth/files/[id] : takes a []string of paths encoded as JSON in the body plus a set "id" URL parameter to store these as the files to back up for the set.

PUT /rest/v1/auth/dirs/[id] : takes a []string of paths encoded as JSON in the body plus a set "id" URL parameter to store these as the directories to recursively back up for the set.

GET /rest/v1/auth/discover/[id]: takes a set "id" URL parameter to trigger the discovery of files for the set.

GET /rest/v1/auth/entries/[id] : takes a set "id" URL parameter and returns the set.Entries with backup status about each file (both set with /rest/v1/auth/files and discovered inside /rest/v1/auth/dirs).

GET /rest/v1/auth/failed_entries/[id] : takes a set "id" URL parameter and returns the set.Entries with backup status about each file (both set with /rest/v1/auth/files and discovered inside /rest/v1/auth/dirs) that has failed status. Up to 10 are returned in an object with Entries and Skipped, where Skipped is the number of failed files not returned.

GET /rest/v1/auth/requests : returns about 10GB worth of upload requests from the global put queue. Only the user who started the server has permission to call this.

PUT /rest/v1/auth/working : takes a []string of Request ids encoded as JSON in the body received from the requests endpoint to advise the server you're still working on uploading those requests. Only the user who started the server has permission to call this.

PUT /rest/v1/auth/file_status : takes a put.Request encoded as JSON in the body to update the status of the corresponding set's file entry.

GET /rest/v1/auth/retry : takes a set "id" URL parameter to trigger the retry of failed file uploads for the set.

If the database indicates there are sets we were in the middle of working on, the upload requests will be added to our in-memory queue, just like during discovery.

You must call EnableAuthWithServerToken() before calling this method, and the endpoints will only let you work on sets where the Requester matches your logged-in username, or if the logged-in user is the same as the user who started the Server.

func (*Server) MakeQueueEndPoints added in v0.3.0 

func (s *Server) MakeQueueEndPoints() error

MakeQueueEndPoints adds a number of endpoints to the REST API for working with the global in-memory put request queue:

GET /rest/v1/auth/status : get the summary queue status.

GET /rest/v1/auth/buried: get details about buried items in the queue.

PUT /rest/v1/auth/buried: retry all buried items in the queue, or if there is a QueueRequest encoded as JSON in the body, limit to just those items that correspond.

DELETE /rest/v1/auth/buried: delete all buried items in the queue, or if there is a QueueRequest encoded as JSON in the body, limit to just those items that correspond.

GET /rest/v1/auth/uploading: get details about uploading items in the queue.

GET /rest/v1/auth/allrequests: get details about all items in the queue.

POST /rest/v1/auth/collcreation: notify the server you are a put client that is about to start creating collections. Provide your host:pid as a hostpid parameter.

DELETE /rest/v1/auth/collcreation: notify the server you are a put client that has just stopped creating collections. Provide your host:pid as a hostpid parameter.

You must call EnableAuthWithServerToken() before calling this method, and the non-GET endpoints will only work if the logged-in user is the same as the user who started the Server.

func (*Server) QueueStatus 

func (s *Server) QueueStatus() *QStatus

QueueStatus returns current information about the queue's stats and any possibly stuck requests.

func (*Server) RemoveBuried added in v0.3.0 

func (s *Server) RemoveBuried(bf *BuriedFilter) int

RemoveBuried removes all buried items in the global put queue, and returns the number removed.

func (*Server) RetryBuried added in v0.3.0 

func (s *Server) RetryBuried(bf *BuriedFilter) int

RetryBuried retries all buried items in the global put queue, and returns the number retried.

func (*Server) SetRemoteHardlinkLocation added in v0.9.0 

func (s *Server) SetRemoteHardlinkLocation(path string)

func (*Server) UploadingRequests added in v0.4.0 

func (s *Server) UploadingRequests() []*put.Request

UploadingRequests returns the put requests that are currently uploading from the global put queue.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.