request

Documentation

Index

• Variables
• func NewServer(rs requestServer, clock mclock.Clock) server
• type Event
  • func (e *Event) IsRequestEvent() bool
  • func (e *Event) RequestInfo() (ServerAndID, Request, Response)
• type EventType
• type ID
• type Module
• type Request
• type RequestResponse
• type Requester
• type Response
• type Scheduler
  • func NewScheduler() *Scheduler
  • func (s *Scheduler) RegisterModule(m Module, name string)
  • func (s *Scheduler) RegisterServer(server server)
  • func (s *Scheduler) RegisterTarget(t targetData)
  • func (s *Scheduler) Start()
  • func (s *Scheduler) Stop()
  • func (s *Scheduler) Trigger()
  • func (s *Scheduler) UnregisterServer(server server)
• type Server
• type ServerAndID

Variables

var (
	// request events
	EvResponse = &EventType{Name: "response", requestEvent: true} // data: RequestResponse; sent by requestServer
	EvFail     = &EventType{Name: "fail", requestEvent: true}     // data: RequestResponse; sent by requestServer
	EvTimeout  = &EventType{Name: "timeout", requestEvent: true}  // data: RequestResponse; sent by serverWithTimeout
	// server events
	EvRegistered      = &EventType{Name: "registered"}      // data: nil; sent by Scheduler
	EvUnregistered    = &EventType{Name: "unregistered"}    // data: nil; sent by Scheduler
	EvCanRequestAgain = &EventType{Name: "canRequestAgain"} // data: nil; sent by serverWithLimits
)

Functions

func NewServer

func NewServer(rs requestServer, clock mclock.Clock) server

NewServer wraps a requestServer and returns a server

Types

type Event

type Event struct {
	Type   *EventType
	Server Server // filled by Scheduler
	Data   any
}

Event describes an event where the type of Data depends on Type. Server field is not required when sent through the event callback; it is filled out when processed by the Scheduler. Note that the Scheduler can also create and send events (EvRegistered, EvUnregistered) directly.

func (*Event) IsRequestEvent

func (e *Event) IsRequestEvent() bool

IsRequestEvent returns true if the event is a request event

func (*Event) RequestInfo

func (e *Event) RequestInfo() (ServerAndID, Request, Response)

RequestInfo assumes that the event is a request event and returns its contents in a convenient form.

type EventType

type EventType struct {
	Name string
	// contains filtered or unexported fields
}

EventType identifies an event type, either related to a request or the server in general. Server events can also be externally defined.

type ID

type ID uint64

type Module

type Module interface {
	// Process is a non-blocking function responsible for starting requests,
	// processing events and updating the target data structures(s) and the
	// internal state of the module. Module state typically consists of information
	// about pending requests and registered servers.
	// Process is always called after an event is received or after a target data
	// structure has been changed.
	//
	// Note: Process functions of different modules are never called concurrently;
	// they are called by Scheduler in the same order of priority as they were
	// registered in.
	Process(Requester, []Event)
}

Module represents a mechanism which is typically responsible for downloading and updating a passive data structure. It does not directly interact with the servers. It can start requests using the Requester interface, maintain its internal state by receiving and processing Events and update its target data structure based on the obtained data. It is the Scheduler's responsibility to feed events to the modules, call Process as long as there might be something to process and then generate request candidates using MakeRequest and start the best possible requests. Modules are called by Scheduler whenever a global trigger is fired. All events fire the trigger. Changing a target data structure also triggers a next processing round as it could make further actions possible either by the same or another Module.

type Request

type Request any

type RequestResponse

type RequestResponse struct {
	ID       ID
	Request  Request
	Response Response
}

RequestResponse is the Data type of request events.

type Requester

type Requester interface {
	CanSendTo() []Server
	Send(Server, Request) ID
	Fail(Server, string)
}

Requester allows Modules to obtain the list of momentarily available servers, start new requests and report server failure when a response has been proven to be invalid in the processing phase. Note that all Requester functions should be safe to call from Module.Process.

type Response

type Response any

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler is a modular network data retrieval framework that coordinates multiple servers and retrieval mechanisms (modules). It implements a trigger mechanism that calls the Process function of registered modules whenever either the state of existing data structures or events coming from registered servers could allow new operations.

func NewScheduler

func NewScheduler() *Scheduler

NewScheduler creates a new Scheduler.

func (*Scheduler) RegisterModule

func (s *Scheduler) RegisterModule(m Module, name string)

RegisterModule registers a module. Should be called before starting the scheduler. In each processing round the order of module processing depends on the order of registration.

func (*Scheduler) RegisterServer

func (s *Scheduler) RegisterServer(server server)

RegisterServer registers a new server.

func (*Scheduler) RegisterTarget

func (s *Scheduler) RegisterTarget(t targetData)

RegisterTarget registers a target data structure, ensuring that any changes made to it trigger a new round of Module.Process calls, giving a chance to modules to react to the changes.

func (*Scheduler) Start

func (s *Scheduler) Start()

Start starts the scheduler. It should be called after registering all modules and before registering any servers.

func (*Scheduler) Stop

func (s *Scheduler) Stop()

Stop stops the scheduler.

func (*Scheduler) Trigger

func (s *Scheduler) Trigger()

Trigger starts a new processing round. If fired during processing, it ensures another full round of processing all modules.

func (*Scheduler) UnregisterServer

func (s *Scheduler) UnregisterServer(server server)

UnregisterServer removes a registered server.

type Server

type Server interface {
	Name() string
}

Server identifies a server without allowing any direct interaction. Note: server interface is used by Scheduler and Tracker but not used by the modules that do not interact with them directly. In order to make module testing easier, Server interface is used in events and modules.

type ServerAndID

type ServerAndID struct {
	Server Server
	ID     ID
}