node

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 16, 2025 License: MIT Imports: 21 Imported by: 0

README

/pkg/node

cd /

[!NOTE] asyncmachine-go is a declarative control flow library implementing AOP and Actor Model through a clock-based state machine.

/pkg/node provides distributed workflows via state-based orchestration of worker pools. Features a failsafe supervision, as well as state machines for workers and clients. All actors communicate via aRPC, as each worker is started in a separate OS process.

Installation
import amnode "github.com/pancsta/asyncmachine-go/pkg/node"

Workflow Rules

  • worker can serve only 1 client
  • supervisor can serve many clients
  • client has a direct connection to the worker
  • supervisor can supervise only 1 kind of workers
  • client can be connected to 1 supervisor

[!NOTE] Node Worker and RPC Worker are different things. RPC Worker reports it's state to the RPC Client, which can mutate it, while a Node Worker performs work for a Node Client. Node Worker is also an RPC Worker, just like Node Supervisor is.

Flow

  • OS starts the supervisor
  • supervisor opens a public port
  • supervisor starts workers based on pool criteria
  • supervisor connects to all the workers via private ports
  • client connects to supervisor's public port and requests a worker
  • supervisor confirms with a worker and provides the client with connection info
  • client connects to the worker, and delegates work via states
  • worker reports state changes to both the client and supervisor
  • supervisor maintains the worker, eg triggers log rotation, monitors errors and restarts
  • worker delivers payload to the client via ClientSendPayload

diagram

Components

Worker

Any state machine can be exposed as a Node Worker, as long as it implements /pkg/node/states.WorkerStructDef. This can be done either manually, or by using state helpers (StructMerge, SAdd), or by generating a states file with am-gen. It's also required to have the states verified by Machine.VerifyStates. Worker should respond to WorkRequested and produce ClientSendPayload to send data to the client.

import (
    am "github.com/pancsta/asyncmachine-go/pkg/machine"
    amnode "github.com/pancsta/asyncmachine-go/pkg/node"
    arpc "github.com/pancsta/asyncmachine-go/pkg/rpc"
)

// ...

type workerHandlers struct {
    t *testing.T
}

func (w *workerHandlers) WorkRequestedState(e *am.Event) {
    // client-defined input
    input := e.Args["input"].(int)

    // create payload
    payload := &rpc.ArgsPayload{
        Name:   "mypayload",
        Data:   input * input,
        Source: e.Machine.ID,
    }

    // send payload
    e.Machine.Add1(ssW.ClientSendPayload, arpc.Pass(&arpc.A{
        Name:    payload.Name,
        Payload: payload,
    }))
}

// ...

// inherit from Node worker
ssStruct := am.StructMerge(ssnode.WorkerStruct, am.Struct{
    "Foo": {Require: am.S{"Bar"}},
    "Bar": {},
})
ssNames := am.SAdd(ssnode.WorkerStates.Names(), am.S{"Foo", "Bar"})

// init
mach := am.New(ctx, ssStruct, nil)
mach.VerifyStates(ssNames)
worker, err := NewWorker(ctx, workerKind, mach.GetStruct(), mach.StateNames(), nil)
Supervisor

Supervisor needs a path to the worker's binary (with optional parameters) for exec.Command. It exposes states like Ready, PoolReady, WorkersAvailable and awaits ProvideWorker.

import (
    am "github.com/pancsta/asyncmachine-go/pkg/machine"
    amnode "github.com/pancsta/asyncmachine-go/pkg/node"
)

// ...

var ssNames am.S
var ssStruct am.Struct
var workerKind string
var workerBin []string

// supervisor
super, err := amnode.NewSupervisor(ctx, workerKind, workerBin, ssStruct, ssNames, nil)
if err != nil {
    t.Fatal(err)
}
super.Start("localhost:1234")
err := amhelp.WaitForAll(ctx, 2*time.Second,
    super.When1(ssnode.SupervisorStates.PoolReady, ctx))

Client

Any state machine can a Node Client, as long as it implements /pkg/node/states.ClientStructDef. This can be done either manually, or by using state helpers (StructMerge, SAdd), or by generating a states file with am-gen. Client also needs to know his worker's states structure and order. To connect to the worker pool, client accepts a list of supervisor addresses (PubSub discovery in on the roadmap), and will be trying to connect to them in order. After SuperReady activates, client can call Client.ReqWorker(ctx), which will request a worker from the supervisor, resulting in WorkerReady. At this point client can access the worker at Client.WorkerRpc.Worker, add WorkRequested multiple times, and handle WorkerPayload.

import (
    am "github.com/pancsta/asyncmachine-go/pkg/machine"
    amnode "github.com/pancsta/asyncmachine-go/pkg/node"
)

// ...

var ssWorkerNames am.S
var ssWorkerStruct am.Struct
var workerKind string
var superAddrs []string

// inherit from Node client
ssStruct := am.StructMerge(ssnode.ClientStruct, am.Struct{
    "Foo": {Require: am.S{"Bar"}},
    "Bar": {},
})
ssNames := am.SAdd(ssnode.ClientStates.Names(), am.S{"Foo", "Bar"})

// describe client and worker
deps := &ClientStateDeps{
    WorkerSStruct: ssWorkerStruct,
    WorkerSNames:  ssWorkerNames,
    ClientSStruct: states.ClientStruct,
    ClientSNames:  states.ClientStates.Names(),
}

// init
client, err := amnode.NewClient(ctx, "myclient", workerKind, deps, nil)
client.Start(superAddrs)
err := amhelp.WaitForAll(ctx, 2*time.Second,
    super.When1(ssnode.ClientStates.SuperReady, ctx))

// request a worker
client.ReqWorker(ctx)
err := amhelp.WaitForAll(ctx, 2*time.Second,
    super.When1(ssnode.ClientStates.WorkerReady, ctx))
worker := client.WorkerRpc.Worker
worker.Add1(ssnode.WorkerStates.WorkRequested, am.A{"input": 2})

TODO

  • supervisor redundancy
  • connecting to a pool of supervisors +rotation

Documentation

Tests

At this point only a test suite exists, but a reference implementation is on the way. See /pkg/node/node_test.go, eg TestClientWorkerPayload and uncomment amhelp.EnableDebugging. Below a command to run an exported debugging session (no installation needed).

go run github.com/pancsta/asyncmachine-go/tools/cmd/am-dbg@latest \
  --select-machine ns-TCSF-dev-170605-0 \
  --select-transition 1 \
  --import-data https://pancsta.github.io/assets/asyncmachine-go/am-dbg-exports/worker-payload.gob.br

Status

Alpha, work in progress, not semantically versioned.

monorepo

Go back to the monorepo root to continue reading.

Documentation

Overview

Package node provides distributed worker pools with supervisors.

Index

Constants

View Source
const (
	// EnvAmNodeLogSupervisor enables machine logging for node supervisor.
	EnvAmNodeLogSupervisor = "AM_NODE_LOG_SUPERVISOR"
	// EnvAmNodeLogClient enables machine logging for node client.
	EnvAmNodeLogClient = "AM_NODE_LOG_CLIENT"
)

Variables

View Source
var (
	ErrWorker        = errors.New("worker error")
	ErrWorkerMissing = errors.New("worker missing")
	ErrWorkerHealth  = errors.New("worker failed healthcheck")
	ErrWorkerConn    = errors.New("error starting connection")
	ErrWorkerKill    = errors.New("error killing worker")
	ErrPool          = errors.New("pool error")
	ErrHeartbeat     = errors.New("heartbeat failed")
	ErrRpc           = errors.New("rpc error")
)

Functions

func AddErrPool

func AddErrPool(mach *am.Machine, err error, args am.A) error

AddErrPool wraps an error in the ErrPool sentinel and adds to a machine.

func AddErrPoolStr

func AddErrPoolStr(mach *am.Machine, msg string, args am.A) error

AddErrPoolStr wraps a msg in the ErrPool sentinel and adds to a machine.

func AddErrRpc

func AddErrRpc(mach *am.Machine, err error, args am.A) error

AddErrRpc wraps an error in the ErrRpc sentinel and adds to a machine.

func AddErrWorker

func AddErrWorker(
	event *am.Event, mach *am.Machine, err error, args am.A,
) error

AddErrWorker wraps an error in the ErrWorker sentinel and adds to a machine.

func AddErrWorkerStr

func AddErrWorkerStr(mach *am.Machine, msg string, args am.A) error

AddErrWorkerStr wraps a msg in the ErrWorker sentinel and adds to a machine.

func GetSuperClientId added in v0.9.0

func GetSuperClientId(name string) string

GetClientId returns a Node Client machine ID from a name.

func GetWorkerClientId added in v0.9.0

func GetWorkerClientId(name string) string

GetWorkerClientId returns a Node Client machine ID from a name.

func LogArgs

func LogArgs(args am.A) map[string]string

LogArgs is an args logger for A and rpc.A.

func Pass

func Pass(args *A) am.A

Pass prepares am.A from A to pass to further mutations.

func PassRpc

func PassRpc(args *A) am.A

PassRpc prepares am.A from A to pass over RPC.

Types

type A

type A struct {
	// Id is a machine ID.
	Id string `log:"id"`
	// PublicAddr is the public address of a Supervisor or WorkerRpc.
	PublicAddr string `log:"public_addr"`
	// LocalAddr is the public address of a Supervisor or WorkerRpc.
	LocalAddr string `log:"local_addr"`
	// BootAddr is the local address of the Bootstrap machine.
	BootAddr string `log:"boot_addr"`
	// NodesList is a list of available nodes (supervisors' public RPC addresses).
	NodesList []string
	// WorkerRpcId is a machine ID of the worker RPC client.
	WorkerRpcId string `log:"id"`
	// SuperRpcId is a machine ID of the super RPC client.
	SuperRpcId string `log:"id"`

	// WorkerRpc is the RPC client connected to a WorkerRpc.
	WorkerRpc *rpc.Client
	// Bootstrap is the RPC machine used to connect WorkerRpc to the Supervisor.
	Bootstrap *bootstrap
	// Dispose the worker.
	Dispose bool
	// WorkerAddr is an index for WorkerInfo.
	WorkerAddr string
	// WorkerInfo describes a worker.
	WorkerInfo *workerInfo
	// WorkersCh returns a list of workers. This channel has to be buffered.
	WorkersCh chan<- []*workerInfo
	// WorkerState is a requested state of workers, eg for listings.
	WorkerState WorkerState
}

A is a struct for node arguments. It's a typesafe alternative to am.A.

func ParseArgs

func ParseArgs(args am.A) *A

ParseArgs extracts A from am.Event.Args["am_node"].

type ARpc

type ARpc struct {
	// Id is a machine ID.
	Id string `log:"id"`
	// PublicAddr is the public address of a Supervisor or Worker.
	PublicAddr string `log:"public_addr"`
	// LocalAddr is the public address of a Supervisor, Worker, or [bootstrap].
	LocalAddr string `log:"local_addr"`
	// BootAddr is the local address of the Bootstrap machine.
	BootAddr string `log:"boot_addr"`
	// NodesList is a list of available nodes (supervisors' public RPC addresses).
	NodesList []string
	// WorkerRpcId is a machine ID of the worker RPC client.
	WorkerRpcId string `log:"worker_rpc_id"`
	// SuperRpcId is a machine ID of the super RPC client.
	SuperRpcId string `log:"super_rpc_id"`
}

ARpc is a subset of A, that can be passed over RPC.

type Client

type Client struct {
	*am.ExceptionHandler
	Mach *am.Machine

	Name       string
	SuperAddr  string
	LogEnabled bool
	// LeaveSuper is a flag to leave the supervisor after connecting to the
	// worker. TODO
	LeaveSuper bool
	// ConnTimeout is the time to wait for an outbound connection to be
	// established. Default is 5 seconds.
	ConnTimeout time.Duration

	SuperRpc  *rpc.Client
	WorkerRpc *rpc.Client
	// contains filtered or unexported fields
}

Client is a node client, connecting to a supervisor and then a worker.

func NewClient

func NewClient(ctx context.Context, id string, workerKind string,
	stateDeps *ClientStateDeps, opts *ClientOpts,
) (*Client, error)

NewClient creates a new Client instance with the provided context, id, workerKind, state dependencies, and options. Returns a pointer to the Client instance and an error if any validation or initialization fails.

func (*Client) Dispose

func (c *Client) Dispose(ctx context.Context)

Dispose deallocates resources and stops the client's RPC connections.

func (*Client) ReqWorker

func (c *Client) ReqWorker(ctx context.Context) error

ReqWorker sends a request to add a "WorkerRequested" state to the client's state machine and waits for "WorkerReady" state.

func (*Client) Start

func (c *Client) Start(nodesList []string)

Start initializes the client with a list of node addresses to connect to.

func (*Client) StartEnd

func (c *Client) StartEnd(e *am.Event)

func (*Client) StartEnter

func (c *Client) StartEnter(e *am.Event) bool

func (*Client) StartState

func (c *Client) StartState(e *am.Event)

func (*Client) Stop

func (c *Client) Stop(ctx context.Context)

Stop halts the client's connection to both the supervisor and worker RPCs, and removes the client state from the state machine.

func (*Client) WorkerPayloadEnter

func (c *Client) WorkerPayloadEnter(e *am.Event) bool

func (*Client) WorkerPayloadState

func (c *Client) WorkerPayloadState(e *am.Event)

WorkerPayloadState handles both Supervisor and Worker inbound payloads.

func (*Client) WorkerReadyState added in v0.9.0

func (c *Client) WorkerReadyState(e *am.Event)

func (*Client) WorkerRequestedEnter

func (c *Client) WorkerRequestedEnter(e *am.Event) bool

func (*Client) WorkerRequestedState

func (c *Client) WorkerRequestedState(e *am.Event)

type ClientOpts

type ClientOpts struct {
	// Parent is a parent state machine for a new client state machine. See
	// [am.Opts].
	Parent am.Api
	// TODO
	Tags []string
}

ClientOpts provides configuration options for creating a new client state machine.

type ClientStateDeps

type ClientStateDeps struct {
	ClientSStruct am.Struct
	ClientSNames  am.S
	WorkerSStruct am.Struct
	WorkerSNames  am.S
}

ClientStateDeps contains the state definitions and names of the client and worker machines, needed to create a new client.

type Supervisor

type Supervisor struct {
	*am.ExceptionHandler
	Mach *am.Machine

	// WorkerKind is the kind of worker this supervisor is managing.
	WorkerKind string
	// WorkerBin is the path and args to the worker binary.
	WorkerBin []string
	// Name is the name of the supervisor.
	Name       string
	LogEnabled bool

	// Max is the maximum number of workers. Default is 10.
	Max int
	// Min is the minimum number of workers. Default is 2.
	Min int
	// Warm is the number of warm (ready) workers. Default is 5.
	Warm int
	// MaxClientWorkers is the maximum number of workers per 1 client. Defaults to
	// Max.
	MaxClientWorkers int
	// WorkerErrTtl is the time to keep worker errors in memory. Default is 10m.
	WorkerErrTtl time.Duration
	// WorkerErrRecent is the time to consider recent errors. Default is 1m.
	WorkerErrRecent time.Duration
	// WorkerErrKill is the number of errors to kill a worker. Default is 3.
	WorkerErrKill int

	// ConnTimeout is the time to wait for an outbound connection to be
	// established. Default is 5s.
	ConnTimeout time.Duration
	// DeliveryTimeout is a timeout for RPC delivery.
	DeliveryTimeout time.Duration
	// OpTimeout is the default timeout for operations (eg getters).
	OpTimeout time.Duration
	// PoolPause is the time to wait between normalizing the pool. Default is 5s.
	PoolPause time.Duration
	// HealthcheckPause is the time between trying to get a Healtcheck response
	// from a worker.
	HealthcheckPause time.Duration
	// Heartbeat is the frequency of the Heartbeat state, which normalized the
	// pool and checks workers. Default 1m.
	Heartbeat time.Duration
	// WorkerCheckInterval defines how often to pull a worker's state. Default 1s.
	WorkerCheckInterval time.Duration

	// PublicAddr is the address for the public RPC server to listen on. The
	// effective address is at [PublicMux.Addr].
	PublicAddr string
	// PublicMux is the public listener to create RPC servers for each client.
	PublicMux *rpc.Mux
	// PublicRpc are the public RPC servers of connected clients, indexed by
	// remote addresses.
	PublicRpcs map[string]*rpc.Server

	// LocalAddr is the address for the local RPC server to listen on. The
	// effective address is at [LocalRpc.Addr].
	LocalAddr string
	// LocalRpc is the local RPC server, used by other supervisors to connect.
	// TODO rpc/mux
	LocalRpc *rpc.Server

	TestFork func(string) error
	TestKill func(string) error

	WorkerReadyState       am.HandlerFinal
	WorkerGoneState        am.HandlerFinal
	KillWorkerState        am.HandlerFinal
	ClientSendPayloadState am.HandlerFinal
	SuperSendPayloadState  am.HandlerFinal
	HealthcheckState       am.HandlerFinal
	// contains filtered or unexported fields
}

func NewSupervisor

func NewSupervisor(
	ctx context.Context, workerKind string, workerBin []string,
	workerStruct am.Struct, workerSNames am.S, opts *SupervisorOpts,
) (*Supervisor, error)

NewSupervisor initializes and returns a new Supervisor instance with specified context, worker attributes, and options.

func (*Supervisor) CheckPool

func (s *Supervisor) CheckPool() bool

CheckPool tries to set pool as ready and normalizes it, if not.

func (*Supervisor) ClientConnectedState

func (s *Supervisor) ClientConnectedState(e *am.Event)

func (*Supervisor) ClientDisconnectedEnter

func (s *Supervisor) ClientDisconnectedEnter(e *am.Event) bool

func (*Supervisor) ClientDisconnectedState

func (s *Supervisor) ClientDisconnectedState(e *am.Event)

func (*Supervisor) Dispose

func (s *Supervisor) Dispose()

func (*Supervisor) ErrWorkerState

func (s *Supervisor) ErrWorkerState(e *am.Event)

func (*Supervisor) ForkWorkerEnter

func (s *Supervisor) ForkWorkerEnter(e *am.Event) bool

func (*Supervisor) ForkWorkerState

func (s *Supervisor) ForkWorkerState(e *am.Event)

func (*Supervisor) ForkingWorkerEnter

func (s *Supervisor) ForkingWorkerEnter(e *am.Event) bool

func (*Supervisor) ForkingWorkerState

func (s *Supervisor) ForkingWorkerState(e *am.Event)

func (*Supervisor) HeartbeatState

func (s *Supervisor) HeartbeatState(e *am.Event)

func (*Supervisor) KillingWorkerEnter

func (s *Supervisor) KillingWorkerEnter(e *am.Event) bool

func (*Supervisor) KillingWorkerState

func (s *Supervisor) KillingWorkerState(e *am.Event)

func (*Supervisor) ListWorkersEnter added in v0.10.0

func (s *Supervisor) ListWorkersEnter(e *am.Event) bool

func (*Supervisor) ListWorkersState added in v0.10.0

func (s *Supervisor) ListWorkersState(e *am.Event)

func (*Supervisor) NormalizingPoolState

func (s *Supervisor) NormalizingPoolState(e *am.Event)

func (*Supervisor) PoolReadyEnter

func (s *Supervisor) PoolReadyEnter(e *am.Event) bool

func (*Supervisor) PoolReadyExit

func (s *Supervisor) PoolReadyExit(e *am.Event) bool

func (*Supervisor) ProvideWorkerEnter

func (s *Supervisor) ProvideWorkerEnter(e *am.Event) bool

func (*Supervisor) ProvideWorkerState

func (s *Supervisor) ProvideWorkerState(e *am.Event)

func (*Supervisor) SetPool

func (s *Supervisor) SetPool(min, max, warm, maxPerClient int)

SetPool sets the pool parameters with defaults.

func (*Supervisor) SetWorkerEnter added in v0.10.0

func (s *Supervisor) SetWorkerEnter(e *am.Event) bool

func (*Supervisor) SetWorkerState added in v0.10.0

func (s *Supervisor) SetWorkerState(e *am.Event)

func (*Supervisor) Start

func (s *Supervisor) Start(publicAddr string)

func (*Supervisor) StartEnd

func (s *Supervisor) StartEnd(e *am.Event)

func (*Supervisor) StartEnter

func (s *Supervisor) StartEnter(e *am.Event) bool

func (*Supervisor) StartState

func (s *Supervisor) StartState(e *am.Event)

func (*Supervisor) Stop

func (s *Supervisor) Stop()

func (*Supervisor) WorkerConnectedEnter added in v0.9.0

func (s *Supervisor) WorkerConnectedEnter(e *am.Event) bool

func (*Supervisor) WorkerConnectedState added in v0.9.0

func (s *Supervisor) WorkerConnectedState(e *am.Event)

func (*Supervisor) WorkerForkedEnter

func (s *Supervisor) WorkerForkedEnter(e *am.Event) bool

func (*Supervisor) WorkerForkedState

func (s *Supervisor) WorkerForkedState(e *am.Event)

func (*Supervisor) WorkerKilledEnter

func (s *Supervisor) WorkerKilledEnter(e *am.Event) bool

func (*Supervisor) WorkerKilledState

func (s *Supervisor) WorkerKilledState(e *am.Event)

func (*Supervisor) Workers added in v0.10.0

func (s *Supervisor) Workers(
	ctx context.Context, state WorkerState,
) ([]*workerInfo, error)

Workers returns a list of workers in a desired state. If [ctx] expires, it will reutrn nil, nil.

type SupervisorOpts

type SupervisorOpts struct {
	// InstanceNum is the number of this instance in a failsafe config, used for
	// the ID.
	InstanceNum int
	// Parent is a parent state machine for a new Supervisor state machine. See
	// [am.Opts].
	Parent am.Api
	// TODO
	Tags []string
}

type Worker

type Worker struct {
	*am.ExceptionHandler
	Mach *am.Machine

	Name string
	Kind string
	// AcceptClient is the ID of a client, passed by the supervisor. Worker should
	// only accept connections from this client.
	AcceptClient string

	// ConnTimeout is the time to wait for an outbound connection to be
	// established.
	ConnTimeout     time.Duration
	DeliveryTimeout time.Duration

	// BootAddr is the address of the bootstrap machine.
	BootAddr string
	// BootRpc is the RPC client connection to bootstrap machine, which passes
	// connection info to the Supervisor.
	BootRpc *rpc.Client

	// LocalAddr is the address of the local RPC server.
	LocalAddr string
	// LocalRpc is the local RPC server, used by the Supervisor to connect.
	LocalRpc *rpc.Server

	// PublicAddr is the address of the public RPC server.
	PublicAddr string
	// PublicRpc is the public RPC server, used by the Client to connect.
	PublicRpc *rpc.Server
}

func NewWorker

func NewWorker(ctx context.Context, kind string, workerStruct am.Struct,
	stateNames am.S, opts *WorkerOpts,
) (*Worker, error)

NewWorker initializes a new Worker instance and returns it, or an error if validation fails.

func (*Worker) ErrNetworkState

func (w *Worker) ErrNetworkState(e *am.Event)

func (*Worker) HealthcheckState

func (w *Worker) HealthcheckState(e *am.Event)

func (*Worker) LocalRpcReadyState

func (w *Worker) LocalRpcReadyState(e *am.Event)

func (*Worker) PublicRpcReadyState

func (w *Worker) PublicRpcReadyState(e *am.Event)

func (*Worker) RpcReadyState

func (w *Worker) RpcReadyState(e *am.Event)

func (*Worker) SendPayloadEnter

func (w *Worker) SendPayloadEnter(e *am.Event) bool

func (*Worker) ServeClientEnter

func (w *Worker) ServeClientEnter(e *am.Event) bool

func (*Worker) ServeClientState

func (w *Worker) ServeClientState(e *am.Event)

func (*Worker) Start

func (w *Worker) Start(bootAddr string) am.Result

Start connects the worker to the bootstrap RPC.

func (*Worker) StartEnd

func (w *Worker) StartEnd(e *am.Event)

func (*Worker) StartEnter

func (w *Worker) StartEnter(e *am.Event) bool

func (*Worker) StartState

func (w *Worker) StartState(e *am.Event)

func (*Worker) Stop

func (w *Worker) Stop(dispose bool)

Stop halts the worker's state machine and optionally disposes of its resources based on the dispose flag.

type WorkerOpts

type WorkerOpts struct {
	// Parent is a parent state machine for a new Worker state machine. See
	// [am.Opts].
	Parent am.Api
	// TODO
	Tags []string
}

type WorkerState added in v0.10.0

type WorkerState string

states of a worker

const (
	StateIniting WorkerState = "initing"
	StateRpc     WorkerState = "rpc"
	StateIdle    WorkerState = "idle"
	StateBusy    WorkerState = "busy"
	StateReady   WorkerState = "ready"
)

Directories

Path Synopsis
test

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL