consensus

package
v0.10.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 5, 2017 License: Apache-2.0 Imports: 25 Imported by: 0

README

The core consensus algorithm.

  • state.go - The state machine as detailed in the whitepaper
  • reactor.go - A reactor that connects the state machine to the gossip network

Go-routine summary

The reactor runs 2 go-routines for each added peer: gossipDataRoutine and gossipVotesRoutine.

The consensus state runs two persistent go-routines: timeoutRoutine and receiveRoutine. Go-routines are also started to trigger timeouts and to avoid blocking when the internalMsgQueue is really backed up.

Replay/WAL

A write-ahead log is used to record all messages processed by the receiveRoutine, which amounts to all inputs to the consensus state machine: messages from peers, messages from ourselves, and timeouts. They can be played back deterministically at startup or using the replay console.

Documentation

Index

Constants

View Source
const (
	StateChannel       = byte(0x20)
	DataChannel        = byte(0x21)
	VoteChannel        = byte(0x22)
	VoteSetBitsChannel = byte(0x23)
)
View Source
const (
	RoundStepNewHeight     = RoundStepType(0x01) // Wait til CommitTime + timeoutCommit
	RoundStepNewRound      = RoundStepType(0x02) // Setup new round and go to RoundStepPropose
	RoundStepPropose       = RoundStepType(0x03) // Did propose, gossip proposal
	RoundStepPrevote       = RoundStepType(0x04) // Did prevote, gossip prevotes
	RoundStepPrevoteWait   = RoundStepType(0x05) // Did receive any +2/3 prevotes, start timeout
	RoundStepPrecommit     = RoundStepType(0x06) // Did precommit, gossip precommits
	RoundStepPrecommitWait = RoundStepType(0x07) // Did receive any +2/3 precommits, start timeout
	RoundStepCommit        = RoundStepType(0x08) // Entered commit state machine

)

Variables

View Source
var (
	ErrPeerStateHeightRegression = errors.New("Error peer state height regression")
	ErrPeerStateInvalidStartTime = errors.New("Error peer state invalid startTime")
)
View Source
var (
	ErrInvalidProposalSignature = errors.New("Error invalid proposal signature")
	ErrInvalidProposalPOLRound  = errors.New("Error invalid proposal POL round")
	ErrAddingVote               = errors.New("Error adding vote")
	ErrVoteHeightMismatch       = errors.New("Error vote height mismatch")
)
View Source
var Major = "0" //
View Source
var Minor = "2" // replay refactor
View Source
var Revision = "2" // validation -> commit
View Source
var Spec = "1" // async

kind of arbitrary

View Source
var Version = Fmt("v%s/%s.%s.%s", Spec, Major, Minor, Revision)

Functions

func CompareHRS

func CompareHRS(h1, r1 int, s1 RoundStepType, h2, r2 int, s2 RoundStepType) int

func RunReplayFile

func RunReplayFile(config cfg.BaseConfig, csConfig *cfg.ConsensusConfig, console bool)

Types

type BlockPartMessage

type BlockPartMessage struct {
	Height int
	Round  int
	Part   *types.Part
}

BlockPartMessage is sent when gossipping a piece of the proposed block.

func (*BlockPartMessage) String

func (m *BlockPartMessage) String() string

String returns a string representation.

type CommitStepMessage added in v0.7.0

type CommitStepMessage struct {
	Height           int
	BlockPartsHeader types.PartSetHeader
	BlockParts       *cmn.BitArray
}

CommitStepMessage is sent when a block is committed.

func (*CommitStepMessage) String added in v0.7.0

func (m *CommitStepMessage) String() string

String returns a string representation.

type ConsensusMessage

type ConsensusMessage interface{}

ConsensusMessage is a message that can be sent and received on the ConsensusReactor

func DecodeMessage added in v0.7.0

func DecodeMessage(bz []byte) (msgType byte, msg ConsensusMessage, err error)

DecodeMessage decodes the given bytes into a ConsensusMessage. TODO: check for unnecessary extra bytes at the end.

type ConsensusReactor

type ConsensusReactor struct {
	p2p.BaseReactor // BaseService + p2p.Switch
	// contains filtered or unexported fields
}

ConsensusReactor defines a reactor for the consensus service.

func NewConsensusReactor

func NewConsensusReactor(consensusState *ConsensusState, fastSync bool) *ConsensusReactor

NewConsensusReactor returns a new ConsensusReactor with the given consensusState.

func (*ConsensusReactor) AddPeer

func (conR *ConsensusReactor) AddPeer(peer *p2p.Peer)

AddPeer implements Reactor

func (*ConsensusReactor) FastSync

func (conR *ConsensusReactor) FastSync() bool

FastSync returns whether the consensus reactor is in fast-sync mode.

func (*ConsensusReactor) GetChannels

func (conR *ConsensusReactor) GetChannels() []*p2p.ChannelDescriptor

GetChannels implements Reactor

func (*ConsensusReactor) OnStart

func (conR *ConsensusReactor) OnStart() error

OnStart implements BaseService.

func (*ConsensusReactor) OnStop

func (conR *ConsensusReactor) OnStop()

OnStop implements BaseService

func (*ConsensusReactor) Receive

func (conR *ConsensusReactor) Receive(chID byte, src *p2p.Peer, msgBytes []byte)

Receive implements Reactor NOTE: We process these messages even when we're fast_syncing. Messages affect either a peer state or the consensus state. Peer state updates can happen in parallel, but processing of proposals, block parts, and votes are ordered by the receiveRoutine NOTE: blocks on consensus state for proposals, block parts, and votes

func (*ConsensusReactor) RemovePeer

func (conR *ConsensusReactor) RemovePeer(peer *p2p.Peer, reason interface{})

RemovePeer implements Reactor

func (*ConsensusReactor) SetEventSwitch added in v0.7.0

func (conR *ConsensusReactor) SetEventSwitch(evsw types.EventSwitch)

SetEventSwitch implements events.Eventable

func (*ConsensusReactor) String

func (conR *ConsensusReactor) String() string

String returns a string representation of the ConsensusReactor. NOTE: For now, it is just a hard-coded string to avoid accessing unprotected shared variables. TODO: improve!

func (*ConsensusReactor) StringIndented

func (conR *ConsensusReactor) StringIndented(indent string) string

StringIndented returns an indented string representation of the ConsensusReactor

func (*ConsensusReactor) SwitchToConsensus

func (conR *ConsensusReactor) SwitchToConsensus(state *sm.State)

SwitchToConsensus switches from fast_sync mode to consensus mode. It resets the state, turns off fast_sync, and starts the consensus state-machine

type ConsensusState

type ConsensusState struct {
	cmn.BaseService

	RoundState
	// contains filtered or unexported fields
}

ConsensusState handles execution of the consensus algorithm. It processes votes and proposals, and upon reaching agreement, commits blocks to the chain and executes them against the application. The internal state machine receives input from peers, the internal validator, and from a timer.

func NewConsensusState

func NewConsensusState(config *cfg.ConsensusConfig, state *sm.State, proxyAppConn proxy.AppConnConsensus, blockStore types.BlockStore, mempool types.Mempool) *ConsensusState

NewConsensusState returns a new ConsensusState.

func (*ConsensusState) AddProposalBlockPart

func (cs *ConsensusState) AddProposalBlockPart(height, round int, part *types.Part, peerKey string) error

AddProposalBlockPart inputs a part of the proposal block.

func (*ConsensusState) AddVote

func (cs *ConsensusState) AddVote(vote *types.Vote, peerKey string) (added bool, err error)

AddVote inputs a vote.

func (*ConsensusState) GetRoundState

func (cs *ConsensusState) GetRoundState() *RoundState

GetRoundState returns a copy of the internal consensus state.

func (*ConsensusState) GetState

func (cs *ConsensusState) GetState() *sm.State

GetState returns a copy of the chain state.

func (*ConsensusState) GetValidators

func (cs *ConsensusState) GetValidators() (int, []*types.Validator)

GetValidators returns a copy of the current validators.

func (*ConsensusState) LoadCommit

func (cs *ConsensusState) LoadCommit(height int) *types.Commit

LoadCommit loads the commit for a given height.

func (*ConsensusState) OnStart

func (cs *ConsensusState) OnStart() error

OnStart implements cmn.Service. It loads the latest state via the WAL, and starts the timeout and receive routines.

func (*ConsensusState) OnStop

func (cs *ConsensusState) OnStop()

OnStop implements cmn.Service. It stops all routines and waits for the WAL to finish.

func (*ConsensusState) OpenWAL

func (cs *ConsensusState) OpenWAL(walFile string) (err error)

OpenWAL opens a file to log all consensus messages and timeouts for deterministic accountability

func (*ConsensusState) ReplayFile

func (cs *ConsensusState) ReplayFile(file string, console bool) error

Replay msgs in file or start the console

func (*ConsensusState) SetEventSwitch added in v0.7.0

func (cs *ConsensusState) SetEventSwitch(evsw types.EventSwitch)

SetEventSwitch implements events.Eventable

func (*ConsensusState) SetLogger

func (cs *ConsensusState) SetLogger(l log.Logger)

SetLogger implements Service.

func (*ConsensusState) SetPrivValidator

func (cs *ConsensusState) SetPrivValidator(priv PrivValidator)

SetPrivValidator sets the private validator account for signing votes.

func (*ConsensusState) SetProposal

func (cs *ConsensusState) SetProposal(proposal *types.Proposal, peerKey string) error

SetProposal inputs a proposal.

func (*ConsensusState) SetProposalAndBlock

func (cs *ConsensusState) SetProposalAndBlock(proposal *types.Proposal, block *types.Block, parts *types.PartSet, peerKey string) error

SetProposalAndBlock inputs the proposal and all block parts.

func (*ConsensusState) SetTimeoutTicker

func (cs *ConsensusState) SetTimeoutTicker(timeoutTicker TimeoutTicker)

SetTimeoutTicker sets the local timer. It may be useful to overwrite for testing.

func (*ConsensusState) String

func (cs *ConsensusState) String() string

String returns a string.

func (*ConsensusState) Wait

func (cs *ConsensusState) Wait()

Wait waits for the the main routine to return. NOTE: be sure to Stop() the event switch and drain any event channels or this may deadlock

type Handshaker

type Handshaker struct {
	// contains filtered or unexported fields
}

func NewHandshaker

func NewHandshaker(state *sm.State, store types.BlockStore) *Handshaker

func (*Handshaker) Handshake

func (h *Handshaker) Handshake(proxyApp proxy.AppConns) error

TODO: retry the handshake/replay if it fails ?

func (*Handshaker) NBlocks

func (h *Handshaker) NBlocks() int

func (*Handshaker) ReplayBlocks

func (h *Handshaker) ReplayBlocks(appHash []byte, appBlockHeight int, proxyApp proxy.AppConns) ([]byte, error)

Replay all blocks since appBlockHeight and ensure the result matches the current state. Returns the final AppHash or an error

func (*Handshaker) SetLogger

func (h *Handshaker) SetLogger(l log.Logger)

type HasVoteMessage

type HasVoteMessage struct {
	Height int
	Round  int
	Type   byte
	Index  int
}

HasVoteMessage is sent to indicate that a particular vote has been received.

func (*HasVoteMessage) String

func (m *HasVoteMessage) String() string

String returns a string representation.

type HeightVoteSet added in v0.7.0

type HeightVoteSet struct {
	// contains filtered or unexported fields
}

Keeps track of all VoteSets from round 0 to round 'round'.

Also keeps track of up to one RoundVoteSet greater than 'round' from each peer, to facilitate catchup syncing of commits.

A commit is +2/3 precommits for a block at a round, but which round is not known in advance, so when a peer provides a precommit for a round greater than mtx.round, we create a new entry in roundVoteSets but also remember the peer to prevent abuse. We let each peer provide us with up to 2 unexpected "catchup" rounds. One for their LastCommit round, and another for the official commit round.

func NewHeightVoteSet added in v0.7.0

func NewHeightVoteSet(chainID string, height int, valSet *types.ValidatorSet) *HeightVoteSet

func (*HeightVoteSet) AddVote added in v0.8.0

func (hvs *HeightVoteSet) AddVote(vote *types.Vote, peerKey string) (added bool, err error)

Duplicate votes return added=false, err=nil. By convention, peerKey is "" if origin is self.

func (*HeightVoteSet) Height added in v0.7.0

func (hvs *HeightVoteSet) Height() int

func (*HeightVoteSet) POLInfo added in v0.8.0

func (hvs *HeightVoteSet) POLInfo() (polRound int, polBlockID types.BlockID)

Last round and blockID that has +2/3 prevotes for a particular block or nil. Returns -1 if no such round exists.

func (*HeightVoteSet) Precommits added in v0.7.0

func (hvs *HeightVoteSet) Precommits(round int) *types.VoteSet

func (*HeightVoteSet) Prevotes added in v0.7.0

func (hvs *HeightVoteSet) Prevotes(round int) *types.VoteSet

func (*HeightVoteSet) Reset added in v0.7.0

func (hvs *HeightVoteSet) Reset(height int, valSet *types.ValidatorSet)

func (*HeightVoteSet) Round added in v0.7.0

func (hvs *HeightVoteSet) Round() int

func (*HeightVoteSet) SetPeerMaj23 added in v0.8.0

func (hvs *HeightVoteSet) SetPeerMaj23(round int, type_ byte, peerID string, blockID types.BlockID)

If a peer claims that it has 2/3 majority for given blockKey, call this. NOTE: if there are too many peers, or too much peer churn, this can cause memory issues. TODO: implement ability to remove peers too

func (*HeightVoteSet) SetRound added in v0.7.0

func (hvs *HeightVoteSet) SetRound(round int)

Create more RoundVoteSets up to round.

func (*HeightVoteSet) String added in v0.7.0

func (hvs *HeightVoteSet) String() string

func (*HeightVoteSet) StringIndented added in v0.7.0

func (hvs *HeightVoteSet) StringIndented(indent string) string

type NewRoundStepMessage

type NewRoundStepMessage struct {
	Height                int
	Round                 int
	Step                  RoundStepType
	SecondsSinceStartTime int
	LastCommitRound       int
}

NewRoundStepMessage is sent for every step taken in the ConsensusState. For every height/round/step transition

func (*NewRoundStepMessage) String

func (m *NewRoundStepMessage) String() string

String returns a string representation.

type PeerRoundState added in v0.7.0

type PeerRoundState struct {
	Height                   int                 // Height peer is at
	Round                    int                 // Round peer is at, -1 if unknown.
	Step                     RoundStepType       // Step peer is at
	StartTime                time.Time           // Estimated start of round 0 at this height
	Proposal                 bool                // True if peer has proposal for this round
	ProposalBlockPartsHeader types.PartSetHeader //
	ProposalBlockParts       *cmn.BitArray       //
	ProposalPOLRound         int                 // Proposal's POL round. -1 if none.
	ProposalPOL              *cmn.BitArray       // nil until ProposalPOLMessage received.
	Prevotes                 *cmn.BitArray       // All votes peer has for this round
	Precommits               *cmn.BitArray       // All precommits peer has for this round
	LastCommitRound          int                 // Round of commit for last height. -1 if none.
	LastCommit               *cmn.BitArray       // All commit precommits of commit for last height.
	CatchupCommitRound       int                 // Round that we have commit for. Not necessarily unique. -1 if none.
	CatchupCommit            *cmn.BitArray       // All commit precommits peer has for this height & CatchupCommitRound
}

PeerRoundState contains the known state of a peer. NOTE: Read-only when returned by PeerState.GetRoundState().

func (PeerRoundState) String added in v0.8.0

func (prs PeerRoundState) String() string

String returns a string representation of the PeerRoundState

func (PeerRoundState) StringIndented added in v0.8.0

func (prs PeerRoundState) StringIndented(indent string) string

StringIndented returns a string representation of the PeerRoundState

type PeerState

type PeerState struct {
	Peer *p2p.Peer

	PeerRoundState
	// contains filtered or unexported fields
}

PeerState contains the known state of a peer, including its connection and threadsafe access to its PeerRoundState.

func NewPeerState

func NewPeerState(peer *p2p.Peer) *PeerState

NewPeerState returns a new PeerState for the given Peer

func (*PeerState) ApplyCommitStepMessage added in v0.7.0

func (ps *PeerState) ApplyCommitStepMessage(msg *CommitStepMessage)

ApplyCommitStepMessage updates the peer state for the new commit.

func (*PeerState) ApplyHasVoteMessage

func (ps *PeerState) ApplyHasVoteMessage(msg *HasVoteMessage)

ApplyHasVoteMessage updates the peer state for the new vote.

func (*PeerState) ApplyNewRoundStepMessage

func (ps *PeerState) ApplyNewRoundStepMessage(msg *NewRoundStepMessage)

ApplyNewRoundStepMessage updates the peer state for the new round.

func (*PeerState) ApplyProposalPOLMessage

func (ps *PeerState) ApplyProposalPOLMessage(msg *ProposalPOLMessage)

ApplyProposalPOLMessage updates the peer state for the new proposal POL.

func (*PeerState) ApplyVoteSetBitsMessage

func (ps *PeerState) ApplyVoteSetBitsMessage(msg *VoteSetBitsMessage, ourVotes *cmn.BitArray)

ApplyVoteSetBitsMessage updates the peer state for the bit-array of votes it claims to have for the corresponding BlockID. `ourVotes` is a BitArray of votes we have for msg.BlockID NOTE: if ourVotes is nil (e.g. msg.Height < rs.Height), we conservatively overwrite ps's votes w/ msg.Votes.

func (*PeerState) EnsureVoteBitArrays

func (ps *PeerState) EnsureVoteBitArrays(height int, numValidators int)

EnsureVoteVitArrays ensures the bit-arrays have been allocated for tracking what votes this peer has received. NOTE: It's important to make sure that numValidators actually matches what the node sees as the number of validators for height.

func (*PeerState) GetHeight

func (ps *PeerState) GetHeight() int

GetHeight returns an atomic snapshot of the PeerRoundState's height used by the mempool to ensure peers are caught up before broadcasting new txs

func (*PeerState) GetRoundState

func (ps *PeerState) GetRoundState() *PeerRoundState

GetRoundState returns an atomic snapshot of the PeerRoundState. There's no point in mutating it since it won't change PeerState.

func (*PeerState) PickSendVote

func (ps *PeerState) PickSendVote(votes types.VoteSetReader) bool

PickSendVote picks a vote and sends it to the peer. Returns true if vote was sent.

func (*PeerState) PickVoteToSend

func (ps *PeerState) PickVoteToSend(votes types.VoteSetReader) (vote *types.Vote, ok bool)

PickVoteToSend picks a vote to send to the peer. Returns true if a vote was picked. NOTE: `votes` must be the correct Size() for the Height().

func (*PeerState) SetHasProposal

func (ps *PeerState) SetHasProposal(proposal *types.Proposal)

SetHasProposal sets the given proposal as known for the peer.

func (*PeerState) SetHasProposalBlockPart

func (ps *PeerState) SetHasProposalBlockPart(height int, round int, index int)

SetHasProposalBlockPart sets the given block part index as known for the peer.

func (*PeerState) SetHasVote

func (ps *PeerState) SetHasVote(vote *types.Vote)

SetHasVote sets the given vote as known by the peer

func (*PeerState) String

func (ps *PeerState) String() string

String returns a string representation of the PeerState

func (*PeerState) StringIndented

func (ps *PeerState) StringIndented(indent string) string

StringIndented returns a string representation of the PeerState

type PrivValidator added in v0.8.0

type PrivValidator interface {
	GetAddress() []byte
	SignVote(chainID string, vote *types.Vote) error
	SignProposal(chainID string, proposal *types.Proposal) error
	SignHeartbeat(chainID string, heartbeat *types.Heartbeat) error
}

PrivValidator is a validator that can sign votes and proposals.

type ProposalHeartbeatMessage added in v0.10.3

type ProposalHeartbeatMessage struct {
	Heartbeat *types.Heartbeat
}

ProposalHeartbeatMessage is sent to signal that a node is alive and waiting for transactions for a proposal.

func (*ProposalHeartbeatMessage) String added in v0.10.3

func (m *ProposalHeartbeatMessage) String() string

String returns a string representation.

type ProposalMessage

type ProposalMessage struct {
	Proposal *types.Proposal
}

ProposalMessage is sent when a new block is proposed.

func (*ProposalMessage) String

func (m *ProposalMessage) String() string

String returns a string representation.

type ProposalPOLMessage

type ProposalPOLMessage struct {
	Height           int
	ProposalPOLRound int
	ProposalPOL      *cmn.BitArray
}

ProposalPOLMessage is sent when a previous proposal is re-proposed.

func (*ProposalPOLMessage) String

func (m *ProposalPOLMessage) String() string

String returns a string representation.

type RoundState added in v0.7.0

type RoundState struct {
	Height             int // Height we are working on
	Round              int
	Step               RoundStepType
	StartTime          time.Time
	CommitTime         time.Time // Subjective time when +2/3 precommits for Block at Round were found
	Validators         *types.ValidatorSet
	Proposal           *types.Proposal
	ProposalBlock      *types.Block
	ProposalBlockParts *types.PartSet
	LockedRound        int
	LockedBlock        *types.Block
	LockedBlockParts   *types.PartSet
	Votes              *HeightVoteSet
	CommitRound        int            //
	LastCommit         *types.VoteSet // Last precommits at Height-1
	LastValidators     *types.ValidatorSet
}

RoundState defines the internal consensus state. It is Immutable when returned from ConsensusState.GetRoundState() TODO: Actually, only the top pointer is copied, so access to field pointers is still racey

func (*RoundState) RoundStateEvent added in v0.7.0

func (rs *RoundState) RoundStateEvent() types.EventDataRoundState

RoundStateEvent returns the H/R/S of the RoundState as an event.

func (*RoundState) String added in v0.7.0

func (rs *RoundState) String() string

String returns a string

func (*RoundState) StringIndented added in v0.7.0

func (rs *RoundState) StringIndented(indent string) string

StringIndented returns a string

func (*RoundState) StringShort added in v0.7.0

func (rs *RoundState) StringShort() string

StringShort returns a string

type RoundStepType added in v0.7.0

type RoundStepType uint8 // These must be numeric, ordered.

RoundStepType enumerates the state of the consensus state machine

func (RoundStepType) String added in v0.7.0

func (rs RoundStepType) String() string

String returns a string

type RoundVoteSet added in v0.7.0

type RoundVoteSet struct {
	Prevotes   *types.VoteSet
	Precommits *types.VoteSet
}

type TimedWALMessage

type TimedWALMessage struct {
	Time time.Time  `json:"time"`
	Msg  WALMessage `json:"msg"`
}

type TimeoutTicker

type TimeoutTicker interface {
	Start() (bool, error)
	Stop() bool
	Chan() <-chan timeoutInfo       // on which to receive a timeout
	ScheduleTimeout(ti timeoutInfo) // reset the timer

	SetLogger(log.Logger)
}

TimeoutTicker is a timer that schedules timeouts conditional on the height/round/step in the timeoutInfo. The timeoutInfo.Duration may be non-positive.

func NewTimeoutTicker

func NewTimeoutTicker() TimeoutTicker

NewTimeoutTicker returns a new TimeoutTicker.

type VoteMessage

type VoteMessage struct {
	Vote *types.Vote
}

VoteMessage is sent when voting for a proposal (or lack thereof).

func (*VoteMessage) String

func (m *VoteMessage) String() string

String returns a string representation.

type VoteSetBitsMessage

type VoteSetBitsMessage struct {
	Height  int
	Round   int
	Type    byte
	BlockID types.BlockID
	Votes   *cmn.BitArray
}

VoteSetBitsMessage is sent to communicate the bit-array of votes seen for the BlockID.

func (*VoteSetBitsMessage) String

func (m *VoteSetBitsMessage) String() string

String returns a string representation.

type VoteSetMaj23Message

type VoteSetMaj23Message struct {
	Height  int
	Round   int
	Type    byte
	BlockID types.BlockID
}

VoteSetMaj23Message is sent to indicate that a given BlockID has seen +2/3 votes.

func (*VoteSetMaj23Message) String

func (m *VoteSetMaj23Message) String() string

String returns a string representation.

type WAL

type WAL struct {
	BaseService
	// contains filtered or unexported fields
}

Write ahead logger writes msgs to disk before they are processed. Can be used for crash-recovery and deterministic replay TODO: currently the wal is overwritten during replay catchup

give it a mode so it's either reading or appending - must read to end to start appending again

func NewWAL

func NewWAL(walFile string, light bool) (*WAL, error)

func (*WAL) OnStart added in v0.8.0

func (wal *WAL) OnStart() error

func (*WAL) OnStop added in v0.8.0

func (wal *WAL) OnStop()

func (*WAL) Save added in v0.7.0

func (wal *WAL) Save(wmsg WALMessage)

called in newStep and for each pass in receiveRoutine

type WALMessage

type WALMessage interface{}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL