Documentation
¶
Overview ¶
Package rla provides an implementation of RLA (Recurrent Linear Attention). See: "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention" by Katharopoulos et al., 2020. TODO: support arbitrary mapping functions
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
InputSize int
}
Config provides configuration settings for a RLA Model.
type Model ¶
type Model struct { nn.BaseModel Config Wk nn.Param `spago:"type:weights"` Bk nn.Param `spago:"type:biases"` Wv nn.Param `spago:"type:weights"` Bv nn.Param `spago:"type:biases"` Wq nn.Param `spago:"type:weights"` Bq nn.Param `spago:"type:biases"` States []*State `spago:"scope:processor"` }
Model contains the serializable parameters for an RLA neural network.
func (*Model) Forward ¶
Forward performs the forward step for each input node and returns the result.
func (*Model) LastState ¶
LastState returns the last state of the recurrent network. It returns nil if there are no states.
func (*Model) SetInitialState ¶
SetInitialState sets the initial state of the recurrent network. It panics if one or more states are already present.