layernorm

Documentation

Overview

Package layernorm implements the Layer Normalization (LayerNorm) i method.

Reference: "Layer normalization" by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton (2016). (https://arxiv.org/pdf/1607.06450.pdf)

Index

• type Model
  • func New[T float.DType](size int, eps float64) *Model
  • func (m *Model) Forward(xs ...mat.Tensor) []mat.Tensor

Types

type Model

type Model struct {
	nn.Module
	W   *nn.Param
	B   *nn.Param
	Eps *nn.Buffer
}

Model contains the serializable parameters.

func New

func New[T float.DType](size int, eps float64) *Model

New returns a new model with parameters initialized to zeros.

func (*Model) Forward

func (m *Model) Forward(xs ...mat.Tensor) []mat.Tensor

Forward performs the forward step for each input node and returns the result. y = (x - E\[x\]) / sqrt(VAR\[x\] + [EPS]) * g + b