adanorm

Documentation

Overview

Package adanorm implements the Adaptive Normalization (AdaNorm) method.

Reference: "Understanding and Improving Layer Normalization" by Jingjing Xu, Xu Sun, Zhiyuan Zhang,Guangxiang Zhao, Junyang Lin (2019). (https://papers.nips.cc/paper/8689-understanding-and-improving-layer-normalization.pdf)

Index

• type Model
  • func New(scale mat.Float) *Model
  • func (m *Model) Forward(xs ...ag.Node) []ag.Node
  • func (m *Model) InitProcessor()
  • func (m *Model) Mean(xs []ag.Node) []ag.Node
  • func (m *Model) StdDev(meanVectors []ag.Node, xs []ag.Node) []ag.Node

Types

type Model

type Model struct {
	nn.BaseModel
	Scale mat.Float
	// contains filtered or unexported fields
}

Model contains the scaling factor.

func New

func New(scale mat.Float) *Model

New returns a new model.

func (*Model) Forward

func (m *Model) Forward(xs ...ag.Node) []ag.Node

Forward performs the forward step for each input node and returns the result.

func (*Model) InitProcessor

func (m *Model) InitProcessor()

InitProcessor initializes constants needed by the Forward().

func (*Model) Mean

func (m *Model) Mean(xs []ag.Node) []ag.Node

Mean computes the mean of the input.

func (*Model) StdDev

func (m *Model) StdDev(meanVectors []ag.Node, xs []ag.Node) []ag.Node

StdDev computes the standard deviation of the input.