rc

package

v0.1.0 Latest Latest Go to latest Published: Dec 9, 2020 License: BSD-2-Clause Imports: 1 Imported by: 0

Details

This package contains built-in Residual Connections (RC).

This section is empty.

This section is empty.

func PostNorm(
	g *ag.Graph,
	f func(...ag.Node) []ag.Node,
	norm func(...ag.Node) []ag.Node,
	xs ...ag.Node,
) []ag.Node

PostNorm performs post-norm residual connections:

y = Norm(x + F(x))

func PreNorm(
	g *ag.Graph,
	f func(...ag.Node) []ag.Node,
	norm func(...ag.Node) []ag.Node,
	xs ...ag.Node,
) []ag.Node

PreNorm performs pre-norm residual connections:

y = x + F(Norm(x))

func ReZero(
	g *ag.Graph,
	f func(...ag.Node) []ag.Node,
	alpha ag.Node,
	xs ...ag.Node,
) []ag.Node

ReZero performs residual connections by rescaling the function with an alpha learnable parameter (Bachlechner et al., 2020):

y = x + alpha * F(x)

This section is empty.