fsfffb

README

FS (Fast & Slow) FFFB Inhibition

FFFB is the feedforward (FF) and feedback (FB) inhibition mechanism, originally developed for the Leabra model. It applies inhibition as a function of the average Ge (excitatory conductance) of units in the layer (this is FF = reflecting all the excitatory input to a layer) and average Act rate-code-like activation within the layer (which is FB = reflecting the activity level within the layer itself). Act is slowly integrated over time as a function of the ISI (inter-spike-interval).

FS-FFFB is a fast and slow (FS) version of FFFB that works directly with spike input signals instead of using "internal" variables like Ge and Act, and uses time dynamics based on an emerging consensus about the differences between three major classes of inhibitory interneurons, and their functional properties, e.g., Cardin, 2018.

• PV: fast-spiking basket cells that target the cell bodies of excitatory neurons and coexpress the calcium-binding protein parvalbumin (PV). These are the "first responders", and are also rapidly depressing -- they provide quick control of activity, responding to FF new input and FB drive, allowing the first spiking pyramidal neurons to quickly shut off other competitors, and maintain a sparse overall level of activity. The PV activity level (and consequent inhibitory conductance into pyramidal cells, Gi) closely tracks the incoming excitatory conductance Ge, which keeps neurons in their sensitive dynamic range e.g., Shadlen & Newsome, 1994.

• SST: more slowly-responding, higher-threshold spiking cells that target the distal dendrites of excitatory neurons and coexpress the peptide somatostatin (SST). These require repetitive, facilitating, afferent input to be activated, and may regulate the dendritic integration of synaptic inputs over a longer timescale. The dependence in the original FFFB of FB on the slower integrated Act variable, which only comes on after the first spike (in order to compute the ISI), is consistent with these slower SST dynamics.

• VIP: sparse dendrite-targeting cells that synapse onto SST interneurons and the dendrites of pyramidal neurons, and coexpress vasoactive intestinal peptide (VIP). VIP interneurons are a subset of the larger 5HT3aR-expressing interneuron class. These can provide disinhibition of SST inhibition. These are targeted by thalamic projections into layer 1 of cortex, and may be responsible for arousal and gating-like modulation from the thalamus. We do not directly implement them in axon, but do indirectly capture their effects in the gating dynamics of the pcore model.

Remarkably, the parameters that work to enable successful error-driven learning in axon using the new FS-FFFB equations end up very closely reproducing the net inhibition produced by the original FFFB model, as shown in the following figure.

Figure 1: Comparison of original FFFB inhibition (OGi) vs. new FS-FFFB inhibition (SGi) from the inhib example simulation, showing that the FS-FFFB parameters that enable successful learning produce nearly identical overall levels of inhibition compared to the original. Act.Avg shows the time-averaged activity in the lower layer that feeds into the one shown, and resembles the slow SST response, while the jagged ups and downs are due to the fast PV component.

PV is fast and highly chaotic

The dramatic swings in Gi levels as shown in the above figure are all due to the PV Fast component, which increases directly as a function of incoming FF and FB spikes, and decays with a fast time constant of 6 msec (default):

	FSi += (FFs + FB*FBs) - FSi/FSTau

where:

• FSi = fast PV contribution to inhibition, time-integrated.
• FFs = normalized sum of incoming FF spikes.
• FBs = normalized sum of incoming FB spikes from neurons in the same pool being inhibited.
• FB = weighting factor for FB spikes, which defaults to 1 but needs to be smaller for smaller networks (0.5) and larger for larger ones (e.g., 4).
• FSTau = time constant for decaying FSi (6 msec default).

SST is a slow time-average

The slow SST contribution slowly tracks overall spiking activity in the pool, roughly as the Act.Avg green line in the above figure, based on the following equations:

	SSi += (SSf*FBs - SSi) / SSiTau
	SSf += FBs*(1-SSf) - SSf / SSfTau

where:

• SSi = slow SST contribution to inhibition, time-integrated.
• SSf = synaptic facilitation component for SS, which increases as a function of spiking activity as shown in the 2nd equation.
• FBs = normalized sum of incoming FB spikes from neurons in the same pool being inhibited.
• SSiTau = integration time constant for SSi, which is 50 msec by default (slow).
• SSfTau = time constant for SSf, which is 20 msec by default.

Combined Gi

The combined overall inhibitory conductance Gi is based on a thresholded version of the FSi component plus a weighted contribution of the SSi level, which tends to be very weak due to the long time integral:

    Gi = |FSi > FS0|_+ + SS * SSi

where:

• Gi = overall inhibitory conductance.
• FSi = fast-spiking inhibition per above.
• SSi = slow-spiking inhibition per above.
• FS0 = threshold for FSi, default .1 as in the original FFFB, below which it contributes 0. This factor is important for filtering out small levels of incoming spikes and produces an observed nonlinearity in the Gi response.
• SS = multiplier for SSi, which is 30 by default: SSi is relatively weak so this needs to be a strong multiplier to get into the range of FSi.

Additional SSGi -> VmDend

In addition to the pooled Gi inhibition value described above, the slow inhibition value SSi is applied with a separate weighting factor to the dendritic membrane potential update, as an additional inhibitory component, weighted by the Act.Dend.SSGi parameter (default = 2). This is important for specifically balancing the positive feedback loop in increasing NMDA channel activation, which is voltage gated as a function of VmDend. Over the course of learning VmDend tends to increase for the most active neurons, and this additional SSGi factor balances that and prevents it from entering into a runaway positive feedback loop.

Biologically, the idea is that SSi inhibition affects the overall cellular Vm, but because it comes directly into the distal dendrites, it has an extra impact there.

References

• Cardin, J. A. (2018). Inhibitory interneurons regulate temporal precision and correlations in cortical circuits. Trends in Neurosciences, 41(10), 689–700. https://doi.org/10.1016/j.tins.2018.07.015

• Shadlen, M. N., & Newsome, W. T. (1994). Noise, neural codes and cortical organization. Current Opinion in Neurobiology, 4, 569–579. http://www.ncbi.nlm.nih.gov/pubmed/7812147

Documentation

Overview

Package fsfffb provides Fast and Slow feedforward (FF) and feedback (FB) inhibition (FFFB) based on incoming spikes (FF) and outgoing spikes (FB).

This produces a robust, graded k-Winners-Take-All dynamic of sparse distributed representations having approximately k out of N neurons active at any time, where k is typically 10-20 percent of N.

Index

• type GiParams
  • func (fb *GiParams) Defaults()
  • func (fb *GiParams) FS(fsi, gext float32, clamped bool) float32
  • func (fb *GiParams) FS0Thr(val float32) float32
  • func (fb *GiParams) FSiFmFFs(fsi *float32, ffs, fbs float32)
  • func (fb *GiParams) Inhib(inh *Inhib, gimult float32)
  • func (fb *GiParams) SSFmFBs(ssf, ssi *float32, fbs float32)
  • func (fb *GiParams) Update()
• type Inhib
  • func (fi *Inhib) Decay(decay float32)
  • func (fi *Inhib) FloatFmIntFactor() float32
  • func (fi *Inhib) FloatFromInt(ival int32) float32
  • func (fi *Inhib) FloatToInt(val float32, nneurons int) int32
  • func (fi *Inhib) FloatToIntFactor() float32
  • func (fi *Inhib) GiFmFSSS() float32
  • func (fi *Inhib) Init()
  • func (fi *Inhib) InitRaw()
  • func (fi *Inhib) IntToRaw()
  • func (fi *Inhib) LayerMax(liGi float32)
  • func (fi *Inhib) PoolMax(piGi float32)
  • func (fi *Inhib) RawIncr(spike, geRaw, geExt float32, nneurons int)
  • func (fi *Inhib) RawIncrInt(spike, geRaw, geExt float32, nneurons int)
  • func (fi *Inhib) SaveOrig()
  • func (fi *Inhib) SpikesFmRaw(nneurons int)
  • func (fi *Inhib) Zero()
• type Inhibs

Types

type GiParams

type GiParams struct {

	// enable this level of inhibition
	On slbool.Bool `desc:"enable this level of inhibition"`

	// [def: 1,1.1,0.75,0.9] [viewif: On] [min: 0] [0.8-1.5 typical, can go lower or higher as needed] overall inhibition gain -- this is main parameter to adjust to change overall activation levels -- it scales both the the FS and SS factors uniformly
	Gi float32 `` /* 249-byte string literal not displayed */

	// [def: 0.5,1,4] [viewif: On] [min: 0] amount of FB spikes included in FF for driving FS -- for small networks, 0.5 or 1 works best; larger networks and more demanding inhibition requires higher levels.
	FB float32 `` /* 204-byte string literal not displayed */

	// [def: 6] [viewif: On] [min: 0] fast spiking (PV+) intgration time constant in cycles (msec) -- tau is roughly how long it takes for value to change significantly -- 1.4x the half-life.
	FSTau float32 `` /* 188-byte string literal not displayed */

	// [def: 30] [viewif: On] [min: 0] multiplier on SS slow-spiking (SST+) in contributing to the overall Gi inhibition -- FS contributes at a factor of 1
	SS float32 `` /* 152-byte string literal not displayed */

	// [def: 20] [viewif: On] [min: 0] slow-spiking (SST+) facilitation decay time constant in cycles (msec) -- facilication factor SSf determines impact of FB spikes as a function of spike input-- tau is roughly how long it takes for value to change significantly -- 1.4x the half-life.
	SSfTau float32 `` /* 284-byte string literal not displayed */

	// [def: 50] [viewif: On] [min: 0] slow-spiking (SST+) intgration time constant in cycles (msec) cascaded on top of FSTau -- tau is roughly how long it takes for value to change significantly -- 1.4x the half-life.
	SSiTau float32 `` /* 215-byte string literal not displayed */

	// [def: 0.1] [viewif: On] fast spiking zero point -- below this level, no FS inhibition is computed, and this value is subtracted from the FSi
	FS0 float32 `` /* 145-byte string literal not displayed */

	// [def: 50] [viewif: On] time constant for updating a running average of the feedforward inhibition over a longer time scale, for computing FFPrv
	FFAvgTau float32 `` /* 148-byte string literal not displayed */

	// [def: 0] [viewif: On] proportion of previous average feed-forward inhibition (FFAvgPrv) to add, resulting in an accentuated temporal-derivative dynamic where neurons respond most strongly to increases in excitation that exceeds inhibition from last time.
	FFPrv float32 `` /* 259-byte string literal not displayed */

	// [def: 0.05] [viewif: On] minimum GeExt value required to drive external clamping dynamics (if clamp is set), where only GeExt drives inhibition.  If GeExt is below this value, then the usual FS-FFFB drivers are used.
	ClampExtMin float32 `` /* 221-byte string literal not displayed */

	// [view: -] rate = 1 / tau
	FSDt float32 `inactive:"+" view:"-" json:"-" xml:"-" desc:"rate = 1 / tau"`

	// [view: -] rate = 1 / tau
	SSfDt float32 `inactive:"+" view:"-" json:"-" xml:"-" desc:"rate = 1 / tau"`

	// [view: -] rate = 1 / tau
	SSiDt float32 `inactive:"+" view:"-" json:"-" xml:"-" desc:"rate = 1 / tau"`

	// [view: -] rate = 1 / tau
	FFAvgDt float32 `inactive:"+" view:"-" json:"-" xml:"-" desc:"rate = 1 / tau"`
	// contains filtered or unexported fields
}

GiParams parameterizes feedforward (FF) and feedback (FB) inhibition (FFFB) based on incoming spikes (FF) and outgoing spikes (FB) across Fast (PV+) and Slow (SST+) timescales. FF -> PV -> FS fast spikes, FB -> SST -> SS slow spikes (slow to get going)

func (*GiParams) Defaults

func (fb *GiParams) Defaults()

func (*GiParams) FS

func (fb *GiParams) FS(fsi, gext float32, clamped bool) float32

FS returns the current effective FS value based on fsi and fsd if clamped, then only use gext, without applying FS0

func (*GiParams) FS0Thr

func (fb *GiParams) FS0Thr(val float32) float32

FS0Thr applies FS0 threshold to given value

func (*GiParams) FSiFmFFs

func (fb *GiParams) FSiFmFFs(fsi *float32, ffs, fbs float32)

FSiFmFFs updates fast-spiking inhibition from FFs spikes

func (*GiParams) Inhib

func (fb *GiParams) Inhib(inh *Inhib, gimult float32)

Inhib is full inhibition computation for given inhib state which has aggregated FFs and FBs spiking values

func (*GiParams) SSFmFBs

func (fb *GiParams) SSFmFBs(ssf, ssi *float32, fbs float32)

SSFmFBs updates slow-spiking inhibition from FBs

func (*GiParams) Update

func (fb *GiParams) Update()

type Inhib

type Inhib struct {

	// all feedforward incoming spikes into neurons in this pool -- raw aggregation
	FFsRaw float32 `desc:"all feedforward incoming spikes into neurons in this pool -- raw aggregation"`

	// all feedback outgoing spikes generated from neurons in this pool -- raw aggregation
	FBsRaw float32 `desc:"all feedback outgoing spikes generated from neurons in this pool -- raw aggregation"`

	// all extra GeExt conductances added to neurons
	GeExtRaw float32 `desc:"all extra GeExt conductances added to neurons"`

	// all feedforward incoming spikes into neurons in this pool, normalized by pool size
	FFs float32 `desc:"all feedforward incoming spikes into neurons in this pool, normalized by pool size"`

	// all feedback outgoing spikes generated from neurons in this pool, normalized by pool size
	FBs float32 `desc:"all feedback outgoing spikes generated from neurons in this pool, normalized by pool size"`

	// all extra GeExt conductances added to neurons, normalized by pool size
	GeExts float32 `desc:"all extra GeExt conductances added to neurons, normalized by pool size"`

	// if true, this layer is hard-clamped and should use GeExts exclusively for PV
	Clamped slbool.Bool `desc:"if true, this layer is hard-clamped and should use GeExts exclusively for PV"`

	// fast spiking PV+ fast integration of FFs feedforward spikes
	FSi float32 `desc:"fast spiking PV+ fast integration of FFs feedforward spikes"`

	// slow spiking SST+ integration of FBs feedback spikes
	SSi float32 `desc:"slow spiking SST+ integration of FBs feedback spikes"`

	// slow spiking facilitation factor, representing facilitating effects of recent activity
	SSf float32 `desc:"slow spiking facilitation factor, representing facilitating effects of recent activity"`

	// overall fast-spiking inhibitory conductance
	FSGi float32 `desc:"overall fast-spiking inhibitory conductance"`

	// overall slow-spiking inhibitory conductance
	SSGi float32 `desc:"overall slow-spiking inhibitory conductance"`

	// overall inhibitory conductance = FSGi + SSGi
	Gi float32 `desc:"overall inhibitory conductance = FSGi + SSGi"`

	// original value of the inhibition (before pool or other effects)
	GiOrig float32 `desc:"original value of the inhibition (before pool or other effects)"`

	// for pools, this is the layer-level inhibition that is MAX'd with the pool-level inhibition to produce the net inhibition
	LayGi float32 `` /* 127-byte string literal not displayed */

	// longer time scale running average FF drive -- used for FFAvgPrv
	FFAvg float32 `desc:"longer time scale running average FF drive -- used for FFAvgPrv"`

	// previous theta cycle FFAvg value -- for FFPrv factor -- updated in Decay function that is called at start of new ThetaCycle
	FFAvgPrv float32 `` /* 130-byte string literal not displayed */

	// int32 atomic add compatible integration of FFsRaw
	FFsRawInt int32 `desc:"int32 atomic add compatible integration of FFsRaw"`

	// int32 atomic add compatible integration of FBsRaw
	FBsRawInt int32 `desc:"int32 atomic add compatible integration of FBsRaw"`

	// int32 atomic add compatible integration of GeExtRaw
	GeExtRawInt int32 `desc:"int32 atomic add compatible integration of GeExtRaw"`
}

Inhib contains state values for computed FFFB inhibition

func (*Inhib) Decay

func (fi *Inhib) Decay(decay float32)

Decay reduces inhibition values by given decay proportion

func (*Inhib) FloatFmIntFactor

func (fi *Inhib) FloatFmIntFactor() float32

FloatFmIntFactor returns the factor used for converting int32 back to float32 -- this is 1 / FloatToIntFactor for faster multiplication instead of dividing.

func (*Inhib) FloatFromInt

func (fi *Inhib) FloatFromInt(ival int32) float32

FloatFromInt converts the given int32 value produced via FloatToInt back into a float32 (divides by factor)

func (*Inhib) FloatToInt

func (fi *Inhib) FloatToInt(val float32, nneurons int) int32

FloatToInt converts the given floating point value to a large int for max updating.

func (*Inhib) FloatToIntFactor

func (fi *Inhib) FloatToIntFactor() float32

FloatToIntFactor returns the factor used for converting float32 to int32 for summing, assuming that the overall value is in the general order of 0-1 (512 is the max).

func (*Inhib) GiFmFSSS

func (fi *Inhib) GiFmFSSS() float32

GiFmFSSS returns the sum of FSGi and SSGi as overall inhibition

func (*Inhib) Init

func (fi *Inhib) Init()

func (*Inhib) InitRaw

func (fi *Inhib) InitRaw()

InitRaw clears raw spike counters -- done every cycle prior to accumulating

func (*Inhib) IntToRaw

func (fi *Inhib) IntToRaw()

IntToRaw computes int values into float32 raw values

func (*Inhib) LayerMax

func (fi *Inhib) LayerMax(liGi float32)

LayerMax updates given pool-level inhib values from given layer-level Gi with resulting value being the Max of either

func (*Inhib) PoolMax

func (fi *Inhib) PoolMax(piGi float32)

PoolMax updates given layer-level inhib values from given pool-level with resulting value being the Max of either

func (*Inhib) RawIncr

func (fi *Inhib) RawIncr(spike, geRaw, geExt float32, nneurons int)

RawIncr increments raw values from given neuron-based input values

func (*Inhib) RawIncrInt

func (fi *Inhib) RawIncrInt(spike, geRaw, geExt float32, nneurons int)

RawIncrInt increments raw values from given neuron-based input values for the int-based values (typically use Atomic InterlockedAdd instead)

func (*Inhib) SaveOrig

func (fi *Inhib) SaveOrig()

SaveOrig saves the current Gi values as original values

func (*Inhib) SpikesFmRaw

func (fi *Inhib) SpikesFmRaw(nneurons int)

SpikesFmRaw updates spike values from raw, dividing by given number in pool

func (*Inhib) Zero

func (fi *Inhib) Zero()

Zero resets all accumulating inhibition factors to 0

type Inhibs

type Inhibs []Inhib

Inhibs is a slice of Inhib records