token

Documentation

Index

• type Comment
• type Float
• type Identifier
  • func (i *Identifier) ComputeEntropy(probs map[rune]float64)
• type IdentifierType
  • func IdentifierTypes() []IdentifierType
  • func ParseIdentifierType(s string) IdentifierType
  • func (t IdentifierType) MarshalJSON() ([]byte, error)
  • func (t IdentifierType) String() string
  • func (t *IdentifierType) UnmarshalJSON(data []byte) error
• type Int
• type Position
  • func (pos Position) Col() int
  • func (pos Position) Row() int
• type String
  • func (s *String) ComputeEntropy(probs map[rune]float64)
  • func (s *String) LevenshteinDist() int

Types

type Comment

type Comment struct {
	Text string `json:"text"`
}

Comment records the entire text of a source code comment. It may contain newline characters.

type Float

type Float struct {
	Value float64 `json:"value"`
	Raw   string  `json:"raw"`
}

Float records a floating point literal occurring in source code.

type Identifier

type Identifier struct {
	Name    string         `json:"name"`
	Type    IdentifierType `json:"type"`
	Entropy float64        `json:"entropy"`
}

Identifier records some kind of user-defined symbol name in source code. Valid types of identifier are defined using IdentifierType.

func (*Identifier) ComputeEntropy

func (i *Identifier) ComputeEntropy(probs map[rune]float64)

ComputeEntropy computes the entropy of this identifier's name under the given character distribution, and sets its Entropy field to the resulting value.

type IdentifierType

type IdentifierType int

IdentifierType enumerates the possible types of a source code identifier, encountered during static analysis.

const (
	Unknown        IdentifierType = iota
	Function                      // function declaration / definition
	Variable                      // variable declaration / definition
	Parameter                     // parameters to functions, constructors, catch blocks
	Class                         // class declaration / definition
	Member                        // access/mutation of an object member
	Property                      // declaration of class property
	StatementLabel                // loop label
	Other                         // something the parser picked up that isn't accounted for above
)

func IdentifierTypes

func IdentifierTypes() []IdentifierType

func ParseIdentifierType

func ParseIdentifierType(s string) IdentifierType

func (IdentifierType) MarshalJSON

func (t IdentifierType) MarshalJSON() ([]byte, error)

MarshalJSON serializes this IdentifierType using its string representation

func (IdentifierType) String

func (t IdentifierType) String() string

func (*IdentifierType) UnmarshalJSON

func (t *IdentifierType) UnmarshalJSON(data []byte) error

UnmarshalJSON deserializes an IdentifierType serialized using MarshalJSON. If the supplied JSON contains an unrecognised name, the deserialised value is Unknown, and no error is returned.

type Int

type Int struct {
	Value int64  `json:"value"`
	Raw   string `json:"raw"`
}

Int records an integer literal occurring in source code. For languages without explicit integer types such as JavaScript, an Int literal is any numeric literal whose raw string representation in source code is parseable (with strconv.ParseInt) as an integer.

type Position

type Position [2]int

Position records the position of a source code token in terms of row and column in the original source file.

func (Position) Col

func (pos Position) Col() int

func (Position) Row

func (pos Position) Row() int

type String

type String struct {
	Value   string  `json:"value"`
	Raw     string  `json:"raw"`
	Entropy float64 `json:"entropy"`
}

String records a string literal occurring in the source code.

func (*String) ComputeEntropy

func (s *String) ComputeEntropy(probs map[rune]float64)

ComputeEntropy computes the entropy of this string literal's value under the given character distribution, and sets its Entropy field to the resulting value.

func (*String) LevenshteinDist

func (s *String) LevenshteinDist() int

LevenshteinDist computes the Levenshtein distance between the parsed and raw versions of this string literal. A character substitution is treated as deletion and insertion (2 operations).