Documentation
¶
Overview ¶
Package triegen implements a code generator for a trie for associating unsigned integer values with UTF-8 encoded runes.
Many of the go.text packages use tries for storing per-rune information. A trie is especially useful if many of the runes have the same value. If this is the case, many blocks can be expected to be shared allowing for information on many runes to be stored in little space.
As most of the lookups are done directly on []byte slices, the tries use the UTF-8 bytes directly for the lookup. This saves a conversion from UTF-8 to runes and contributes a little bit to better performance. It also naturally provides a fast path for ASCII.
Space is also an issue. There are many code points defined in Unicode and as a result tables can get quite large. So every byte counts. The triegen package automatically chooses the smallest integer values to represent the tables. Compacters allow further compression of the trie by allowing for alternative representations of individual trie blocks.
triegen allows generating multiple tries as a single structure. This is useful when, for example, one wants to generate tries for several languages that have a lot of values in common. Some existing libraries for internationalization store all per-language data as a dynamically loadable chunk. The go.text packages are designed with the assumption that the user typically wants to compile in support for all supported languages, in line with the approach common to Go to create a single standalone binary. The multi-root trie approach can give significant storage savings in this scenario.
triegen generates both tables and code. The code is optimized to use the automatically chosen data types. The following code is generated for a Trie or multiple Tries named "foo":
type fooTrie The trie type.
func newFooTrie(x int) *fooTrie Trie constructor, where x is the index of the trie passed to Gen.
func (t *fooTrie) lookup(s []byte) (v uintX, sz int) The lookup method, where uintX is automatically chosen.
func lookupString, lookupUnsafe and lookupStringUnsafe Variants of the above.
var fooValues and fooIndex and any tables generated by Compacters. The core trie data.
var fooTrieHandles Indexes of starter blocks in case of multiple trie roots.
It is recommended that users test the generated trie by checking the returned value for every rune. Such exhaustive tests are possible as the number of runes in Unicode is limited.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Gen ¶
Gen writes Go code for a shared trie lookup structure to w for the given Tries. The generated trie type will be called nameTrie. newNameTrie(x) will return the *nameTrie for tries[x]. A value can be looked up by using one of the various lookup methods defined on nameTrie. It returns the table size of the generated trie.
Types ¶
type Compacter ¶
type Compacter interface {
// Size returns whether the Compacter could encode the given block as well
// as its size in case it can. len(v) is always 64.
Size(v []uint64) (sz int, ok bool)
// Store stores the block using the Compacter's compression method.
// It returns a handle with which the block can be retrieved.
// len(v) is always 64.
Store(v []uint64) uint32
// Print writes the data structures associated to the given store to w.
Print(w io.Writer) error
// Handler returns the name of a function that gets called during trie
// lookup for blocks generated by the Compacter. The function should be of
// the form func (n uint32, b byte) uint64, where n is the index returned by
// the Compacter's Store method and b is the last byte of the UTF-8
// encoding, where 0x80 <= b < 0xC0, for which to do the lookup in the
// block.
Handler() string
}
A Compacter generates an alternative, more space-efficient way to store a trie value block. A trie value block holds all possible values for the last byte of a UTF-8 encoded rune. Excluding ASCII characters, a trie value block always has 64 values, as a UTF-8 encoding ends with a byte in [0x80, 0xC0).
type Trie ¶
type Trie struct {
// contains filtered or unexported fields
}
A Trie represents a single root node of a trie. A builder may build several overlapping tries at once.
Source Files