graph

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2022 License: MPL-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package graph contains the main API to the graph datastore.

Manager API

The main API is provided by a Manager object which can be created with the NewGraphManager() constructor function. The manager CRUD functionality for nodes and edges through store, fetch and remove functions. It also provides the basic traversal functionality which allos the traversal from one node to other nodes.

Node iterator

All available node keys in a partition of a given kind can be iterated by using a NodeKeyIterator. The manager can produce these with the NodeKeyIterator() function.

All nodes and edges in the datastore are indexed. The index can be queried using a IndexQuery object. The manager can produce these with the NodeIndexQuery() or EdgeIndexQuery function.

Transactions

A transaction is used to build up multiple store and delete tasks for the graph database. Nothing is written to the database before calling commit(). A transaction commit does an automatic rollback if an error occurs (except fatal disk write errors which might cause a panic).

A trans object can be created with the NewGraphTrans() function.

Rules

(Use with caution)

Graph rules provide automatic operations which help to keep the graph consistent. Rules trigger on global graph events. The rules SystemRuleDeleteNodeEdges and SystemRuleUpdateNodeStats are automatically loaded when a new Manager is created. See the code for further details.

Graph databases

A graph manager handles the graph storage and provides the API for the graph database. The storage is divided into several databases:

Main database

MainDB stores various meta information such as known node/edge kinds, attributes or version information.

Names database

Names can be encoded (into a number) or decoded (into a string) 

32 bit values for any given node attribute names
16 bit values for any given edge role names
16 bit values for any given edge kind names

Nodes database

Each node kind database stores: 

PrefixNSAttrs + node key -> [ ATTRS ]
(a list of attributes of a certain node)

PrefixNSAttr + node key + attr num -> value
(attribute value of a certain node)

PrefixNSSpecs + node key -> map[spec]<empty string>
(a lookup for available specs for a certain node)

PrefixNSEdge + node key + spec -> map[edge key]edgeinfo{other node key, other node kind}]
(connection from one node to another via a spec)

Edges database

Each edge kind database stores: 

PrefixNSAttrs + edge key -> [ ATTRS ]
(a list of attributes of a certain edge)

PrefixNSAttr + edge key + attr num -> value
(attribute value of a certain edge)

Index database

The text index managed by util/indexmanager.go. IndexQuery provides access to the full text search index.

Index

Constants

View Source 
const EventEdgeCreated = 0x04

EventEdgeCreated is thrown when an edge was created.

Parameters: partition of created edge, created edge

View Source 
const EventEdgeDelete = 0x0B

EventEdgeDelete is thrown before an edge is deleted.

Parameters: partition of deleted edge, key of edge to delete, kind of edge to delete

View Source 
const EventEdgeDeleted = 0x06

EventEdgeDeleted is thrown when an edge was deleted.

Parameters: partition of deleted edge, deleted edge

View Source 
const EventEdgeStore = 0x0A

EventEdgeStore is thrown before an edge is stored (always overwriting existing values).

Parameters: partition of stored edge, stored edge

View Source 
const EventEdgeUpdated = 0x05

EventEdgeUpdated is thrown when an edge was updated.

Parameters: partition of updated edge, updated edge, old edge

View Source 
const EventNodeCreated = 0x01

EventNodeCreated is thrown when a node was created.

Parameters: partition of created node, created node

View Source 
const EventNodeDelete = 0x09

EventNodeDelete is thrown before a node is deleted.

Parameters: partition of node to delete, key of node to delete, kind of node to delete

View Source 
const EventNodeDeleted = 0x03

EventNodeDeleted is thrown when a node was deleted.

Parameters: partition of deleted node, deleted node

View Source 
const EventNodeStore = 0x07

EventNodeStore is thrown before a node is stored (always overwriting existing values).

Parameters: partition of node to store, node to store

View Source 
const EventNodeUpdate = 0x08

EventNodeUpdate is thrown before a node is updated.

Parameters: partition of node to update, node to update

View Source 
const EventNodeUpdated = 0x02

EventNodeUpdated is thrown when a node was updated.

Parameters: partition of updated node, updated node, old node

View Source 
const MainDBEdgeAttrs = MainDBEntryPrefix + "eatt"

MainDBEdgeAttrs is the MainDB entry key for a list of edge attributes

View Source 
const MainDBEdgeCount = MainDBEntryPrefix + "ecnt"

MainDBEdgeCount is the MainDB entry key for an edge count

View Source 
const MainDBEdgeKinds = MainDBEntryPrefix + "edgekind"

MainDBEdgeKinds is the MainDB entry key for edge kind information

View Source 
const MainDBEntryPrefix = "\x02"

MainDBEntryPrefix is the prefix for entries stored in the main database

View Source 
const MainDBNodeAttrs = MainDBEntryPrefix + "natt"

MainDBNodeAttrs is the MainDB entry key for a list of node attributes

View Source 
const MainDBNodeCount = MainDBEntryPrefix + "ncnt"

MainDBNodeCount is the MainDB entry key for a node count

View Source 
const MainDBNodeEdges = MainDBEntryPrefix + "nrel"

MainDBNodeEdges is the MainDB entry key for a list of node relationships

View Source 
const MainDBNodeKinds = MainDBEntryPrefix + "nodekind"

MainDBNodeKinds is the MainDB entry key for node kind information

View Source 
const MainDBParts = MainDBEntryPrefix + "part"

MainDBParts is the MainDB entry key for partition information

View Source 
const MainDBVersion = MainDBEntryPrefix + "ver"

MainDBVersion is the MainDB entry key for version information

View Source 
const PrefixNSAttr = "\x02"

PrefixNSAttr is the prefix for storing the value of a node attribute

View Source 
const PrefixNSAttrs = "\x01"

PrefixNSAttrs is the prefix for storing attributes of a node

View Source 
const PrefixNSEdge = "\x04"

PrefixNSEdge is the prefix for storing a link from a node (and a spec) to an edge

View Source 
const PrefixNSSpecs = "\x03"

PrefixNSSpecs is the prefix for storing specs of edges related to a node

View Source 
const RootIDNodeHTree = 2

RootIDNodeHTree is the root ID for the HTree holding primary information

View Source 
const RootIDNodeHTreeSecond = 3

RootIDNodeHTreeSecond is the root ID for the HTree holding secondary information

View Source 
const StorageSuffixEdges = ".edges"

StorageSuffixEdges is the suffix for an edge storage

View Source 
const StorageSuffixEdgesIndex = ".edgeidx"

StorageSuffixEdgesIndex is the suffix for an edge index

View Source 
const StorageSuffixNodes = ".nodes"

StorageSuffixNodes is the suffix for a node storage

View Source 
const StorageSuffixNodesIndex = ".nodeidx"

StorageSuffixNodesIndex is the suffix for a node index

View Source 
const VERSION = 1

VERSION of the GraphManager

Variables

View Source 
var ErrEventHandled = errors.New("Event handled upstream")

ErrEventHandled is a special error which an event handler can return to notify the GraphManager that no further action is necessary. No error will be returned by the GraphManager operation.

Functions

func ExportPartition 

func ExportPartition(out io.Writer, part string, gm *Manager) error

ExportPartition dumps the contents of a partition to an io.Writer in JSON format: 

{
	nodes : [ { <attr> : <value> }, ... ]
	edges : [ { <attr> : <value> }, ... ]
}

func ImportPartition 

func ImportPartition(in io.Reader, part string, gm *Manager) error

ImportPartition imports the JSON contents of an io.Reader into a given partition. The following format is expected: 

{
	nodes : [ { <attr> : <value> }, ... ]
	edges : [ { <attr> : <value> }, ... ]
}

func IsFullSpec 

func IsFullSpec(spec string) bool

IsFullSpec is a function to determine if a given spec is a fully specified spec (i.e. all spec components are specified)

func SortDump 

func SortDump(in string) string

SortDump sorts a string result which was produced by ExportPartition. Do not use this for very large results. Panics if the input data is not valid.

Types

type IndexQuery 

type IndexQuery interface {

	/*
		LookupPhrase finds all nodes where an attribute contains a certain
		phrase. This call returns a list of node keys which contain the phrase
		at least once.
	*/
	LookupPhrase(attr, phrase string) ([]string, error)

	/*
		LookupWord finds all nodes where an attribute contains a certain word.
		This call returns a map which maps node key to a list of word positions.
	*/
	LookupWord(attr, word string) (map[string][]uint64, error)

	/*
		LookupValue finds all nodes where an attribute has a certain value.
		This call returns a list of node keys.
	*/
	LookupValue(attr, value string) ([]string, error)
}

IndexQuery models the interface to the full text search index.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager data structure

func NewGraphManager 

func NewGraphManager(gs graphstorage.Storage) *Manager

NewGraphManager returns a new GraphManager instance.

func (*Manager) EdgeAttrs 

func (gm *Manager) EdgeAttrs(kind string) []string

EdgeAttrs returns all possible edge attributes for a given edge kind.

func (*Manager) EdgeCount 

func (gm *Manager) EdgeCount(kind string) uint64

EdgeCount returns the edge count for a given edge kind.

func (*Manager) EdgeIndexQuery 

func (gm *Manager) EdgeIndexQuery(part string, kind string) (IndexQuery, error)

EdgeIndexQuery returns an object to query the full text search index for edges.

func (*Manager) EdgeKinds 

func (gm *Manager) EdgeKinds() []string

EdgeKinds returns all possible node kinds.

func (*Manager) FetchEdge 

func (gm *Manager) FetchEdge(part string, key string, kind string) (data.Edge, error)

FetchEdge fetches a single edge from a partition of the graph.

func (*Manager) FetchEdgePart 

func (gm *Manager) FetchEdgePart(part string, key string, kind string,
	attrs []string) (data.Edge, error)

FetchEdgePart fetches part of a single edge from a partition of the graph.

func (*Manager) FetchNode 

func (gm *Manager) FetchNode(part string, key string, kind string) (data.Node, error)

FetchNode fetches a single node from a partition of the graph.

func (*Manager) FetchNodeEdgeSpecs 

func (gm *Manager) FetchNodeEdgeSpecs(part string, key string, kind string) ([]string, error)

FetchNodeEdgeSpecs returns all possible edge specs for a certain node.

func (*Manager) FetchNodePart 

func (gm *Manager) FetchNodePart(part string, key string, kind string,
	attrs []string) (data.Node, error)

FetchNodePart fetches part of a single node from a partition of the graph.

func (*Manager) GraphRules 

func (gm *Manager) GraphRules() []string

GraphRules returns a list of all available graph rules.

func (*Manager) IsValidAttr 

func (gm *Manager) IsValidAttr(attr string) bool

IsValidAttr checks if a given string can be a valid node attribute.

func (*Manager) Name 

func (gm *Manager) Name() string

Name returns the name of this graph manager.

func (*Manager) NodeAttrs 

func (gm *Manager) NodeAttrs(kind string) []string

NodeAttrs returns all possible node attributes for a given node kind.

func (*Manager) NodeCount 

func (gm *Manager) NodeCount(kind string) uint64

NodeCount returns the node count for a given node kind.

func (*Manager) NodeEdges 

func (gm *Manager) NodeEdges(kind string) []string

NodeEdges returns all possible node edge specs for a given node kind.

func (*Manager) NodeIndexQuery 

func (gm *Manager) NodeIndexQuery(part string, kind string) (IndexQuery, error)

NodeIndexQuery returns an object to query the full text search index for nodes.

func (*Manager) NodeKeyIterator 

func (gm *Manager) NodeKeyIterator(part string, kind string) (*NodeKeyIterator, error)

NodeKeyIterator iterates node keys of a certain kind.

func (*Manager) NodeKinds 

func (gm *Manager) NodeKinds() []string

NodeKinds returns all possible node kinds.

func (*Manager) Partitions 

func (gm *Manager) Partitions() []string

Partitions returns all existing partitions.

func (*Manager) RemoveEdge 

func (gm *Manager) RemoveEdge(part string, key string, kind string) (data.Edge, error)

RemoveEdge removes a single edge from a partition of the graph.

func (*Manager) RemoveNode 

func (gm *Manager) RemoveNode(part string, key string, kind string) (data.Node, error)

RemoveNode removes a single node from a partition of the graph.

func (*Manager) SetGraphRule 

func (gm *Manager) SetGraphRule(rule Rule)

SetGraphRule sets a GraphRule.

func (*Manager) StoreEdge 

func (gm *Manager) StoreEdge(part string, edge data.Edge) error

StoreEdge stores a single edge in a partition of the graph. This function will overwrites any existing edge.

func (*Manager) StoreNode 

func (gm *Manager) StoreNode(part string, node data.Node) error

StoreNode stores a single node in a partition of the graph. This function will overwrites any existing node.

func (*Manager) Traverse 

func (gm *Manager) Traverse(part string, key string, kind string,
	spec string, allData bool) ([]data.Node, []data.Edge, error)

Traverse traverses from a given node to other nodes following a given edge spec. The last parameter allData specifies if all data should be retrieved for the connected nodes and edges. If set to false only the minimal set of attributes will be populated.

func (*Manager) TraverseMulti 

func (gm *Manager) TraverseMulti(part string, key string, kind string,
	spec string, allData bool) ([]data.Node, []data.Edge, error)

TraverseMulti traverses from a given node to other nodes following a given partial edge spec. Since the edge spec can be partial it is possible to traverse multiple edge kinds. A spec with the value ":::" would follow all relationships. The last parameter allData specifies if all data should be retrieved for the connected nodes and edges. If set to false only the minimal set of attributes will be populated.

func (*Manager) UpdateNode 

func (gm *Manager) UpdateNode(part string, node data.Node) error

UpdateNode updates a single node in a partition of the graph. This function will only update the given values of the node.

type NodeKeyIterator 

type NodeKeyIterator struct {
	LastError error // Last encountered error
	// contains filtered or unexported fields
}

NodeKeyIterator can be used to iterate node keys of a certain node kind.

func (*NodeKeyIterator) Error 

func (it *NodeKeyIterator) Error() error

Error returns the last encountered error.

func (*NodeKeyIterator) HasNext 

func (it *NodeKeyIterator) HasNext() bool

HasNext returns if there is a next node key.

func (*NodeKeyIterator) Next 

func (it *NodeKeyIterator) Next() string

Next returns the next node key. Sets the LastError attribute if an error occurs.

type Rule 

type Rule interface {

	/*
	   Name returns the name of the rule.
	*/
	Name() string

	/*
		Handles returns a list of events which are handled by this rule.
	*/
	Handles() []int

	/*
		Handle handles an event. The function should write all changes to the
		given transaction.
	*/
	Handle(gm *Manager, trans Trans, event int, data ...interface{}) error
}

Rule models a graph rule.

type SystemRuleDeleteNodeEdges 

type SystemRuleDeleteNodeEdges struct {
}

SystemRuleDeleteNodeEdges is a system rule to delete all edges when a node is deleted. Deletes also the other end if the cascading flag is set on the edge.

func (*SystemRuleDeleteNodeEdges) Handle 

func (r *SystemRuleDeleteNodeEdges) Handle(gm *Manager, trans Trans, event int, ed ...interface{}) error

Handle handles an event.

func (*SystemRuleDeleteNodeEdges) Handles 

func (r *SystemRuleDeleteNodeEdges) Handles() []int

Handles returns a list of events which are handled by this rule.

func (*SystemRuleDeleteNodeEdges) Name 

func (r *SystemRuleDeleteNodeEdges) Name() string

Name returns the name of the rule.

type SystemRuleUpdateNodeStats 

type SystemRuleUpdateNodeStats struct {
}

SystemRuleUpdateNodeStats is a system rule to update info entries such as known node or edge kinds in the MainDB.

func (*SystemRuleUpdateNodeStats) Handle 

func (r *SystemRuleUpdateNodeStats) Handle(gm *Manager, trans Trans, event int, ed ...interface{}) error

Handle handles an event.

func (*SystemRuleUpdateNodeStats) Handles 

func (r *SystemRuleUpdateNodeStats) Handles() []int

Handles returns a list of events which are handled by this rule.

func (*SystemRuleUpdateNodeStats) Name 

func (r *SystemRuleUpdateNodeStats) Name() string

Name returns the name of the rule.

type Trans 

type Trans interface {

	/*
	   ID returns a unique transaction ID.
	*/
	ID() string

	/*
	   String returns a string representation of this transatction.
	*/
	String() string

	/*
	   Counts returns the transaction size in terms of objects. Returned values
	   are nodes to store, edges to store, nodes to remove and edges to remove.
	*/
	Counts() (int, int, int, int)

	/*
	   IsEmpty returns if this transaction is empty.
	*/
	IsEmpty() bool

	/*
	   Commit writes the transaction to the graph database. An automatic rollback is done if
	   any non-fatal error occurs. Failed transactions cannot be committed again.
	   Serious write errors which may corrupt the database will cause a panic.
	*/
	Commit() error

	/*
	   StoreNode stores a single node in a partition of the graph. This function will
	   overwrites any existing node.
	*/
	StoreNode(part string, node data.Node) error

	/*
	   UpdateNode updates a single node in a partition of the graph. This function will
	   only update the given values of the node.
	*/
	UpdateNode(part string, node data.Node) error

	/*
	   RemoveNode removes a single node from a partition of the graph.
	*/
	RemoveNode(part string, nkey string, nkind string) error

	/*
	   StoreEdge stores a single edge in a partition of the graph. This function will
	   overwrites any existing edge.
	*/
	StoreEdge(part string, edge data.Edge) error

	/*
	   RemoveEdge removes a single edge from a partition of the graph.
	*/
	RemoveEdge(part string, ekey string, ekind string) error
}

Trans is a transaction object which should be used to group node and edge operations.

func NewConcurrentGraphTrans 

func NewConcurrentGraphTrans(gm *Manager) Trans

NewConcurrentGraphTrans creates a new thread-safe graph transaction.

func NewGraphTrans 

func NewGraphTrans(gm *Manager) Trans

NewGraphTrans creates a new graph transaction. This object is not thread safe and should only be used for non-concurrent use cases; use NewConcurrentGraphTrans for concurrent use cases.

func NewRollingTrans 

func NewRollingTrans(t Trans, n int, gm *Manager, newTrans func(*Manager) Trans) Trans

NewRollingTrans wraps an existing transaction into a rolling transaction. Rolling transactions can be used for VERY large datasets and will commit themselves after n operations. Rolling transactions are always thread-safe.

Source Files

View all Source files

Directories

Path Synopsis
Package data contains classes and functions to handle graph data.
 Package data contains classes and functions to handle graph data.
Package graphstorage contains classes which model storage objects for graph data.
 Package graphstorage contains classes which model storage objects for graph data.
Package util contains utility classes for the graph storage.
 Package util contains utility classes for the graph storage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.