deep6

README

n3-deep6

An opinionated hexastore for linking education data

Work in progress, much more documentation to come, but the main.go in /example/d6 shows most of the key fetures.

Sample datasets for typical ed-tech data; SIF, XAPI are provided along with samples of arbitrary json formats that represent the sort of applicaiton data you might need to integrate.

Deep6 in this repo only really of use to anyone who wants to use an embedded datastore from within golang. The other layers that add data version control and access via web endpoints (and queries using GraphQL) will be added in the next couple of weeks.

To build and run a demo, go to /example/d6

then go build as normal, to run use:

>./clean.sh && ./d6 > out.txt

running the clean script is optional it just removes any existing database files, the output of d6 is piped to a file simply because it's verbose (multiple queries with annotations), so is easier to read in a file.

Documentation

Overview

Deep6 is a simple embeddable hexastore graph database. To the core hexastore tuple index we add auto-generated links based on selected property values, and offer object traversals over linked data types.

The use-case deep6 was creted for is to link data formats of different types in the education-technology space together so that user can retrieve, for example, all SIF, XAPI statements or abitrary json objects related to a student, a teacher or a school.

Index

• Variables
• func Flatten(m map[string]interface{}) map[string]interface{}
• func MergeErrors(cs ...<-chan error) <-chan error
• func WaitForPipeline(errs ...<-chan error) error
• type Deep6DB
  • func Open() (*Deep6DB, error)
  • func OpenFromFile(folderPath string) (*Deep6DB, error)
  • func (d6 *Deep6DB) Close()
  • func (d6 *Deep6DB) Delete(id string) error
  • func (d6 *Deep6DB) FindById(id string) (map[string][]map[string]interface{}, error)
  • func (d6 *Deep6DB) FindByPredicate(predicate string, filterspec FilterSpec) (map[string][]map[string]interface{}, error)
  • func (d6 *Deep6DB) FindByType(typename string, filterspec FilterSpec) (map[string][]map[string]interface{}, error)
  • func (d6 *Deep6DB) FindByValue(term string, filterspec FilterSpec) (map[string][]map[string]interface{}, error)
  • func (d6 *Deep6DB) IngestFromFile(fname string) error
  • func (d6 *Deep6DB) IngestFromHTTPRequest(r *http.Request) error
  • func (d6 *Deep6DB) IngestFromJSONChannel(c <-chan []byte) error
  • func (d6 *Deep6DB) IngestFromReader(r io.Reader) error
  • func (d6 *Deep6DB) TraversalWithId(id string, t Traversal, filterspec FilterSpec) (map[string][]map[string]interface{}, error)
  • func (d6 *Deep6DB) TraversalWithValue(val string, t Traversal, filterspec FilterSpec) (map[string][]map[string]interface{}, error)
• type Filter
• type FilterSpec
• type IngestData
• type Traversal
• type TraversalData
• type Triple
  • func NewTriple(tupla string) Triple
  • func (t Triple) Sextuple() []string
  • func (t Triple) SextupleLink() []string

Variables

var ErrNotFound = errors.New("object not found")

Functions

func Flatten

func Flatten(m map[string]interface{}) map[string]interface{}

Flatten takes a map of a json file and returns a new one where nested maps are replaced by dot-delimited keys.

func MergeErrors

func MergeErrors(cs ...<-chan error) <-chan error

MergeErrors merges multiple channels of errors. Based on https://blog.golang.org/pipelines.

func WaitForPipeline

func WaitForPipeline(errs ...<-chan error) error

WaitForPipeline waits for results from all error channels. It returns early on the first error.

Types

type Deep6DB

type Deep6DB struct {

	//
	// set level of audit ouput, one of: none, basic, high
	//
	AuditLevel string
	// contains filtered or unexported fields
}

func Open

func Open() (*Deep6DB, error)

Open a d6db will use the local path ./db/d6 by default

func OpenFromFile

func OpenFromFile(folderPath string) (*Deep6DB, error)

Open the database using the specified directory It will be created if it doesn't exist.

func (*Deep6DB) Close

func (d6 *Deep6DB) Close()

shuts down the and ensures all writes are committed.

func (*Deep6DB) Delete

func (d6 *Deep6DB) Delete(id string) error

Deletes object with specified id, and all links to that object.

func (*Deep6DB) FindById

func (d6 *Deep6DB) FindById(id string) (map[string][]map[string]interface{}, error)

Finds all tuples for a given object reinflates into a json object and returns a map[string]interface{} representing the json object

id - unique id of the object being searched for

func (*Deep6DB) FindByPredicate

func (d6 *Deep6DB) FindByPredicate(predicate string, filterspec FilterSpec) (map[string][]map[string]interface{}, error)

Finds all objects where a predicate exists. More useful with xapi than other data.

Predicate must be in dotted path form e.g. 'XAPI.verb' or 'StudentPersonal.PersonInfo.Name'

returns a map[string]interface{} representing the json object

predicate - string value to search for can be a partial predicate e.g. (full) StudentPersonal.PersonInfo.Name.FamilyName (partial) .FamilyName

filterspec - optional filter for results

func (*Deep6DB) FindByType

func (d6 *Deep6DB) FindByType(typename string, filterspec FilterSpec) (map[string][]map[string]interface{}, error)

Finds all objects of a specified type will be object name for SIF e.g. StudentPersonal or generic e.g. XAPI returns an array of map[string]interface{} containing the json objects

typename - object type to search for

func (*Deep6DB) FindByValue

func (d6 *Deep6DB) FindByValue(term string, filterspec FilterSpec) (map[string][]map[string]interface{}, error)

Finds all objects where a property has the given value. returns a map[string]interface{} representing the json object

term - string value to search for supports partial (prefix) search e.g. jaqueline as opposed to jaqueline5@email.com

func (*Deep6DB) IngestFromFile

func (d6 *Deep6DB) IngestFromFile(fname string) error

Load data into D6 from a file

func (*Deep6DB) IngestFromHTTPRequest

func (d6 *Deep6DB) IngestFromHTTPRequest(r *http.Request) error

Load data into D6 from an http request

func (*Deep6DB) IngestFromJSONChannel

func (d6 *Deep6DB) IngestFromJSONChannel(c <-chan []byte) error

Feed data into db from a channel providing json objects as byte slices - typically for us the iterator from crdt manager receiver

func (*Deep6DB) IngestFromReader

func (d6 *Deep6DB) IngestFromReader(r io.Reader) error

Feed data in D6 from any io.Reader

func (*Deep6DB) TraversalWithId

func (d6 *Deep6DB) TraversalWithId(id string, t Traversal, filterspec FilterSpec) (map[string][]map[string]interface{}, error)

Follows a traversal spec starting with an object found to have the supplied initial value

id - the value to search for
traversalspec - object types to traverse

func (*Deep6DB) TraversalWithValue

func (d6 *Deep6DB) TraversalWithValue(val string, t Traversal, filterspec FilterSpec) (map[string][]map[string]interface{}, error)

Follows a traversal spec starting with an object found to have the supplied value

Unlike find by id this traversal may find multiple matches so dataset returned can contain mmultiple result arrays, but data will be constrained by the Types of the traversal-spec.

Allows traversal using for example a localid for a user, or a user name as opposed to a unique id

val - the value to search for
traversalspec - object types to traverse

type Filter

type Filter struct {
	Predicate   string
	TargetValue string
}

type FilterSpec

type FilterSpec map[string][]Filter

data strucure

Predicate: predicte string (can be part predicate) in dotted form e.g. PersonInfo.FamilyName

or simply .FamilyName

TargetValue: the value to use as a filter, if the value matches the value of the predicate in the object the object will be returned in results

type IngestData

type IngestData struct {
	// Unique id for the object being processed
	// will be derived from the inbound json object
	// or created by the pipeline
	N3id string
	// The declared type of the objecct
	// such as a SIF StudentPersonal
	// for data with no type system, will use
	// the object datamodel, all xAPI objects
	// for instance end up as type XAPI
	Type string
	// The datamodel that the object being processed
	// appears to belong to based on the
	// datatype.toml classification
	// if no model can be derived will default to JSON
	DataModel string
	// The unmarshaled json of the object
	// as a map[string]interface{}
	RawData map[string]interface{}
	// The specifications for which features of an object should
	// be surfaced as links within the graph.
	// Provided in ./config/datatype.toml.
	// Searches triple predicates for the spec, so use e.g.
	// ".RefId" to find the precise refid of a SIF object
	// but use "RefId" (no dot) to find refids of referenced objects
	// such as SchoolInfoRefId
	LinkSpecs []string
	// Array of  values extracted during classification
	// which will be concatenated to make
	// a unique property identifier for objects
	// that have no discriminating features, e.g.
	// a syllabus has a stage, but so do lessons
	// it has a subject, but so do lessons and subjects
	// so to avoid filtering in a traversal
	// a combination of stage and subject will link
	// to only one sylabus.
	UniqueValues []string
	//
	// The resulting psuedo-unique key for this object.
	//
	Unique string
	// Potential links are derived from the inbound object
	// but need to be verified and written, this is done
	// by different parts of the ingest pipeline and so
	// are carried between stages in this slice
	LinkCandidates []Triple
	// The slice of hexastore triples parsed
	// from the original inbound data object
	Triples []Triple
	// The set of generated triiples that link
	// the features requested in LinkSpecs to the
	// rest of the graph
	LinkTriples []Triple
}

strucuture used by the ingest pipeeline to pass original data and derived data between each stage.

Clssification of the data, and derivation of the object unique id etc. are governed by the configuration found in the ./config/datatype.toml file, see comments there for more details.

type Traversal

type Traversal struct {
	TraversalSpec []string
}

Data type for traversal specification

Each element is be an object type name the traversal finds objects of each type in the order specified as long as there is some forward or backward link available.

type TraversalData

type TraversalData struct {
	//
	// Triples that successfully meet the
	// traversal spec requirements
	//
	// this data is persisted between stages of the
	// pipeline
	//
	// Stores the id and type of the match
	//
	TraversalMatches map[string]string
	//
	// Triples to be considered by next stage
	//
	// each stage replaces this data with new targets
	//
	TraversalStageTargets map[string]string
}

Data strucure to move between stages of a graph traversal pipeline

type Triple

type Triple struct {
	// subject
	S string
	// predicate
	P string
	// object
	O string
}

func NewTriple

func NewTriple(tupla string) Triple

func (Triple) Sextuple

func (t Triple) Sextuple() []string

func (Triple) SextupleLink

func (t Triple) SextupleLink() []string