grammes

README

Grammes

Grammes is an API/Wrapper for Gremlin and Janusgraph. It's written purely in Golang and allows for easy use of Gremlin without touching the Gremlin terminal.

Table of Contents

• Grammes
  • Table of Contents
  • Local Setup
    • Cloning Grammes
    • Setting up JanusGraph
    • Using Grammes
  • Testing Grammes
  • Additional Resources
    • Documentation on Gremlin
    • Examples
  • Troubleshooting
    • Fixing time outs when starting Janusgraph

Local Setup

You need to setup all of the following tools to run the service locally

• Go 1.12
• Git
• Elastic Search
• Cassandra
  • Java 8

---

Cloning Grammes

Begin by opening up a terminal or command prompt and clone the grammes repository.

go get -u github.com/00security/grammes

---

Setting up JanusGraph

if you have decided to use another graph database then you may move on to project setup

First off, direct your terminal to the Grammes' scripts directory.

cd $GOPATH/src/github.com/00security/grammes/scripts

In here you can find the gremlin.sh and janusgraph.sh scripts. To set up JanusGraph just run the janusgraph.sh script.

./janusgraph.sh

This should download and/or begin the graph and TinkerPop server.

To make sure that everything is running try running gremlin.sh.

$ ./gremlin.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/<username>/projects/nm/gocode/src/github.com/00security/grammes/bin/janusgraph-0.3.1-hadoop2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/<username>/projects/nm/gocode/src/github.com/00security/grammes/bin/janusgraph-0.3.1-hadoop2/lib/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15:05:59 WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
gremlin>

Once Gremlin starts then you may begin by running this command.

gremlin> :remote connect tinkerpop.server conf/remote.yaml
===>Configured localhost/127.0.0.1:8182

If you see the message that Gremlin was configured to the localhost then quit Gremlin.

gremlin> :exit

Finally, run the janusgraph.sh script again, but this time with the status flag.

./janusgraph.sh status

---

Using Grammes

Once you have cloned the repository then you may begin implementing it into a project. Let's begin with creating a place for your code in the $GOPATH, i.e.,

$GOPATH/src/github.com/<username-here>/<project-name-here>

Next, you'll want to create a main.go file. For this example I will be using MS Code, but you may use any editor you prefer.

code main.go

In this file we can begin by making it a typical empty main.go file like this:

package main

func main() {
}

Next, import the grammes package and begin using it by connecting your client to a gremlin server.

package main

import (
    "log"

    "github.com/00security/grammes"
)

func main() {
    // Creates a new client with the localhost IP.
    client, err := grammes.DialWithWebSocket("ws://127.0.0.1:8182")
    if err != nil {
        log.Fatalf("Error while creating client: %s\n", err.Error())
    }

    // Executing a basic query to assure that the client is working.
    res, err := client.ExecuteStringQuery("1+3")
    if err != nil {
        log.Fatalf("Querying error: %s\n", err.Error())
    }

    // Print out the result as a string
    for _, r := range res {
        log.Println(string(r))
    }
}

Once the client is created then you can begin querying the gremlin server via the .ExecuteQuery method in the client. To use this function you must put in a Query which is an interface for any kind of Query-able type in the package. These include: graph.String and traversal.String. For an example of querying the gremlin server for all of the Vertex labels:

package main

import (
    "log"

    "github.com/00security/grammes"
)

func main() {
    // Creates a new client with the localhost IP.
    client, err := grammes.DialWithWebSocket("ws://127.0.0.1:8182")
    if err != nil {
        log.Fatalf("Error while creating client: %s\n", err.Error())
    }

    // Executing a query to add a vertex to the graph.
    client.AddVertex("testingvertex")

    // Create a new traversal string to build your traverser.
    g := grammes.Traversal()

    // Executing a query to fetch all of the labels from the vertices.
    res, err := client.ExecuteQuery(g.V().Label())
    if err != nil {
        log.Fatalf("Querying error: %s\n", err.Error())
    }

    // Log out the response.
    for _, r := range res {
        log.Println(string(r))
    }
}

After this is all written you may run this file by saving it and hopping back on to your terminal. After starting your Gremlin Server and graph database in the terminal let's run this command to run the file:

go run main.go

For more examples look in the examples/ directory of the project. In there you'll find multiple examples on how to use the Grammes package.

Testing Grammes

Grammes uses goconvey by smartystreets for its tests. Before trying to run the unit tests in Grammes you should update your version of this repository using this command.

go get -u github.com/smartystreets/goconvey/convey

Once you have this downloaded you may run the tests in Grammes how you normally would in Golang.

go test ./...

Additional Resources

Documentation on Gremlin

To learn more about how to use Gremlin I highly recommend looking through their Tinkerpop3 documentation. It's full of examples and documentation on every traversal step available.

Examples

To find examples look in the examples/ directory of Grammes. In there you'll find plenty of examples related to how to use this package. Make sure you're running Janusgraph before you begin running the examples.

Troubleshooting

Fixing time outs when starting Janusgraph

If Nodetool times out or any other part of the setup times out then the most common issue is that Cassandra is already running on your machine. To fix this run this command.

# only for Unix at the moment.
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.cassandra.plist

Documentation

Overview

Package grammes is an API/Wrapper for the Gremlin traversal language to interact with graph databases. It includes various levels of functionality to add, remove, and change vertices and edges in the database. Usage of higher level API is shown in various examples in the `examples/` directory with full documentation.

To get started with this package you may begin by making a Grammes client using either the Dial function or the DialWithWebSocket function. With this client you may begin interacting with your graph database with the client's multitude of function options. To narrow down what you want to do it may be easier to choose one of the `client.Querier` options.

What this example does is create a new Grammes Client using the DialWithWebSocket function. With this client it executes a simple string query that just does some simple addition. Then it will return the raw result out.

For further customizability you may check out packages within the `query/` directory.

To see examples on how to use this package further then check out the `examples/` directory.

Example (ExecuteQuery)

package main

import (
	"log"

	"github.com/00security/grammes"
)

func main() {
	// Creates a new client with the localhost IP.
	client, err := grammes.DialWithWebSocket("ws://127.0.0.1:8182")
	if err != nil {
		log.Fatalf("Error while creating client: %s\n", err.Error())
	}

	// Create a graph traversal to use when querying.
	g := grammes.Traversal()

	// Executing a basic query to add a vertex to the graph
	// with label "testingVertex" and property "name" that equals "damien"
	res, err := client.ExecuteQuery(g.AddV("testingVertex").Property("name", "damien"))
	if err != nil {
		log.Fatalf("Querying error: %s\n", err.Error())
	}

	// Print out the result as a string
	for _, r := range res {
		log.Println(string(r))
	}
}

Example (ExecuteStringQuery)

package main

import (
	"log"

	"github.com/00security/grammes"
)

func main() {
	// Creates a new client with the localhost IP.
	client, err := grammes.DialWithWebSocket("ws://127.0.0.1:8182")
	if err != nil {
		log.Fatalf("Error while creating client: %s\n", err.Error())
	}

	// Executing a basic query to assure that the client is working.
	res, err := client.ExecuteStringQuery("1+3")
	if err != nil {
		log.Fatalf("Querying error: %s\n", err.Error())
	}

	// Print out the result as a string
	for _, r := range res {
		log.Println(string(r))
	}
}

Example (NewClient)

package main

import (
	"log"

	"github.com/00security/grammes"
)

func main() {
	// Creates a new client with the localhost IP.
	client, err := grammes.DialWithWebSocket("ws://127.0.0.1:8182")
	if err != nil {
		log.Fatalf("Error while creating client: %s\n", err.Error())
	}

	_ = client
}

Index

• Variables
• func CustomTraversal(q string) traversal.String
• func Traversal() traversal.String
• func VerboseTraversal() graph.String
• type APIData
• type Client
  • func Dial(conn gremconnect.Dialer, cfgs ...ClientConfiguration) (*Client, error)
  • func DialWithWebSocket(host string, cfgs ...ClientConfiguration) (*Client, error)
  • func (c *Client) Address() string
  • func (c *Client) Auth() (*gremconnect.Auth, error)
  • func (c *Client) Close()
  • func (c *Client) Connect() error
  • func (c *Client) IsBroken() bool
  • func (c *Client) IsConnected() bool
  • func (c *Client) Redial(dialer gremconnect.Dialer) error
  • func (c *Client) SetLogger(newLogger logging.Logger)
• type ClientConfiguration
  • func WithAuthUserPass(user, pass string) ClientConfiguration
  • func WithCompression(enableCompression bool) ClientConfiguration
  • func WithErrorChannel(err chan error) ClientConfiguration
  • func WithGremlinVersion(versionNumber int) ClientConfiguration
  • func WithHTTPAuth(provider gremconnect.AuthProvider) ClientConfiguration
  • func WithHandshakeTimeout(handshakeTimeout time.Duration) ClientConfiguration
  • func WithLogger(newLogger logging.Logger) ClientConfiguration
  • func WithMaxConcurrentMessages(limit int) ClientConfiguration
  • func WithMaxConcurrentRequests(limit int) ClientConfiguration
  • func WithPingInterval(interval time.Duration) ClientConfiguration
  • func WithReadBufferSize(readBufferSize int) ClientConfiguration
  • func WithReadingWait(interval time.Duration) ClientConfiguration
  • func WithRequestTimeout(interval time.Duration) ClientConfiguration
  • func WithTLS(conf *tls.Config) ClientConfiguration
  • func WithTimeout(interval time.Duration) ClientConfiguration
  • func WithWriteBufferResizing(writeBufferResizing bool) ClientConfiguration
  • func WithWriteBufferSize(writeBufferSize int) ClientConfiguration
  • func WithWritingWait(interval time.Duration) ClientConfiguration
• type Data
• type Edge
• type EdgeList
• type IDList
• type Property
• type PropertyList
• type PropertyMap
• type PropertyValue
• type SimpleValue
• type Vertex
• type VertexList
• type VertexValue

Examples

• Package (ExecuteQuery)
• Package (ExecuteStringQuery)
• Package (NewClient)

Variables

var (
	// NewWebSocketDialer returns websocket with established connection.
	NewWebSocketDialer = gremconnect.NewWebSocketDialer
	// NewVertex returns a vertex struct meant for adding it.
	NewVertex = model.NewVertex
	// NewProperty returns a property struct meant for adding it to a vertex.
	NewProperty = model.NewProperty

	// UnmarshalEdgeList is a utility to unmarshal a list
	// or array of edges properly.
	UnmarshalEdgeList = model.UnmarshalEdgeList
	// UnmarshalVertexList is a utility to unmarshal a list
	// or array of vertices properly.
	UnmarshalVertexList = model.UnmarshalVertexList
	// UnmarshalPropertyList is a utility to unmarshal a list
	// or array of IDs properly.
	UnmarshalPropertyList = model.UnmarshalPropertyList
)

Functions

func CustomTraversal

func CustomTraversal(q string) traversal.String

CustomTraversal could be used when you need to specifically need to change some property of the traversal. This can be something such as:

// ==> graph.traversal().withoutStrategies(LazyBarrierStrategy)

func Traversal

func Traversal() traversal.String

Traversal will return a new traversal string ready to use when executing a query.

func VerboseTraversal

func VerboseTraversal() graph.String

VerboseTraversal will return a new graph traversal string ready to use when executing a query.

Types

type APIData

type APIData = model.APIData

APIData is used to get quick access to the model.APIData without having to import it everywhere in the grammes package.

APIData holds the request in which you can make a query with using the Grammes library.

type Client

type Client struct {
	// GraphManager is an interface used to have query functions that
	// handle all interactions to the graph database.
	manager.GraphManager
	// contains filtered or unexported fields
}

Client is used to handle the graph, schema, connection, and basic debug logging when querying the graph database. When handling more than one queries to the graph database, it is better to use a client rather than the quick package.

func Dial

func Dial(conn gremconnect.Dialer, cfgs ...ClientConfiguration) (*Client, error)

Dial returns a working client with the given dialer and configurations.

func DialWithWebSocket

func DialWithWebSocket(host string, cfgs ...ClientConfiguration) (*Client, error)

DialWithWebSocket returns a new client with a websocket dialer and possible client configurations.

func (*Client) Address

func (c *Client) Address() string

Address returns the current host address from the dialer.

func (*Client) Auth

func (c *Client) Auth() (*gremconnect.Auth, error)

Auth will get the authentication user and pass from the dialer.

func (*Client) Close

func (c *Client) Close()

Close the connection to the Gremlin-server.

func (*Client) Connect

func (c *Client) Connect() error

Connect will connect to the configured host address. If this is reconnecting then make sure that your errs channel has a new handler function set up, because it won't set up automatically.

func (*Client) IsBroken

func (c *Client) IsBroken() bool

IsBroken returns whether the client is broken or not.

func (*Client) IsConnected

func (c *Client) IsConnected() bool

IsConnected returns if the client currently has an established connection to any servers.

func (*Client) Redial

func (c *Client) Redial(dialer gremconnect.Dialer) error

Redial will dial a new connecting using the provided.

func (*Client) SetLogger

func (c *Client) SetLogger(newLogger logging.Logger)

SetLogger will switch out the old logger with a new one provided as a parameter.

type ClientConfiguration

type ClientConfiguration func(*Client)

ClientConfiguration is the type used for configuring and changing values in the client and the dialer.

func WithAuthUserPass

func WithAuthUserPass(user, pass string) ClientConfiguration

WithAuthUserPass sets the authentication credentials within the dialer. (This includes the username and password)

func WithCompression

func WithCompression(enableCompression bool) ClientConfiguration

WithCompression sets the compression flag for websocket connections

func WithErrorChannel

func WithErrorChannel(err chan error) ClientConfiguration

WithErrorChannel will assign an error channel to send connection errors through for the user to handle.

func WithGremlinVersion

func WithGremlinVersion(versionNumber int) ClientConfiguration

WithGremlinVersion sets the version of the gremlin traversal language being used by the client.

func WithHTTPAuth

func WithHTTPAuth(provider gremconnect.AuthProvider) ClientConfiguration

WithHTTPAuth sets the authentication provider within the dialer

func WithHandshakeTimeout

func WithHandshakeTimeout(handshakeTimeout time.Duration) ClientConfiguration

WithHandshakeTimeout sets the websocket handshake timeout

func WithLogger

func WithLogger(newLogger logging.Logger) ClientConfiguration

WithLogger will replace the default zap.Logger with a custom logger that implements the logging.Logger interface.

func WithMaxConcurrentMessages

func WithMaxConcurrentMessages(limit int) ClientConfiguration

WithMaxConcurrentMessages sets the limit as to how many requests can be stored in the requests buffer.

func WithMaxConcurrentRequests

func WithMaxConcurrentRequests(limit int) ClientConfiguration

WithMaxConcurrentRequests sets the limit as to how many requests can done simultaneously.

func WithPingInterval

func WithPingInterval(interval time.Duration) ClientConfiguration

WithPingInterval sets the interval of ping sending for know is connection is alive and in consequence the client is connected.

func WithReadBufferSize

func WithReadBufferSize(readBufferSize int) ClientConfiguration

WithReadBufferSize sets the max read buffer size for the websocket frame

func WithReadingWait

func WithReadingWait(interval time.Duration) ClientConfiguration

WithReadingWait sets the time to wait when reading with the dialer in seconds.

func WithRequestTimeout

func WithRequestTimeout(interval time.Duration) ClientConfiguration

WithRequestTimeout sets the timeout when reading a request from the gremlin server

func WithTLS

func WithTLS(conf *tls.Config) ClientConfiguration

WithTLS sets the TLS config

func WithTimeout

func WithTimeout(interval time.Duration) ClientConfiguration

WithTimeout sets the timeout to wait when dialing with the dialer in seconds.

func WithWriteBufferResizing

func WithWriteBufferResizing(writeBufferResizing bool) ClientConfiguration

WithWriteBufferResizing enables dynamic write buffer expansion, effectively disabling websocket frame fragmentation (which TinkerPop doesn't support)

func WithWriteBufferSize

func WithWriteBufferSize(writeBufferSize int) ClientConfiguration

WithWriteBufferSize sets the max write buffer size for the websocket frame

func WithWritingWait

func WithWritingWait(interval time.Duration) ClientConfiguration

WithWritingWait sets the time to wait when writing with the dialer in seconds.

type Data

type Data = model.Data

Data is used to get quick access to the model.Data without having to import it everywhere in the grammes package.

Data holds basic information such as the label, name, ID, and properties of what this is being associated with.

type Edge

type Edge = model.Edge

Edge is used to get quick access to the model.Edge without having to import it everywhere in the grammes package.

Tinkerpop: http://tinkerpop.apache.org/javadocs/3.2.1/core/org/apache/tinkerpop/gremlin/structure/Edge.html

outVertex ---label---> inVertex.

Edge is the object that builds a connection between two or more vertices.

type EdgeList

type EdgeList = model.EdgeList

EdgeList is used to get quick access to the model.EdgeList without having to import it everywhere in the grammes package.

type IDList

type IDList = model.IDList

IDList is used to get quick access to the model.IDList without having to import it everywhere in the grammes package.

type Property

type Property = model.Property

Property is used to get quick access to the model.Property without having to import it everywhere in the grammes package.

Property holds the type and value of the property. It's extra information used by PropertyDetail.

type PropertyList

type PropertyList = model.PropertyList

PropertyList is used to get quick access to the model.PropertyList without having to import it everywhere in the grammes package.

type PropertyMap

type PropertyMap = model.PropertyMap

PropertyMap is used to get quick access to the model.PropertyMap without having to import it everywhere in the grammes package.

Tinkerpop: http://tinkerpop.apache.org/javadocs/3.2.1/core/org/apache/tinkerpop/gremlin/structure/Property.html

PropertyMap is the map used to hold the properties itself. the string key is equivalent to the Gremlin key and the []Property is the value. Properties can have multiple values; this is why we must have it as a slice of Property.

type PropertyValue

type PropertyValue = model.PropertyValue

PropertyValue is used to get quick access to the model.PropertyValue without having to import it everywhere in the grammes package.

PropertyValue contains the ID, value, and label of this property's value.

type SimpleValue

type SimpleValue = model.SimpleValue

SimpleValue is used to get quick access to the model.SimpleValue without having to import it everywhere in the grammes package.

SimpleValue is used to unmarshal simple value responses from the TinkerPop server. These can include simple datatypes like Int, String, Double, Bool, etc.

type Vertex

type Vertex = model.Vertex

Vertex is used to get quick access to the model.Vertex without having to import it everywhere in the grammes package.

TinkerPop: http://tinkerpop.apache.org/javadocs/3.2.1/core/org/apache/tinkerpop/gremlin/structure/Vertex.html

---inEdges---> vertex ---outEdges--->.

Vertex maintains pointers to both a set of incoming and outgoing Edge objects. The outgoing edges are the edges for which the Vertex is a tail. The incoming edges are those edges for which the Vertex is the head.

type VertexList

type VertexList = model.VertexList

VertexList is used to get quick access to the model.VertexList without having to import it everywhere in the grammes package.

type VertexValue

type VertexValue = model.VertexValue

VertexValue is used to get quick access to the model.VertexValue without having to import it everywhere in the grammes package.

VertexValue contains the 'value' data from the Vertex object.