sdutils

package module
v0.0.0-...-652fa3e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 22, 2021 License: Apache-2.0 Imports: 25 Imported by: 0

README

Deprecated

Graviton is currently deprecated. Please see our helm charts for similar functionality on Kuberenetes.

Stardog Graviton: Virtual Appliance Manager

This application creates and controls a stardog database cluster running in a cloud. The current implementation only works in AWS.

Features:

  • Create base images with all software dependencies burnt into them.
  • Create a set of volumes to back a Stardog cluster.
  • Launch a fully working Stardog cluster database.
  • Monitor health of the cluster.
  • Cleanup all resources.

Dependencies:

Quick start

In order to use stardog-graviton in its current form the following environment variables must be set.

AWS_ACCESS_KEY_ID=<a valid aws access key>
AWS_SECRET_ACCESS_KEY=<a valid aws secret key>

The account associated with the access tokens must have the ability to create IAM credentials and full EC2 access.

If the correct versions of terraform and packer are not installed on your system them will be downloaded from hashicorp directly when graviton needs them.

Launch a cluster:

The easiest way to launch a cluster is to run stardog-graviton in interactive mode. This will cause the program to ask a series of questions in order to get the needed values to launch a cluster. Here is a sample session:

$ ./bin/stardog-graviton launch mystardog2
Failed to load the default file /Users/stardog/.graviton/default.json The file /Users/stardog/.graviton/default.json does not exist.
What version of stardog are you launching?: 4.2
What is the path to the Stardog release?: /Users/stardog/stardog-4.2.zip
There is no base image for version 4.2.
Do you wish to build one? (yes/no): yes
| Running packer to build the image...
done
AMI Successfully built: ami-0a1d486a
Creating the new deployment mystardog2
EC2 keyname (default): stardog
Private key path: /Users/stardog/.ssh/stardog
What is the path to your Stardog license?: /Users/stardog/data/stardog/stardog-license-key.bin
| Calling out to terraform to create the volumes...
- Calling out to terraform to stop builder instances...
Successfully created the volumes.
\ Creating the instance VMs...
Successfully created the instance.
Waiting for stardog to come up...
/ Waiting for external health check to pass...
The instance is healthy
Stardog is available here: http://mystardog2sdelb-682913646.us-west-1.elb.amazonaws.com:5821
ssh is available here: mystardog2belb-1558182568.us-west-1.elb.amazonaws.com
The instance is healthy
Coordinator:
   10.0.100.6:5821
Nodes:
   10.0.101.168:5821
   10.0.100.243:5821
Success.

To avoid being asked questions a file name ~/.graviton/default.json can be created. An example can be found int the defaults.json.example file.

All of the components needed to run a Stardog cluster are considered part of a deployment. Every deployment must be given a name that is unique to each cloud account. In the above example the deployment name is mystardog2.

Status

Once the image has been successfully launched its health can be monitored with the status command:

$ stardog-graviton status mystardog2
Stardog is available here: http://mystardog2sdelb-1562381657.us-west-1.elb.amazonaws.com:5821
ssh is available here: mystardog2belb-1941074804.us-west-1.elb.amazonaws.com
The instance is healthy
Coordinator:
   10.0.100.110:5821
Nodes:
   10.0.101.230:5821
   10.0.100.33:5821
Success.
Cleanup

The EC2 charges by the hour for the VMs that Graviton runs thus when the cluster is no longer in use it is important to clean it up with the destroy commmand.

$ ./bin/stardog-graviton destroy mystardog2
Failed to load the default file /Users/stardog/.graviton/default.json The file /Users/stardog/.graviton/default.json does not exist.
This will destroy all volumes and instances associated with this deployment.
Do you really want to destroy? (yes/no): yes
- Deleting the instance VMs...
Successfully destroyed the instance.
\ Calling out to terraform to delete the images...
Successfully destroyed the volumes.
Success.
Base image

The first time this is done a base image needs to be created. This image will have Stardog, Zookeeper, and a set of other dependencies needed to run the cluster baked into it. Even tho this image will be localized to your aws account no secrets (including the license) will be baked into it. Future launches of the cluster with the same stardog version will not require this step.

To create the base image in a separate step use the following subcommands

  baseami [<flags>] <release> <sd-version>
    Create a base ami.
Volumes

Every cluster needs a backing set of volumes to store the data. In AWS this data is stored on elastic block store (ebs) volumes. The launching a new cluster these volumes are created, formated and populated with your stardog license. The database admin password is set at this time as well. Because these volumes will contain your data and stardog licenses it is important to keep them secret.

To control the volumes use the following subcommands:

  volume new [<flags>] <deployment> <license> <size> <count>
    Create new backing storage.

  volume destroy [<flags>] <deployment>
    This will destroy the volumes permanently.

  volume status <deployment>
    Display information about the volumes.
Instances

Running the stardog cluster requires several virtual machines. At least 3 zookeeper nodes are needed for it to run safely and at least 2 stardog nodes. Additionally a bastion node is used in order to allow ssh access to all other VMs as well as provide a configured client environment read to use. AWS charges by the hour for the VMs so it is important to not leave them running. In a given deployment the VMs can be started and stopped without destroying the data backing them. The following subcommands can be used to control the VM instances:

  instance new [<flags>] <deployment> <zk>
    Create new set of VMs running Stardog.

  instance destroy [<flags>] <deployment>
    Destroy the instance.

  instance status [<flags>] <deployment>
    Get information about the instance.
Cluster status

The status of a give deployment can be checked with the status subcommand. The status can also be written to a json file if the --json-file option is included. Here is an example session:

$ ./bin/stardog-graviton status mystardog2 --json-file=output.json
Failed to load the default file /Users/stardog/.graviton/default.json: The file /Users/stardog/.graviton/default.json does not exist.
Stardog is available here: http://mystardog2sdelb-682913646.us-west-1.elb.amazonaws.com:5821
ssh is available here: mystardog2belb-1558182568.us-west-1.elb.amazonaws.com
The instance is healthy
Coordinator:
   10.0.100.6:5821
Nodes:
   10.0.100.243:5821
   10.0.101.168:5821
Success.

The output file looks like the following:

{
    "stardog_url": "http://mystardog2sdelb-682913646.us-west-1.elb.amazonaws.com:5821",
    "ssh_host": "mystardog2belb-1558182568.us-west-1.elb.amazonaws.com",
    "healthy": true,
    "volume": {
        "VolumeIds": [
            "vol-c5070c6b",
            "vol-007183bf",
            "vol-2e070c80"
        ]
    },
    "instane": {
        "ZkNodesContact": [
            "internal-mystardog2zkelb0-770674234.us-west-1.elb.amazonaws.com",
            "internal-mystardog2zkelb1-1838829827.us-west-1.elb.amazonaws.com",
            "internal-mystardog2zkelb2-2057497106.us-west-1.elb.amazonaws.com"
        ]
    }
}

Troubleshooting

Logging

The stardog-graviton program logs to the console and to a log file. To increase the level of console logging and --verbose to the command line multiple times and --log-level=DEBUG. While this will provide details much more verbose logging can be found in the log file. Each deployment will have its own log file located at ~/.graviton/deployments/<deployment name>/logs/graviton.log

ssh access to the cluster is provided via the bastion node. Its contact point is displayed in the status command. Once logged into that node the stardog nodes and zookeeper nodes can be access. The following log files can be helpful in debugging a deployment that is not working:

  • /var/log/cloud-init-output.log
  • /var/log/stardog.image_config.log
  • /mnt/data/stardog-home/stardog.log
  • /mnt/data/stardog-home/zookeeper.log
  • /var/lib/cloud/instance/scripts/part-001
  • /var/log/cloud-init.log
SSH Agent

Some of Graviton's features require a running ssh-agent loaded with the correct private key.

$ eval $(ssh-agent)
Agent pid 77228
$ ssh-add ~/.ssh/stardogbuzztroll
Identity added: /Users/stardog/.ssh/stardogbuzztroll (/Users/stardog/.ssh/stardogbuzztroll)

Build stardog-graviton

go version 1.7.1 is required and must be in your system path in order to build stardog-graviton as is the program make. Make sure that GOPATH is set properly, and that graviton is checkout into $GOPATH/src/github.com/stardog-union

$ make
./scripts/build-local.sh
$ ls -l $GOPATH/bin/stardog-graviton
-rwxr-xr-x  1 stardog  staff  17812772 Nov  9 11:06 bin/stardog-graviton

AWS architecture

This section describes the architecture of the Graviton when running in AWS. Other cloud types may be added in the future.

AWS Components

The following AWS components are the most important to the Graviton deployment.

Autoscaling Groups

This is used to make sure that the zookeeper and stardog clusters are held to N nodes. If a node is detected to be unhealthy AWS will restart that node

Elastic Load Balancer

These are used for two reasons:

  1. To distribute client requests across Stardog server nodes.
  2. As a layer of indirection to make each node in an autoscaling group reliably addressable. This is basically cruft due to some missing features in ec2.
Elastic Block Store

This is used as a backing store for each Stardog node

Interactions

Other AWS components are used as well but those are the key concepts.

The first things that is done for a deployment is create the volume set to back the database. These are created and taged with the deployment name. They remain in the users AWS account until they are deleted. They do not have to be actively attached to a running VM.

When a deployment is launched the following autoscaling groups are created:

  • One autoscaling group for each zookeeper node. Each one of these autoscaling groups monitors a single node. As mentioned above this is essentially a work around of some limited EC2 features.
  • One autoscaling group for all the Stardog nodes. This group makes sure N Stardog nodes are always running and healthy. If one fails that VM will be restarted.
  • One to make sure that a single bastion node is always running.
Boot steps
  1. First the zookeeper cluster comes up and each node joins a pool.
  2. The Stardog nodes start.
    • Each Stardog node searches the EC2 account for an EBS volume tagged with the same deployment name.
    • Each node selects one of the found EBS volumes at random and attempts to mount it.
    • If it fails to mount (most likely due to one of the other nodes beating it to the mount) it retries n times.
    • Once it has a volume mounted to it verifies that it can find the zookeeper pool.
    • Finally it starts the start dog server.
  3. The stardog-graviton program waits until the stardog cluser is healthy. It does this by checking the url http://<stardog load balancer>:5821/admin/healthcheck

Documentation

Index

Constants

View Source 
const (
	// ERROR will only log error level messages
	ERROR = 1
	// WARN will log ERROR and WARN level messages
	WARN = 2
	// INFO is the default level and will log information, warnings and errors
	INFO = 3
	// DEBUG is a very verbose log level and should be used only for finding problems
	DEBUG = 4
)

Variables

View Source 
var (

	// LogLevelNames is an array of strings that define all the valid log levels
	LogLevelNames []string
)

Functions

func AddCloudType 

func AddCloudType(p Plugin)

AddCloudType will associate a new plugin type with this graviton instances

func AskUser 

func AskUser(prompt string, defaultValue string) (string, error)

AskUser prompts a console user to enter input. prompt is the string that will be displayed to them and defaultValue will be the result if the user just hits enter.

func AskUserInteractiveInt 

func AskUserInteractiveInt(prompt string, defaultValue int, skipIfDefault bool, val *int) error

AskUserInteractiveInt prompts the user to enter an integer.

func AskUserInteractiveString 

func AskUserInteractiveString(prompt string, defaultValue string, skipIfDefault bool, val *string) error

AskUserInteractiveString prompts the user to enter a string.

func AskUserYesOrNo 

func AskUserYesOrNo(prompt string) bool

AskUserYesOrNo is just a convenience wrapper around AskUser that looks for a yes or no answer. A case insensitive yes will return true and all other values will return false.

func BbCode 

func BbCode(data string)

BbCode converts the bb ascii art information into console colorsMap

func CopyFile 

func CopyFile(src string, dst string) error

CopyFile will copy the context of 1 path to another

func CreateInstance 

func CreateInstance(context AppContext, baseD *BaseDeployment, dep Deployment, volumeSize int, zkSize int, waitMaxTimeSec int, timeoutSec int, mask string, bastionVolSnapshotId string, noWait bool) error

CreateInstance wraps up the deployment.CreateInstance method and blocks until the deployment is considered healthy. It will then change the password by SSHing into the bastion node. Once that is complete it will open up the the firewall.

func DeleteDeployment 

func DeleteDeployment(context AppContext, name string)

DeleteDeployment will remove all information stored on the local file system that is associated with a deployment.

func DeploymentDir 

func DeploymentDir(confDir string, deploymentName string) string

DeploymentDir abstracts the location of deployment information files into a function

func FindProgramVersion 

func FindProgramVersion(context AppContext, program string, version string, url string) error

FindProgramVersion is used to check for golang style single executable programs of a specific version. We are currently using it for just packer and terraform and there are no guarantees that it will work for other programs with different path expectations

func FullStatus 

func FullStatus(context AppContext, baseD *BaseDeployment, dep Deployment, internal bool, outfile string) error

FullStatus inspects the state of a deployment and prints it out to the console.

func GatherLogs 

func GatherLogs(context AppContext, baseD *BaseDeployment, dep Deployment, outfile string) error

GatherLogs sshes into the bastion node and collects logs from the stardog nodes

func GenerateKey 

func GenerateKey(dir string, keyname string) (string, []byte, error)

func GetLocalOnlyHTTPMask 

func GetLocalOnlyHTTPMask() string

GetLocalOnlyHTTPMask uses a network service to guess the external IP of the local host.

func GetMaxIopsRatio 

func GetMaxIopsRatio() int

func IsHealthy 

func IsHealthy(context AppContext, baseD *BaseDeployment, d Deployment, internal bool) bool

IsHealthy checks the deployment to see if the Stardog service is healthy. if internal is set to true it will test by sshing into the bastion node first.

func LoadJSON 

func LoadJSON(obj interface{}, path string) error

LoadJSON is a convenience function to load a JSON file into an interface object

func PathExists 

func PathExists(filepath string) bool

PathExists is a convenience function to determine if a path path exists.

func RunCommand 

func RunCommand(cliContext AppContext, cmd exec.Cmd, lineScanner LineScanner, spinner *Spinner) (*[]ScanResult, error)

RunCommand will fork and execute a command in the shell. The lineScanner object will be used to collect output and return it to the caller.

func RunSSH 

func RunSSH(context AppContext, baseD *BaseDeployment, d Deployment) error

RunSSH will start an ssh session on the bastion node

func UpdateStardog 

func UpdateStardog(context AppContext, baseD *BaseDeployment, dep Deployment, sdReleaseFile string) error

Upload a new Stardog release zip to the nodes and restart Stardog

func ValueStringToInt 

func ValueStringToInt(i string) (int, error)

ValueStringToInt returns a integer from a string with a unit of g, m, or k.

func WaitForHealth 

func WaitForHealth(context AppContext, baseD *BaseDeployment, d Deployment, waitTimeout int, internal bool) error

WaitForHealth will block until the deployment is considered healthy or the timeout expires. If internal is true it will ssh into the bastion node before checking the health URL.

func WaitForNClusterNodes 

func WaitForNClusterNodes(context AppContext, size int, sdURL string, pw string, waitTimeout int) error

func WriteFile 

func WriteFile(path string, contents string) error

func WriteJSON 

func WriteJSON(obj interface{}, path string) error

WriteJSON will take an interface object and serialize it into JSON and store it in a file at the given path.

Types

type AppContext 

type AppContext interface {
	ConsoleLog(level int, format string, v ...interface{})
	Logf(level int, format string, v ...interface{})
	GetConfigDir() string
	GetVersion() string
	GetInteractive() bool
	HighlightString(a ...interface{}) string
	SuccessString(a ...interface{}) string
	FailString(a ...interface{}) string
}

AppContext provides and abstraction to logging, console interaction and basic configuration information

type BaseDeployment 

type BaseDeployment struct {
	Type            string      `json:"type,omitempty"`
	Name            string      `json:"name,omitempty"`
	Directory       string      `json:"directory,omitempty"`
	Version         string      `json:"version,omitempty"`
	PrivateKey      string      `json:"private_key,omitempty"`
	CustomPropsFile string      `json:"custom_props,omitempty"`
	CustomLog4J     string      `json:"custom_log4j,omitempty"`
	IdleTimeout     int         `json:"idle_timeout,omitempty"`
	Environment     []string    `json:"environment,omitempty"`
	DisableSecurity bool        `json:"disable_security,omitempty"`
	CloudOpts       interface{} `json:"cloud_opts,omitempty"`
	CustomScript    string      `json:"custom_script,omitempty"`
	CustomZkScript  string      `json:"custom_zk_script,omitempty"`
}

BaseDeployment hold information about the deployments and is serialized to JSON. CloudOpts is defined by the specific plugin in use.

type CommandOpts 

type CommandOpts struct {
	Cli                  *kingpin.Application
	LaunchCmd            *kingpin.CmdClause
	DestroyCmd           *kingpin.CmdClause
	StatusCmd            *kingpin.CmdClause
	LeaksCmd             *kingpin.CmdClause
	ClientCmd            *kingpin.CmdClause
	SSHCmd               *kingpin.CmdClause
	PasswdCmd            *kingpin.CmdClause
	AboutCmd             *kingpin.CmdClause
	BuildCmd             *kingpin.CmdClause
	NewDeploymentCmd     *kingpin.CmdClause
	DestroyDeploymentCmd *kingpin.CmdClause
	ListDeploymentCmd    *kingpin.CmdClause
	NewVolumesCmd        *kingpin.CmdClause
	DestroyVolumesCmd    *kingpin.CmdClause
	StatusVolumesCmd     *kingpin.CmdClause
	LaunchInstanceCmd    *kingpin.CmdClause
	DestroyInstanceCmd   *kingpin.CmdClause
	StatusInstanceCmd    *kingpin.CmdClause
}

CommandOpts holds all of the CLI parsing information for the system. It is passed to plugins so that each driver can add their own specific flags.

type ConsoleEffect 

type ConsoleEffect func(a ...interface{}) string

ConsoleEffect is a function for writing lines to the console in a visually pleasing way. For example red text for error messages.

type Deployment 

type Deployment interface {
	CreateVolumeSet(licensePath string, sizeOfEachVolume int, clusterSize int) error
	DeleteVolumeSet() error
	StatusVolumeSet() error
	VolumeExists() bool
	ClusterSize() (int, error)

	CreateInstance(volumeSize int, zookeeperSize int, idleTimeout int, bastionVolSnapshotId string) error
	OpenInstance(volumeSize int, zookeeperSize int, mask string, idleTimeout int) error
	DeleteInstance() error
	StatusInstance() error
	InstanceExists() bool

	FullStatus() (*StardogDescription, error)

	DestroyDeployment() error
}

Deployment is an interface to a plugin that is managing the actual Stardog services.

func LoadDeployment 

func LoadDeployment(context AppContext, baseD *BaseDeployment, new bool) (Deployment, error)

LoadDeployment inflates a Deployment object from the information stored in the configuration directory.

type LineScanner 

type LineScanner func(cliContext AppContext, line string) *ScanResult

LineScanner is a function that will search a line for given values and return results in a ScanResult if it finds something. It may return nil

type Plugin 

type Plugin interface {
	Register(cmdOpts *CommandOpts) error
	DeploymentLoader(context AppContext, baseD *BaseDeployment, new bool) (Deployment, error)
	LoadDefaults(defaultCliOpts interface{}) error
	BuildImage(context AppContext, sdReleaseFilePath string, version string) error
	GetName() string
	FindLeaks(context AppContext, deploymentName string, destroy bool, force bool) error
	HaveImage(context AppContext) bool
}

Plugin defines the interface for adding drivers to the system

func GetPlugin 

func GetPlugin(name string) (Plugin, error)

GetPlugin returns the plugin associate with the given name

type ScanResult 

type ScanResult struct {
	Key   string
	Value string
}

ScanResult is an object returned from a LineScanner. This allows us to use the uility function RunScanner and return different values from the output based on the specific command.

type SdVaLogger 

type SdVaLogger interface {
	Logf(level int, format string, v ...interface{})
}

SdVaLogger is the interface to the Stardog Logger

func NewSdVaLogger 

func NewSdVaLogger(realLogger *log.Logger, logLevel string) (SdVaLogger, error)

NewSdVaLogger creates a new Stardog logging object from a system logger

type Spinner 

type Spinner struct {
	// contains filtered or unexported fields
}

Spinner is an object used to show progress on the console.

func NewSpinner 

func NewSpinner(context AppContext, level int, message string) *Spinner

NewSpinner creates a new spinner object.

func (*Spinner) Close 

func (s *Spinner) Close()

Close ends the spinner session.

func (*Spinner) EchoNext 

func (s *Spinner) EchoNext()

EchoNext prints out the progress character.

type StardogDescription 

type StardogDescription struct {
	StardogURL          string      `json:"stardog_url,omitempty"`
	StardogInternalURL  string      `json:"stardog_internal_url,omitempty"`
	StardogNodes        []string    `json:"stardog_nodes,omitempty"`
	SSHHost             string      `json:"ssh_host,omitempty"`
	Healthy             bool        `json:"healthy,omitempty"`
	TimeStamp           time.Time   `json:"timestamp,omitempty"`
	VolumeDescription   interface{} `json:"volume,omitempty"`
	InstanceDescription interface{} `json:"instance,omitempty"`
}

StardogDescription represents the state of a Stardog deployment. It is effectively the output from a status command.

type TestContext 

type TestContext struct {
	ConfigDir string
	Version   string
}

TestContext is used for mocking out a context in many tests

func (*TestContext) ConsoleLog 

func (c *TestContext) ConsoleLog(level int, format string, v ...interface{})

func (*TestContext) FailString 

func (c *TestContext) FailString(a ...interface{}) string

func (*TestContext) GetConfigDir 

func (c *TestContext) GetConfigDir() string

func (*TestContext) GetInteractive 

func (c *TestContext) GetInteractive() bool

func (*TestContext) GetVersion 

func (c *TestContext) GetVersion() string

func (*TestContext) HighlightString 

func (c *TestContext) HighlightString(a ...interface{}) string

func (*TestContext) Logf 

func (c *TestContext) Logf(level int, format string, v ...interface{})

func (*TestContext) SuccessString 

func (c *TestContext) SuccessString(a ...interface{}) string

Source Files

View all Source files

Directories

Path Synopsis
cmd
stardog-graviton

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.