vulcanizer

package module
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2020 License: MIT Imports: 13 Imported by: 0

README

vulcanizer

GitHub's ops focused Elasticsearch library

build status GoDoc Go Report Card release

This project is a golang library for interacting with an Elasticsearch cluster. It's goal is to provide a high level API to help with common tasks that are associated with operating an Elasticsearch cluster such as querying health status of the cluster, migrating data off of nodes, updating cluster settings, etc.

This project does not aim to be a fully-featured API client for querying or indexing to Elasticsearch.

Go API

You can perform custom operations in your Go application.

import "github.com/github/vulcanizer"

v = vulcanizer.NewClient("localhost", 9200)
oldSetting, newSetting, err := v.SetSetting("indices.recovery.max_bytes_per_sec", "1000mb")
Command line application

This project produces a vulcanizer binary that is a command line application that can be used to manage your Elasticsearch cluster.

$ vulcanizer help
Usage:
  vulcanizer [command]

Available Commands:
  aliases     Interact with aliases of the cluster.
  allocation  Set shard allocation on the cluster.
  analyze     Analyze text given an analyzer or a field and index.
  drain       Drain a server or see what servers are draining.
  fill        Fill servers with data, removing shard allocation exclusion rules.
  health      Display the health of the cluster.
  help        Help about any command
  indices     Display the indices of the cluster.
  nodes       Display the nodes of the cluster.
  repository  Interact with the configured snapshot repositories.
  setting     Interact with cluster settings.
  settings    Display all the settings of the cluster.
  shards      Get shard data by cluster node(s).
  snapshot    Interact with a specific snapshot.

Flags:
  -c, --cluster string      Cluster to connect to defined in config file
  -f, --configFile string   Configuration file to read in (default to "~/.vulcanizer.yaml")
  -h, --help                help for vulcanizer
      --host string         Host to connect to (default "localhost")
      --password string     Password to use during authentication
      --path string         Path to prepend to queries, in case Elasticsearch is behind a reverse proxy
  -p, --port int            Port to connect to (default 9200)
      --protocol string     Protocol to use when querying the cluster. Either 'http' or 'https'. Defaults to 'http' (default "http")
  -k, --skipverify string   Skip verifying server's TLS certificate. Defaults to 'false', ie. verify the server's certificate (default "false")
      --user string         User to use during authentication

Use "vulcanizer [command] --help" for more information about a command.
Roadmap and future releases

The proposed future for vulcanizer can be found in our ROADMAP.

Configuration and connection information

All commands take --cluster <name> to look up information in a configuration file in ~/.vulcanizer.yaml. The configuration should be in the form of

local:
  host: localhost
  port: 9200
staging:
  host: 10.10.2.1
  port: 9201
production:
  host: 10.10.1.1
  port: 9202

Alternatively, all commands take --host and --port for the connection information.

For example:

# Query for cluster health on the "local" cluster
vulcanizer health --cluster local

# Query for nodes against the node 10.10.2.1 and port 9202
vulcanizer nodes --host 10.10.2.1 --port 9202
Development

./script/build will compile the project and install the vulcanizer binary to $GOPATH/bin.

./script/test will run the tests in the project.

Supported Elasticsearch versions

Integration tests are set up to run against the latest v5 and v6 versions of Elasticsearch.

Name

Vulcanization is the process of making rubber more elastic, so vulcanizer is the library that makes Elasticsearch easier to work with!

Project status

This project is under active development.

Contributing

This repository is open to contributions. Please also see code of conduct

To get up and running, install the project into your $GOPATH and run the set up scripts.

go get github.com/github/vulcanizer

cd $GOPATH/src/github.com/github/vulcanizer

./script/bootstrap
./script/test

And the test suite should execute correctly.

License

This project is released under the MIT LICENSE. Please note it includes 3rd party dependencies release under their own licenses; these are found under vendor. When using the GitHub logos, be sure to follow the GitHub logo guidelines.

Authors

Authored by GitHub Engineering

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Alias added in v0.5.0 

type Alias struct {
	Name          string `json:"alias"`
	IndexName     string `json:"index"`
	Filter        string `json:"filter"`
	RoutingIndex  string `json:"routing.index"`
	RoutingSearch string `json:"routing.search"`
}

Holds information about an Elasticsearch alias, based on the _cat/aliases API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-alias.html

type AliasAction added in v0.5.0 

type AliasAction struct {
	ActionType AliasActionType
	IndexName  string `json:"index"`
	AliasName  string `json:"alias"`
}

Holds information needed to perform an alias modification, based on the aliases API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-aliases.html

func (*AliasAction) MarshalJSON added in v0.5.0 

func (ac *AliasAction) MarshalJSON() ([]byte, error)

type AliasActionType added in v0.5.0 

type AliasActionType string

Represent the two possible aliases actions: add or remove 

const (
	AddAlias    AliasActionType = "add"
	RemoveAlias AliasActionType = "remove"
)

type Auth added in v0.4.0 

type Auth struct {
	User     string
	Password string
}

type Client 

type Client struct {
	Host      string
	Port      int
	Secure    bool
	Path      string
	TLSConfig *tls.Config
	Timeout   time.Duration
	*Auth
}

Hold connection information to a Elasticsearch cluster.

func NewClient 

func NewClient(host string, port int) *Client

Initialize a new vulcanizer client to use. Deprecated: NewClient has been deprecated in favor of using struct initialization.

func (*Client) AnalyzeText added in v0.4.0 

func (c *Client) AnalyzeText(analyzer, text string) ([]Token, error)

Call the analyze API with sample text and an analyzer. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html

Use case: You want to see how Elasticsearch will break up sample text given a specific analyzer.

func (*Client) AnalyzeTextWithField added in v0.4.0 

func (c *Client) AnalyzeTextWithField(index, field, text string) ([]Token, error)

Call the analyze API with sample text on an index and a specific field . https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html

Use case: You have a particular field that might have custom analyzers and you want to see how this field will tokenize some particular text.

func (*Client) CloseIndex added in v0.5.0 

func (c *Client) CloseIndex(indexName string) error

Close an index on the cluster

Use case: You want to close an opened index

func (*Client) DeleteIndex added in v0.4.0 

func (c *Client) DeleteIndex(indexName string) error

Delete an index in the cluster.

Use case: You want to remove an index and all of its data.

func (*Client) DeleteSnapshot added in v0.2.0 

func (c *Client) DeleteSnapshot(repository string, snapshot string) error

Delete a snapshot

Use case: You want to delete older snapshots so that they don't take up extra space.

func (*Client) DrainServer 

func (c *Client) DrainServer(serverToDrain string) (ExcludeSettings, error)

Set shard allocation exclusion rules such that the Elasticsearch node with the name `serverToDrain` is excluded. This should cause Elasticsearch to migrate shards away from that node.

Use case: You need to restart an Elasticsearch node. In order to do so safely, you should migrate data away from it. Calling `DrainServer` with the node name will move data off of the specified node.

func (*Client) FillAll 

func (c *Client) FillAll() (ExcludeSettings, error)

Removes all shard allocation exclusion rules.

Use case: You had been performing maintenance on a number of Elasticsearch nodes. They are all ready to receive data again. Calling `FillAll` will remove all the allocation exclusion rules on the cluster, allowing Elasticsearch to freely allocate shards on the previously excluded nodes.

func (*Client) FillOneServer 

func (c *Client) FillOneServer(serverToFill string) (ExcludeSettings, error)

Set shard allocation exclusion rules such that the Elasticsearch node with the name `serverToFill` is no longer being excluded. This should cause Elasticsearch to migrate shards to that node.

Use case: You have completed maintenance on an Elasticsearch node and it's ready to hold data again. Calling `FillOneServer` with the node name will remove that node name from the shard exclusion rules and allow data to be relocated onto the node.

func (*Client) GetAliases added in v0.5.0 

func (c *Client) GetAliases(alias string) ([]Alias, error)

Get a subset the aliases in the cluster.

Use case: You want to see some basic info on a subset of the aliases of the cluster

func (*Client) GetAllAliases added in v0.5.0 

func (c *Client) GetAllAliases() ([]Alias, error)

Get all the aliases in the cluster.

Use case: You want to see some basic info on all the aliases of the cluster

func (*Client) GetAllIndices added in v0.5.0 

func (c *Client) GetAllIndices() ([]Index, error)

Get all the indices in the cluster.

Use case: You want to see some basic info on all the indices of the cluster.

func (*Client) GetClusterExcludeSettings 

func (c *Client) GetClusterExcludeSettings() (ExcludeSettings, error)

Get current cluster settings for shard allocation exclusion rules.

func (*Client) GetClusterSettings added in v0.4.0 

func (c *Client) GetClusterSettings() (ClusterSettings, error)

Get all the persistent and transient cluster settings.

Use case: You want to see the current settings in the cluster.

func (*Client) GetHealth 

func (c *Client) GetHealth() (ClusterHealth, error)

Get the health of the cluster.

Use case: You want to see information needed to determine if the Elasticsearch cluster is healthy (green) or not (yellow/red).

func (*Client) GetIndexSettings added in v0.4.0 

func (c *Client) GetIndexSettings(index string) ([]Setting, error)

Get the settings of an index in a machine-oriented format.

Use case: You can view the custom settings that are set on a particular index.

func (*Client) GetIndices 

func (c *Client) GetIndices(index string) ([]Index, error)

Get a subset of indices

func (*Client) GetNodeAllocations added in v0.5.2 

func (c *Client) GetNodeAllocations() ([]Node, error)

Get all the nodes and their allocation/disk usage in the cluster.

Use case: You want to see how much disk is being used by the nodes in the cluster.

func (*Client) GetNodes 

func (c *Client) GetNodes() ([]Node, error)

Get all the nodes in the cluster.

Use case: You want to see what nodes Elasticsearch considers part of the cluster.

func (*Client) GetPrettyIndexMappings added in v0.4.0 

func (c *Client) GetPrettyIndexMappings(index string) (string, error)

Get the mappings of an index in a pretty-printed format.

Use case: You can view the custom mappings that are set on a particular index.

func (*Client) GetPrettyIndexSettings added in v0.4.0 

func (c *Client) GetPrettyIndexSettings(index string) (string, error)

Get the settings of an index in a pretty-printed format.

Use case: You can view the custom settings that are set on a particular index.

func (*Client) GetRepositories added in v0.3.0 

func (c *Client) GetRepositories() ([]Repository, error)

List snapshot respositories on the cluster

Use case: You want to see all of the configured backup repositories on the given cluster, what types they are and if they are verified.

func (*Client) GetShardOverlap added in v0.5.0 

func (c *Client) GetShardOverlap(nodes []string) (map[string]ShardOverlap, error)

Get details regarding shard distribution across a given set of cluster nodes.

Use case: You can leverage this information to determine if it's safe to remove cluster nodes without losing data.

func (*Client) GetShardRecovery added in v0.5.0 

func (c *Client) GetShardRecovery(nodes []string, onlyActive bool) ([]ShardRecovery, error)

Get details regarding shard recovery operations across a set of cluster nodes.

Use case: You can view the shard recovery progress of the cluster.

func (*Client) GetShards added in v0.5.0 

func (c *Client) GetShards(nodes []string) ([]Shard, error)

Get shard data for all or a subset of nodes

Use case: You can view shard information on all nodes or a subset.

func (*Client) GetSnapshotStatus 

func (c *Client) GetSnapshotStatus(repository string, snapshot string) (Snapshot, error)

Get detailed information about a particular snapshot.

Use case: You had a snapshot fail and you want to see the reason why and what shards/nodes the error occurred on.

func (*Client) GetSnapshots 

func (c *Client) GetSnapshots(repository string) ([]Snapshot, error)

List the snapshots of the given repository.

Use case: You want to see information on snapshots in a repository.

func (*Client) ModifyAliases added in v0.5.0 

func (c *Client) ModifyAliases(actions []AliasAction) error

Interact with aliases in the cluster.

Use case: You want to add, delete or update an index alias

func (*Client) OpenIndex added in v0.5.0 

func (c *Client) OpenIndex(indexName string) error

Open an index on the cluster

Use case: You want to open a closed index

func (*Client) RestoreSnapshotIndices added in v0.3.0 

func (c *Client) RestoreSnapshotIndices(repository string, snapshot string, indices []string, restoredIndexPrefix string, indexSettings map[string]interface{}) error

Restore an index or indices on the cluster

Use case: You want to restore a particular index or indices onto your cluster with a new name.

func (*Client) SetAllocation 

func (c *Client) SetAllocation(allocation string) (string, error)

Enables or disables allocation for the cluster.

Use case: You are performing an operation the cluster where nodes may be dropping in and out. Elasticsearch will typically try to rebalance immediately but you want the cluster to hold off rebalancing until you complete your task. Calling `SetAllocation("disable")` will disable allocation so Elasticsearch won't move/relocate any shards. Once you complete your task, calling `SetAllocation("enable")` will allow Elasticsearch to relocate shards again.

func (*Client) SetClusterSetting added in v0.4.0 

func (c *Client) SetClusterSetting(setting string, value *string) (*string, *string, error)

Set a new value for a cluster setting. Returns existing value and new value as well as error, in that order If the setting is not set in Elasticsearch (it's falling back to default configuration) SetClusterSetting's existingValue will be nil. If the value provided is nil, SetClusterSetting will remove the setting so that Elasticsearch falls back on default configuration for that setting.

Use case: You've doubled the number of nodes in your cluster and you want to increase the number of shards the cluster can relocate at one time. Calling `SetClusterSetting("cluster.routing.allocation.cluster_concurrent_rebalance", "100")` will update that value with the cluster. Once data relocation is complete you can decrease the setting by calling `SetClusterSetting("cluster.routing.allocation.cluster_concurrent_rebalance", "20")`.

func (*Client) SetIndexSetting added in v0.4.0 

func (c *Client) SetIndexSetting(index, setting, value string) (string, string, error)

Set a setting on an index.

Use case: Set or update an index setting for a particular index.

func (*Client) SnapshotAllIndices added in v0.3.0 

func (c *Client) SnapshotAllIndices(repository string, snapshot string) error

Take a snapshot of all indices on the cluster to the given repository

Use case: You want to backup all of the indices on the cluster to the given repository.

func (*Client) SnapshotIndices added in v0.3.0 

func (c *Client) SnapshotIndices(repository string, snapshot string, indices []string) error

Take a snapshot of specific indices on the cluster to the given repository

Use case: You want to backup certain indices on the cluster to the given repository.

func (*Client) VerifyRepository added in v0.2.0 

func (c *Client) VerifyRepository(repository string) (bool, error)

Verify a snapshot repository

Use case: Have Elasticsearch verify a repository to make sure that all nodes can access the snapshot location correctly.

type ClusterHealth added in v0.1.0 

type ClusterHealth struct {
	Cluster                string  `json:"cluster_name"`
	Status                 string  `json:"status"`
	ActiveShards           int     `json:"active_shards"`
	RelocatingShards       int     `json:"relocating_shards"`
	InitializingShards     int     `json:"initializing_shards"`
	UnassignedShards       int     `json:"unassigned_shards"`
	ActiveShardsPercentage float64 `json:"active_shards_percent_as_number"`
	Message                string
	RawIndices             map[string]IndexHealth `json:"indices"`
	HealthyIndices         []IndexHealth
	UnhealthyIndices       []IndexHealth
}

Holds information about the health of an Elasticsearch cluster, based on the cluster health API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-health.html

type ClusterSettings added in v0.1.0 

type ClusterSettings struct {
	PersistentSettings []Setting
	TransientSettings  []Setting
}

Holds slices for persistent and transient cluster settings.

type DiskAllocation added in v0.5.2 

type DiskAllocation struct {
	Name        string `json:"name"`
	Ip          string `json:"ip"`
	Node        string `json:"node"`
	Shards      string `json:"shards"`
	DiskIndices string `json:"disk.indices"`
	DiskUsed    string `json:"disk.used"`
	DiskAvail   string `json:"disk.avail"`
	DiskTotal   string `json:"disk.total"`
	DiskPercent string `json:"disk.percent"`
}

DiskAllocation holds disk allocation information per node, based on _cat/allocationAPI: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-allocation.html

type ExcludeSettings 

type ExcludeSettings struct {
	Ips, Hosts, Names []string
}

Hold the values for what values are in the cluster.allocation.exclude settings. Relevant Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/allocation-filtering.html

type Index added in v0.1.0 

type Index struct {
	Health        string `json:"health"`
	Status        string `json:"status"`
	Name          string `json:"index"`
	PrimaryShards int    `json:"pri,string"`
	ReplicaCount  int    `json:"rep,string"`
	IndexSize     string `json:"store.size"`
	DocumentCount int    `json:"docs.count,string"`
}

Holds information about an Elasticsearch index, based on the _cat/indices API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-indices.html

type IndexHealth added in v0.3.0 

type IndexHealth struct {
	Name               string
	Status             string `json:"status"`
	ActiveShards       int    `json:"active_shards"`
	RelocatingShards   int    `json:"relocating_shards"`
	InitializingShards int    `json:"initializing_shards"`
	UnassignedShards   int    `json:"unassigned_shards"`
}

Holds information about the health of an Elasticsearch index, based on the index level of the cluster health API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-health.html

type Node added in v0.1.0 

type Node struct {
	Name        string `json:"name"`
	Ip          string `json:"ip"`
	Id          string `json:"id"`
	Role        string `json:"role"`
	Master      string `json:"master"`
	Jdk         string `json:"jdk"`
	Version     string `json:"version"`
	Shards      string
	DiskIndices string
	DiskUsed    string
	DiskAvail   string
	DiskTotal   string
	DiskPercent string
}

Holds information about an Elasticsearch node, based on a combination of the _cat/nodes and _cat/allocationAPI: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-nodes.html, https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-allocation.html

type Repository added in v0.3.0 

type Repository struct {
	Name     string
	Type     string
	Settings map[string]interface{}
}

Holds information about an Elasticsearch snapshot repository.

type Setting added in v0.4.0 

type Setting struct {
	Setting string
	Value   string
}

A setting name and value with the setting name to be a "collapsed" version of the setting. A setting of: 

{ "indices": { "recovery" : { "max_bytes_per_sec": "10mb" } } }

would be represented by: 

ClusterSetting{ Setting: "indices.recovery.max_bytes_per_sec", Value: "10mb" }

type Shard added in v0.5.0 

type Shard struct {
	Index string `json:"index"`
	Shard string `json:"shard"`
	Type  string `json:"prirep"`
	State string `json:"state"`
	Docs  string `json:"docs"`
	Store string `json:"store"`
	IP    string `json:"ip"`
	Node  string `json:"node"`
}

Holds information about an index shard, based on the _cat/shards API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-shards.html

type ShardOverlap added in v0.5.0 

type ShardOverlap struct {
	Index         string
	Shard         string
	PrimaryFound  bool
	ReplicasFound int
	ReplicasTotal int
}

Holds information about overlapping shards for a given set of cluster nodes

func (ShardOverlap) SafeToRemove added in v0.5.0 

func (s ShardOverlap) SafeToRemove() bool

Can we safely remove nodes without data loss?

type ShardRecovery added in v0.5.0 

type ShardRecovery struct {
	Index                string `json:"index"`
	Shard                string `json:"shard"`
	Time                 string `json:"time"`
	Type                 string `json:"type"`
	Stage                string `json:"stage"`
	SourceHost           string `json:"source_host"`
	SourceNode           string `json:"source_node"`
	TargetHost           string `json:"target_host"`
	TargetNode           string `json:"target_node"`
	Repository           string `json:"repository"`
	Snapshot             string `json:"snapshot"`
	Files                int    `json:"files,string"`
	FilesRecovered       int    `json:"files_recovered,string"`
	FilesPercent         string `json:"files_percent"`
	FilesTotal           int    `json:"files_total,string"`
	Bytes                int    `json:"bytes,string"`
	BytesRecovered       int    `json:"bytes_recovered,string"`
	BytesPercent         string `json:"bytes_percent"`
	BytesTotal           int    `json:"bytes_total,string"`
	TranslogOps          int    `json:"translog_ops,string"`
	TranslogOpsRecovered int    `json:"translog_ops_recovered,string"`
	TranslogOpsPercent   string `json:"translog_ops_percent"`
}

Holds information about shard recovery based on the _cat/recovery API: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat-recovery.html

func (ShardRecovery) TimeRemaining added in v0.5.0 

func (s ShardRecovery) TimeRemaining() (time.Duration, error)

Estimate time remaining for recovery

type Snapshot added in v0.1.0 

type Snapshot struct {
	State          string    `json:"state"`
	Name           string    `json:"snapshot"`
	StartTime      time.Time `json:"start_time"`
	EndTime        time.Time `json:"end_time"`
	DurationMillis int       `json:"duration_in_millis"`
	Indices        []string  `json:"indices"`
	Shards         struct {
		Total      int `json:"total"`
		Failed     int `json:"failed"`
		Successful int `json:"successful"`
	} `json:"shards"`
	Failures []struct {
		Index   string `json:"index"`
		ShardID int    `json:"shard_id"`
		Reason  string `json:"reason"`
		NodeID  string `json:"node_id"`
		Status  string `json:"status"`
	} `json:"failures"`
}

Holds information about an Elasticsearch snapshot, based on the snapshot API: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

func (*Snapshot) GetDuration added in v0.5.0 

func (s *Snapshot) GetDuration() int

GetDuration gets the total duration of a snapshot

func (*Snapshot) GetEndTime added in v0.5.0 

func (s *Snapshot) GetEndTime() string

GetEndTime gets the time when a snapshot ended

type Token added in v0.4.0 

type Token struct {
	Text        string `json:"token"`
	StartOffset int    `json:"start_offset"`
	EndOffset   int    `json:"end_offset"`
	Type        string `json:"type"`
	Position    int    `json:"position"`
}

Holds information about the tokens that Elasticsearch analyzes

Source Files

View all Source files

Directories

Path Synopsis
cmd
vulcanizer
pkg
cli

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.