esutils

Documentation

Index

• func NewEsClient(config *Config) (*elastic.Client, error)
• type Config
• type ILMPolicy
  • func NewILMPolicy(es *elastic.Client, policyName string) *ILMPolicy
  • func (p *ILMPolicy) DeleteAfter(timeBeforeDelete string) *ILMPolicy
  • func (p *ILMPolicy) FromJSONString(policyJSONStr string) *ILMPolicy
  • func (p *ILMPolicy) Migrate(ctx context.Context) error
  • func (p *ILMPolicy) Rollover(maxSize *string, maxDocs *int, maxAge *string) *ILMPolicy
  • func (p *ILMPolicy) String() string
• type Index
  • func NewIndex(es *elastic.Client) *Index
  • func (i *Index) AddWriteAlias(name string) *Index
  • func (i *Index) FromJSONString(jsonStr string) *Index
  • func (i *Index) Migrate(ctx context.Context) error
  • func (i *Index) Name(indexName string) *Index
• type IndexTemplate
  • func NewIndexTemplate(es *elastic.Client, templateName string) *IndexTemplate
  • func (t *IndexTemplate) AddIndexMappings(mappings map[string]interface{}) *IndexTemplate
  • func (t *IndexTemplate) AddIndexSettings(settings map[string]interface{}) *IndexTemplate
  • func (t *IndexTemplate) AssociateRolloverPolicy(policyName, aliasName string) *IndexTemplate
  • func (t *IndexTemplate) FromJSONString(templJSONStr string) *IndexTemplate
  • func (t *IndexTemplate) Migrate(ctx context.Context) error
• type ManagedIndex
  • func NewManagedIndex(es *elastic.Client, name string) *ManagedIndex
  • func (m *ManagedIndex) ILMPolicy() *ILMPolicy
  • func (m *ManagedIndex) IndexFromJSONString(j string) *ManagedIndex
  • func (m *ManagedIndex) IndexTemplate() *IndexTemplate
  • func (m *ManagedIndex) MaxIndexAge(maxAge string) *ManagedIndex
  • func (m *ManagedIndex) MaxIndexSize(maxSize string) *ManagedIndex
  • func (m *ManagedIndex) Migrate(ctx context.Context) error
  • func (m *ManagedIndex) TimeBeforeDelete(timeBeforeDelete string) *ManagedIndex

Functions

func NewEsClient

func NewEsClient(config *Config) (*elastic.Client, error)

NewEsClient creates an elastic search client from the config.

Types

type Config

type Config struct {
	URL        []string `json:"url"`
	User       string   `json:"user"`
	Passwd     string   `json:"passwd"`
	CaCertFile string   `json:"ca_cert_file"`
}

Config describes the underlying config for elastic.

type ILMPolicy

type ILMPolicy struct {
	// contains filtered or unexported fields
}

ILMPolicy manages the creation/updating of an elastic index lifecycle management policy.

func NewILMPolicy

func NewILMPolicy(es *elastic.Client, policyName string) *ILMPolicy

NewILMPolicy create a new ILMPolicy with the given name and default policy actions.

func (*ILMPolicy) DeleteAfter

func (p *ILMPolicy) DeleteAfter(timeBeforeDelete string) *ILMPolicy

DeleteAfter adds a delete action to the policy's delete phase. This actions causes rolled over indices to be deleted after `timeBeforeDelete` time has passed.

func (*ILMPolicy) FromJSONString

func (p *ILMPolicy) FromJSONString(policyJSONStr string) *ILMPolicy

FromJSONString populates the policy from a marshalled json string. Note that this can be used in conjunction with Rollover and Delete.

func (*ILMPolicy) Migrate

func (p *ILMPolicy) Migrate(ctx context.Context) error

Migrate creates and/or updates the policy in elastic.

func (*ILMPolicy) Rollover

func (p *ILMPolicy) Rollover(maxSize *string, maxDocs *int, maxAge *string) *ILMPolicy

Rollover adds a rollover action to the policy's hot phase. This action causes the indices affected by the policy to rollover to a new index when the conditions specified by maxSize, maxDocs, and maxAge are met. Leaving a parameter as nil will cause that condition to be ignored, eg. Rollover(strPtr("10gb"), nil, nil) will only add a max size condition and not any of the others.

func (*ILMPolicy) String

func (p *ILMPolicy) String() string

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index handles the creation and/or updating of elasticsearch indices.

func NewIndex

func NewIndex(es *elastic.Client) *Index

NewIndex creates a new Index.

func (*Index) AddWriteAlias

func (i *Index) AddWriteAlias(name string) *Index

AddWriteAlias adds an index alias for this index, with is_write_index set to true. Note that only one index per alias can have is_write_index set to true.

func (*Index) FromJSONString

func (i *Index) FromJSONString(jsonStr string) *Index

FromJSONString populates the index spec from a marshalled json string.

func (*Index) Migrate

func (i *Index) Migrate(ctx context.Context) error

Migrate creates and/or updates the elasticsearch index to conform to the spec in i.index. Currently, Migrate does not support reindexing of indices to conform to the new spec. This means certain changes to mappings or settings can't be updated through this migration. But many types of changes to index mappings/settings/aliases are still handled by Migrate.

func (*Index) Name

func (i *Index) Name(indexName string) *Index

Name sets the name of the Index.

type IndexTemplate

type IndexTemplate struct {
	// contains filtered or unexported fields
}

IndexTemplate manages the creation/update of elasticsearch index templates.

func NewIndexTemplate

func NewIndexTemplate(es *elastic.Client, templateName string) *IndexTemplate

NewIndexTemplate creates a new IndexTemplate with the given name.

func (*IndexTemplate) AddIndexMappings

func (t *IndexTemplate) AddIndexMappings(mappings map[string]interface{}) *IndexTemplate

AddIndexMappings populates the mappings of the index template from the map passed. Although, elastic supposedly copies the mappings over when there is an index rollover, there seem to be some cases where this doesn't happen, so we add the index mappings to the index template to make sure it happens.

func (*IndexTemplate) AddIndexSettings

func (t *IndexTemplate) AddIndexSettings(settings map[string]interface{}) *IndexTemplate

AddIndexSettings populates the settings of the index template from the map passed in. I thought that elastic copied over the settings on index rollover, but it seems that doesn't happen sometimes, so copying the settings to the template to make sure that happens.

func (*IndexTemplate) AssociateRolloverPolicy

func (t *IndexTemplate) AssociateRolloverPolicy(policyName, aliasName string) *IndexTemplate

AssociateRolloverPolicy associates an ILMPolicy with this index template. This ensures that all indices that match the index pattern will be under the pervue of the given policy.

func (*IndexTemplate) FromJSONString

func (t *IndexTemplate) FromJSONString(templJSONStr string) *IndexTemplate

FromJSONString populates the index template spec from a marshalled json string. This is useful if more fine-grained control of the index template is required.

func (*IndexTemplate) Migrate

func (t *IndexTemplate) Migrate(ctx context.Context) error

Migrate creates and/or updates the elastic index template to conform to the spec in t.template.

type ManagedIndex

type ManagedIndex struct {
	// contains filtered or unexported fields
}

ManagedIndex creates a lifecycle managed index and handles migrating this index on startup. A ManagedIndex is collection of an elastic index (Index), along with an index template (IndexTemplate), and an elastic index lifecycle management policy (ILMPolicy).

In elastic, indices can be in 1 of 4 phases (hot, warm, cold, delete). The ILMPolicy specifies actions for each phase, and transition conditions for going between the phases. Currently, the main use of the ManagedIndex is to specify transistioning from the hot phase to the delete phase after the index has reached a certain disk size. This is achieved using the "rollover" action on the hot phase, which transistions an index into the next phase after the conditions have been met.

Example usage:

err := NewManagedIndex(es, "my_managed_index").
	IndexFromJSONString(`
		{
			"mappings": {
				"properties": {
					"my_text_field": {
						"type": "text"
					}
				}
			}
		}`).
	MaxIndexSize("1gb").
	TimeBeforeDelete("1d").
	Migrate(ctx)

This example will create a new elastic index called "my_managed_index-000000". This index will have an index alias called "my_managed_index". It will also create an ILMPolicy that specifies that the index should be rolled over once it reaches 1gb in size. When that happens elastic will create a new index "my_managed_index-000001". Then 1 day after this rollover occurs, "my_managed_index-000000" will be deleted. Under the hood, ManagedIndex will also create an IndexTemplate. This is required by elastic so that when the rollover happens the new index "my_managed_index-000001", will still be associated to the ILMPolicy.

Note that since an index alias is created called "my_managed_index", the consumer of this index doesn't need to care about the actual index name, and can just use the managed index's name in all future elastic calls as if it were the index name.

func NewManagedIndex

func NewManagedIndex(es *elastic.Client, name string) *ManagedIndex

NewManagedIndex creates a new ManagedIndex with name.

func (*ManagedIndex) ILMPolicy

func (m *ManagedIndex) ILMPolicy() *ILMPolicy

ILMPolicy gets the ILMPolicy used by the ManagedIndex. This API is provided as a means of getting more fined-grained control over the ilm policy used by the ManagedIndex. Most simple use cases should be able to use the TimeBeforeDelete or MaxIndexSize APIs. Example usage:

myManagedInd := NewManagedIndex(es, "my_managed_ind")
myManagedInd.ILMPolicy().FromJSONString(`{<INSERT_JSON_HERE>}`)
err := myManagedInd.Migrate(ctx)

func (*ManagedIndex) IndexFromJSONString

func (m *ManagedIndex) IndexFromJSONString(j string) *ManagedIndex

IndexFromJSONString sets the index config to be the json string provided. Example usage:

err := NewManagedIndex(es, "my_managed_index").IndexFromJSONString(
`
{
		"mappings": <INSERT_MAPPINGS_HERE>,
		"setings": <INSERT_SETTINGS_HERE>,
}
`
).Migrate(ctx)

Note that as opposed to ILMPolicy(), creating the index config from json is likely to be done for every ManagedIndex so its a top level API. This will also add the mappings from the JSON string to the IndexTemplate.

func (*ManagedIndex) IndexTemplate

func (m *ManagedIndex) IndexTemplate() *IndexTemplate

IndexTemplate gets the index template used by the ManagedIndex. This API is provided as a means of getting more fined-grained control over the index template used by the ManagedIndex. This API shouldn't be necessary for most simple use cases. Example usage:

myManagedInd := NewManagedIndex(es, "my_managed_ind")
myManagedInd.IndexTemplate().FromJSONString(`{<INSERT_JSON_HERE>}`)
err := myManagedInd.Migrate(ctx)

func (*ManagedIndex) MaxIndexAge

func (m *ManagedIndex) MaxIndexAge(maxAge string) *ManagedIndex

MaxIndexAge sets how long the index will live until its rolled over. Rolled over indices can still be accessed until the Delete condition is reached. It should be specified as a string, eg. "30d".

func (*ManagedIndex) MaxIndexSize

func (m *ManagedIndex) MaxIndexSize(maxSize string) *ManagedIndex

MaxIndexSize sets the size to allow the index to grow to before rollover. the size should be specified as a string with unit, eg. "50gb".

func (*ManagedIndex) Migrate

func (m *ManagedIndex) Migrate(ctx context.Context) error

Migrate attempts to migrate from the current managed index to the new specification. See ilm_policy.go, index_template.go, and index.go for information about migrations of each of the corresponding objects.

func (*ManagedIndex) TimeBeforeDelete

func (m *ManagedIndex) TimeBeforeDelete(timeBeforeDelete string) *ManagedIndex

TimeBeforeDelete sets the amount of time that should be waited after rollover, before deleting the index. The time should be specified as a string with unit, eg. "1d".