export

README

influx-tools export

Used with influx-tools import, the export tool transforms existing shards to a new shard duration in order to consolidate into fewer shards. It is also possible to separate into a greater number of shards.

Field type conflicts

A field type for a given measurement can be different per shard. This creates the potential for field type conflicts when exporting new shard durations. If this happens, the field type will be determined by the first shard containing data for that field in order to fulfil the target shard duration. All conflicting data will be written as line protocol and gzipped to the path specified by -conflict-path, unless the -no-conflict-path option is specified.

Range

The optional -range <range> option specifies which target shards should be exported, based on their sequence number. A use case for the range option is to parallelize the reshaping of a large data set. A machine with 4 cores could run two export / import jobs concurrently over a subset of the total target shards.

The sequence number is included in the plan output. For example:

$ influx-tools export -config config.toml -database foo -rp autogen -duration 24h -format=binary -no-conflict-path -print-only
Source data from: 2018-02-19 00:00:00 +0000 UTC -> 2018-04-09 00:00:00 +0000 UTC

Converting source from 4 shard group(s) to 28 shard groups

Seq #           ID              Start                           End
0               608             2018-02-19 00:00:00 +0000 UTC   2018-02-26 00:00:00 +0000 UTC
1               609             2018-03-19 00:00:00 +0000 UTC   2018-03-26 00:00:00 +0000 UTC
2               610             2018-03-26 00:00:00 +0000 UTC   2018-04-02 00:00:00 +0000 UTC
3               612             2018-04-02 00:00:00 +0000 UTC   2018-04-09 00:00:00 +0000 UTC

Seq #           ID              Start                           End
0               0               2018-02-19 00:00:00 +0000 UTC   2018-02-20 00:00:00 +0000 UTC
1               1               2018-02-20 00:00:00 +0000 UTC   2018-02-21 00:00:00 +0000 UTC
2               2               2018-02-21 00:00:00 +0000 UTC   2018-02-22 00:00:00 +0000 UTC
3               3               2018-02-22 00:00:00 +0000 UTC   2018-02-23 00:00:00 +0000 UTC
4               4               2018-02-23 00:00:00 +0000 UTC   2018-02-24 00:00:00 +0000 UTC
5               5               2018-02-24 00:00:00 +0000 UTC   2018-02-25 00:00:00 +0000 UTC
6               6               2018-02-25 00:00:00 +0000 UTC   2018-02-26 00:00:00 +0000 UTC
7               28              2018-03-19 00:00:00 +0000 UTC   2018-03-20 00:00:00 +0000 UTC
8               29              2018-03-20 00:00:00 +0000 UTC   2018-03-21 00:00:00 +0000 UTC
9               30              2018-03-21 00:00:00 +0000 UTC   2018-03-22 00:00:00 +0000 UTC
...
26              47              2018-04-07 00:00:00 +0000 UTC   2018-04-08 00:00:00 +0000 UTC
27              48              2018-04-08 00:00:00 +0000 UTC   2018-04-09 00:00:00 +0000 UTC

Adding -range 2-4 would return the following plan:

$ influx-tools export -config config.toml -database foo -rp autogen -duration 24h -format=binary -no-conflict-path -print-only -range=2-4
Source data from: 2018-02-19 00:00:00 +0000 UTC -> 2018-04-09 00:00:00 +0000 UTC

Converting source from 4 shard group(s) to 3 shard groups

Seq #           ID              Start                           End
0               608             2018-02-19 00:00:00 +0000 UTC   2018-02-26 00:00:00 +0000 UTC
1               609             2018-03-19 00:00:00 +0000 UTC   2018-03-26 00:00:00 +0000 UTC
2               610             2018-03-26 00:00:00 +0000 UTC   2018-04-02 00:00:00 +0000 UTC
3               612             2018-04-02 00:00:00 +0000 UTC   2018-04-09 00:00:00 +0000 UTC

Seq #           ID              Start                           End
2               2               2018-02-21 00:00:00 +0000 UTC   2018-02-22 00:00:00 +0000 UTC
3               3               2018-02-22 00:00:00 +0000 UTC   2018-02-23 00:00:00 +0000 UTC
4               4               2018-02-23 00:00:00 +0000 UTC   2018-02-24 00:00:00 +0000 UTC

A range can either be a single sequence number or an interval as shown previously.

Hint: Include the -print-only option to display the plan and exit without exporting any data.

Example usage for Enterprise

Note: This is only for fully replicated clusters. For non-fully-replicated clusters (RF < # of nodes), the current best method would be exporting in line protocol and re-importing after deleting the old shards.

This example takes many small shards and collapses them to fewer large shards, which can be beneficial for TSI index performance.

The influx_tools export and influx_tools import tools run as if they were a full data node - to run either, the normal data node influxd process must be down, and the meta nodes must be running. Ensure any environment variables and config file settings for the real influxd are reflected in the config file provided to influx_tools

Create the export

It is recommended to do this step on a running backup of the cluster rather than the real cluster.

The -duration must be an integer multiple of the shard group duration being collapsed, to avoid overlapping shards and import problems. For example, 4032h = 24*168h = 24 weeks.

It is recommended to run a dry run with -print-only before running the real export.

On one of the nodes (of the running backup), bring down the influxd process and run:

$ ./influx_tools export -config /root/.influxdb/influxdb.conf -database collapsedb -duration 4032h -format binary -conflict-path /conflicts.tar.gz -print-only
Source data from: 2020-05-25 00:00:00 +0000 UTC -> 2021-07-19 00:00:00 +0000 UTC

Converting source from 60 shard group(s) to 3 shard groups:

Seq #           ID         Start                                End
0               9          2020-05-25 00:00:00 +0000 UTC        2020-06-01 00:00:00 +0000 UTC
1               10         2020-06-01 00:00:00 +0000 UTC        2020-06-08 00:00:00 +0000 UTC
2               11         2020-06-08 00:00:00 +0000 UTC        2020-06-15 00:00:00 +0000 UTC
3               12         2020-06-15 00:00:00 +0000 UTC        2020-06-22 00:00:00 +0000 UTC
4               13         2020-06-22 00:00:00 +0000 UTC        2020-06-29 00:00:00 +0000 UTC
5               14         2020-06-29 00:00:00 +0000 UTC        2020-07-06 00:00:00 +0000 UTC
…
54              63         2021-06-07 00:00:00 +0000 UTC        2021-06-14 00:00:00 +0000 UTC
55              64         2021-06-14 00:00:00 +0000 UTC        2021-06-21 00:00:00 +0000 UTC
56              65         2021-06-21 00:00:00 +0000 UTC        2021-06-28 00:00:00 +0000 UTC
57              66         2021-06-28 00:00:00 +0000 UTC        2021-07-05 00:00:00 +0000 UTC
58              67         2021-07-05 00:00:00 +0000 UTC        2021-07-12 00:00:00 +0000 UTC
59              68         2021-07-12 00:00:00 +0000 UTC        2021-07-19 00:00:00 +0000 UTC

Seq #           ID         Start                                End
0               0          2020-04-06 00:00:00 +0000 UTC        2020-09-21 00:00:00 +0000 UTC
1               1          2020-09-21 00:00:00 +0000 UTC        2021-03-08 00:00:00 +0000 UTC
2               2          2021-03-08 00:00:00 +0000 UTC        2021-08-23 00:00:00 +0000 UTC

To run just one of shard collapses, use the -range argument, like -range '0-0' (the range is inclusive). The range refers to the output shard sequence numbers (the first column of the second table in the plan output).

Once happy with the plan, run without -print-only (make sure to pipe to a file!):

$ ./influx_tools export -config /root/.influxdb/influxdb.conf -database collapsedb -duration 4032h -format binary -conflict-path /conflicts.tar.gz -range '0-0' > export.binary
Source data from: 2020-05-25 00:00:00 +0000 UTC -> 2021-07-19 00:00:00 +0000 UTC

Converting source from 60 shard group(s) to 1 shard groups:

Seq #           ID              Start                           End
0               9               2020-05-25 00:00:00 +0000 UTC   2020-06-01 00:00:00 +0000 UTC
1               10              2020-06-01 00:00:00 +0000 UTC   2020-06-08 00:00:00 +0000 UTC
2               11              2020-06-08 00:00:00 +0000 UTC   2020-06-15 00:00:00 +0000 UTC
3               12              2020-06-15 00:00:00 +0000 UTC   2020-06-22 00:00:00 +0000 UTC
…

Import the previously exported data

Importing the data requires a period of data unavailability.

1. For safety, take a full database backup - see https://docs.influxdata.com/enterprise_influxdb/v1.9/administration/backup-and-restore/
2. With all nodes running, delete all the shards that would overlap with the newly created shard(s):

    i) The output from the export plan (the ‘ID’ column) gives the shard group id’s:

> ./influx_tools export ...
...
Seq #           ID              Start                           End
0               9               2020-05-25 00:00:00 +0000 UTC   2020-06-01 00:00:00 +0000 UTC
                ^
                Shard group ID


    ii) The 'SHOW SHARDS' command gives the shard id's:

> influx -execute 'show shards'
name: _internal
id database  retention_policy shard_group start_time           end_time             expiry_time          owners
-- --------  ---------------- ----------- ----------           --------             -----------          ------
1  _internal monitor          1           2021-07-15T00:00:00Z 2021-07-16T00:00:00Z 2021-07-23T00:00:00Z 5
2  _internal monitor          1           2021-07-15T00:00:00Z 2021-07-16T00:00:00Z 2021-07-23T00:00:00Z 4
14 _internal monitor          8           2021-07-16T00:00:00Z 2021-07-17T00:00:00Z 2021-07-24T00:00:00Z 5
15 _internal monitor          8           2021-07-16T00:00:00Z 2021-07-17T00:00:00Z 2021-07-24T00:00:00Z 4
^                             ^
Shard id                      Shard group id

    iii) The influxql 'DROP SHARD' command must be used to drop each old shard id to be replaced.
> influx -execute 'drop shard 14'

1. Take one node down, and use influx_tools import on that node to import the data, similar to how influx_tools export was used on the running backup . Note -shard-duration is actually the shard group duration, and duration should match the existing time to expiry (default is infinite). If importing to an existing retention policy, it must have a matching shard group duration, shard duration, and replication factor (otherwise you will get an error import: retention policy autogen already exists with different parameters). Use alter retention policy ... to fix the existing policy if necessary.

$ ./influx_tools import -config /root/.influxdb/influxdb.conf -database collapsedb -rp autogen -shard-duration 4032h < export.binary
{"level":"info","ts":1626446664.3253446,"caller":"importer/command.go:64","msg":"import starting"}
{"level":"info","ts":1626446664.3271663,"caller":"importer/importer.go:107","msg":"Starting shard group 2020-04-06 00:00:00 +0000 UTC-2020-09-21 00:00:00 +0000 UTC with 0 existing shard groups"}

1. Bring the node back up, and use influxd-ctl copy-shard to copy the new shard(s) to the other cluster node:

$ ./influxd-ctl copy-shard latest_data_0_1:8088 latest_data_1_1:8088 71
Copied shard 71 from latest_data_0_1:8088 to latest_data_1_1:8088

Documentation

Index

• type Command
  • func NewCommand(server server.Interface) *Command
  • func (cmd *Command) Run(args []string) (err error)

Types

type Command

type Command struct {
	// Standard input/output, overridden for testing.
	Stderr io.Writer
	Stdout io.Writer
	Logger *zap.Logger
	// contains filtered or unexported fields
}

Command represents the program execution for "store query".

func NewCommand

func NewCommand(server server.Interface) *Command

NewCommand returns a new instance of the export Command.

func (*Command) Run

func (cmd *Command) Run(args []string) (err error)

Run executes the export command using the specified args.