README ¶
influx-tools export
Used with influx-tools import
, the export tool transforms existing shards to a new shard duration in order to
consolidate into fewer shards. It is also possible to separate into a greater number of shards.
Field type conflicts
A field type for a given measurement can be different per shard. This creates the potential for field type conflicts
when exporting new shard durations. If this happens, the field type will be determined by the first shard containing
data for that field in order to fulfil the target shard duration. All conflicting data will be written as line protocol
and gzipped to the path specified by -conflict-path
, unless the -no-conflict-path
option is specified.
Range
The optional -range <range>
option specifies which target shards should be exported, based on their sequence number.
A use case for the range
option is to parallelize the reshaping of a large data set. A machine with 4 cores could run two
export / import jobs concurrently over a subset of the total target shards.
The sequence number is included in the plan output. For example:
$ influx-tools export -config config.toml -database foo -rp autogen -duration 24h -format=binary -no-conflict-path -print-only
Source data from: 2018-02-19 00:00:00 +0000 UTC -> 2018-04-09 00:00:00 +0000 UTC
Converting source from 4 shard group(s) to 28 shard groups
Seq # ID Start End
0 608 2018-02-19 00:00:00 +0000 UTC 2018-02-26 00:00:00 +0000 UTC
1 609 2018-03-19 00:00:00 +0000 UTC 2018-03-26 00:00:00 +0000 UTC
2 610 2018-03-26 00:00:00 +0000 UTC 2018-04-02 00:00:00 +0000 UTC
3 612 2018-04-02 00:00:00 +0000 UTC 2018-04-09 00:00:00 +0000 UTC
Seq # ID Start End
0 0 2018-02-19 00:00:00 +0000 UTC 2018-02-20 00:00:00 +0000 UTC
1 1 2018-02-20 00:00:00 +0000 UTC 2018-02-21 00:00:00 +0000 UTC
2 2 2018-02-21 00:00:00 +0000 UTC 2018-02-22 00:00:00 +0000 UTC
3 3 2018-02-22 00:00:00 +0000 UTC 2018-02-23 00:00:00 +0000 UTC
4 4 2018-02-23 00:00:00 +0000 UTC 2018-02-24 00:00:00 +0000 UTC
5 5 2018-02-24 00:00:00 +0000 UTC 2018-02-25 00:00:00 +0000 UTC
6 6 2018-02-25 00:00:00 +0000 UTC 2018-02-26 00:00:00 +0000 UTC
7 28 2018-03-19 00:00:00 +0000 UTC 2018-03-20 00:00:00 +0000 UTC
8 29 2018-03-20 00:00:00 +0000 UTC 2018-03-21 00:00:00 +0000 UTC
9 30 2018-03-21 00:00:00 +0000 UTC 2018-03-22 00:00:00 +0000 UTC
...
26 47 2018-04-07 00:00:00 +0000 UTC 2018-04-08 00:00:00 +0000 UTC
27 48 2018-04-08 00:00:00 +0000 UTC 2018-04-09 00:00:00 +0000 UTC
Adding -range 2-4
would return the following plan:
$ influx-tools export -config config.toml -database foo -rp autogen -duration 24h -format=binary -no-conflict-path -print-only -range=2-4
Source data from: 2018-02-19 00:00:00 +0000 UTC -> 2018-04-09 00:00:00 +0000 UTC
Converting source from 4 shard group(s) to 3 shard groups
Seq # ID Start End
0 608 2018-02-19 00:00:00 +0000 UTC 2018-02-26 00:00:00 +0000 UTC
1 609 2018-03-19 00:00:00 +0000 UTC 2018-03-26 00:00:00 +0000 UTC
2 610 2018-03-26 00:00:00 +0000 UTC 2018-04-02 00:00:00 +0000 UTC
3 612 2018-04-02 00:00:00 +0000 UTC 2018-04-09 00:00:00 +0000 UTC
Seq # ID Start End
2 2 2018-02-21 00:00:00 +0000 UTC 2018-02-22 00:00:00 +0000 UTC
3 3 2018-02-22 00:00:00 +0000 UTC 2018-02-23 00:00:00 +0000 UTC
4 4 2018-02-23 00:00:00 +0000 UTC 2018-02-24 00:00:00 +0000 UTC
A range can either be a single sequence number or an interval as shown previously.
Hint: Include the -print-only
option to display the plan and exit without exporting any data.
Example usage for Enterprise
Note: This is only for fully replicated clusters. For non-fully-replicated clusters (RF < # of nodes), the current best method would be exporting in line protocol and re-importing after deleting the old shards.
This example takes many small shards and collapses them to fewer large shards, which can be beneficial for TSI index performance.
The influx_tools export
and influx_tools import
tools run as if they were a full data node - to run either, the normal
data node influxd
process must be down, and the meta nodes must be running. Ensure any environment
variables and config file settings for the real influxd
are reflected in the config file provided to
influx_tools
Create the export
It is recommended to do this step on a running backup of the cluster rather than the real cluster.
The -duration
must be an integer multiple of the shard group duration being
collapsed, to avoid overlapping shards and import problems. For example, 4032h = 24*168h = 24 weeks.
It is recommended to run a dry run with -print-only
before running the real export.
On one of the nodes (of the running backup), bring down the influxd process and run:
$ ./influx_tools export -config /root/.influxdb/influxdb.conf -database collapsedb -duration 4032h -format binary -conflict-path /conflicts.tar.gz -print-only
Source data from: 2020-05-25 00:00:00 +0000 UTC -> 2021-07-19 00:00:00 +0000 UTC
Converting source from 60 shard group(s) to 3 shard groups:
Seq # ID Start End
0 9 2020-05-25 00:00:00 +0000 UTC 2020-06-01 00:00:00 +0000 UTC
1 10 2020-06-01 00:00:00 +0000 UTC 2020-06-08 00:00:00 +0000 UTC
2 11 2020-06-08 00:00:00 +0000 UTC 2020-06-15 00:00:00 +0000 UTC
3 12 2020-06-15 00:00:00 +0000 UTC 2020-06-22 00:00:00 +0000 UTC
4 13 2020-06-22 00:00:00 +0000 UTC 2020-06-29 00:00:00 +0000 UTC
5 14 2020-06-29 00:00:00 +0000 UTC 2020-07-06 00:00:00 +0000 UTC
…
54 63 2021-06-07 00:00:00 +0000 UTC 2021-06-14 00:00:00 +0000 UTC
55 64 2021-06-14 00:00:00 +0000 UTC 2021-06-21 00:00:00 +0000 UTC
56 65 2021-06-21 00:00:00 +0000 UTC 2021-06-28 00:00:00 +0000 UTC
57 66 2021-06-28 00:00:00 +0000 UTC 2021-07-05 00:00:00 +0000 UTC
58 67 2021-07-05 00:00:00 +0000 UTC 2021-07-12 00:00:00 +0000 UTC
59 68 2021-07-12 00:00:00 +0000 UTC 2021-07-19 00:00:00 +0000 UTC
Seq # ID Start End
0 0 2020-04-06 00:00:00 +0000 UTC 2020-09-21 00:00:00 +0000 UTC
1 1 2020-09-21 00:00:00 +0000 UTC 2021-03-08 00:00:00 +0000 UTC
2 2 2021-03-08 00:00:00 +0000 UTC 2021-08-23 00:00:00 +0000 UTC
To run just one of shard collapses, use the -range
argument, like -range '0-0'
(the range is inclusive).
The range refers to the output shard sequence numbers (the first column of the second table in the plan output).
Once happy with the plan, run without -print-only
(make sure to pipe to a file!):
$ ./influx_tools export -config /root/.influxdb/influxdb.conf -database collapsedb -duration 4032h -format binary -conflict-path /conflicts.tar.gz -range '0-0' > export.binary
Source data from: 2020-05-25 00:00:00 +0000 UTC -> 2021-07-19 00:00:00 +0000 UTC
Converting source from 60 shard group(s) to 1 shard groups:
Seq # ID Start End
0 9 2020-05-25 00:00:00 +0000 UTC 2020-06-01 00:00:00 +0000 UTC
1 10 2020-06-01 00:00:00 +0000 UTC 2020-06-08 00:00:00 +0000 UTC
2 11 2020-06-08 00:00:00 +0000 UTC 2020-06-15 00:00:00 +0000 UTC
3 12 2020-06-15 00:00:00 +0000 UTC 2020-06-22 00:00:00 +0000 UTC
…
Import the previously exported data
Importing the data requires a period of data unavailability.
- For safety, take a full database backup - see https://docs.influxdata.com/enterprise_influxdb/v1.9/administration/backup-and-restore/
- With all nodes running, delete all the shards that would overlap with the newly created shard(s):
i) The output from the export plan (the ‘ID’ column) gives the shard group id’s:
> ./influx_tools export ...
...
Seq # ID Start End
0 9 2020-05-25 00:00:00 +0000 UTC 2020-06-01 00:00:00 +0000 UTC
^
Shard group ID
ii) The 'SHOW SHARDS' command gives the shard id's:
> influx -execute 'show shards'
name: _internal
id database retention_policy shard_group start_time end_time expiry_time owners
-- -------- ---------------- ----------- ---------- -------- ----------- ------
1 _internal monitor 1 2021-07-15T00:00:00Z 2021-07-16T00:00:00Z 2021-07-23T00:00:00Z 5
2 _internal monitor 1 2021-07-15T00:00:00Z 2021-07-16T00:00:00Z 2021-07-23T00:00:00Z 4
14 _internal monitor 8 2021-07-16T00:00:00Z 2021-07-17T00:00:00Z 2021-07-24T00:00:00Z 5
15 _internal monitor 8 2021-07-16T00:00:00Z 2021-07-17T00:00:00Z 2021-07-24T00:00:00Z 4
^ ^
Shard id Shard group id
iii) The influxql 'DROP SHARD' command must be used to drop each old shard id to be replaced.
> influx -execute 'drop shard 14'
- Take one node down, and use
influx_tools import
on that node to import the data, similar to howinflux_tools export
was used on the running backup . Note-shard-duration
is actually the shard group duration, andduration
should match the existing time to expiry (default is infinite). If importing to an existing retention policy, it must have a matching shard group duration, shard duration, and replication factor (otherwise you will get an errorimport: retention policy autogen already exists with different parameters
). Usealter retention policy ...
to fix the existing policy if necessary.
$ ./influx_tools import -config /root/.influxdb/influxdb.conf -database collapsedb -rp autogen -shard-duration 4032h < export.binary
{"level":"info","ts":1626446664.3253446,"caller":"importer/command.go:64","msg":"import starting"}
{"level":"info","ts":1626446664.3271663,"caller":"importer/importer.go:107","msg":"Starting shard group 2020-04-06 00:00:00 +0000 UTC-2020-09-21 00:00:00 +0000 UTC with 0 existing shard groups"}
- Bring the node back up, and use
influxd-ctl copy-shard
to copy the new shard(s) to the other cluster node:
$ ./influxd-ctl copy-shard latest_data_0_1:8088 latest_data_1_1:8088 71
Copied shard 71 from latest_data_0_1:8088 to latest_data_1_1:8088
Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Command ¶
type Command struct { // Standard input/output, overridden for testing. Stderr io.Writer Stdout io.Writer Logger *zap.Logger // contains filtered or unexported fields }
Command represents the program execution for "store query".
func NewCommand ¶
NewCommand returns a new instance of the export Command.