influxdb-csv-import

README

influxdb-csv-import

CSV data written to influx

https://github.com/influxdata/influxdb/issues/17003 introduces a new CSV format for existing influx write command. CSV data on input are transformed to line protocol with the help of CSV annotations.

CSV Annotations

• https://v2.docs.influxdata.com/v2.0/reference/syntax/annotated-csv/#annotations
  • all of them are supported
• additionally
  • column names that start with _ are OOTB ignored, unless: _measurement, _time, _field, _value
    • _measurement: measurement part
    • _time: timestamp part
    • _field: column that contains field name
    • _value: column that contains field value
  • #linetype annotation associated a particular csv column protocol with a line part
    • supported values are: measurement, tag, time, ignore(d)
    • default is =field= unless _field column is present (ignored then)
  • time column can be specified as an int64 number or in RFC3339 format

DRY RUN

A new --dry-run flag helps to validate and tune CSV data.

Example 1 - Flux Query Result

influx write --file --dry-run doc/examples/fluxQueryResult.csv

#group,false,false,true,true,false,false,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string
#default,_result,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,cpu,host
,,0,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:17:57Z,0,time_steal,cpu,cpu1,rsavage.prod
,,0,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:18:07Z,0,time_steal,cpu,cpu1,rsavage.prod

#group,false,false,true,true,false,false,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string
#default,_result,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,cpu,host
,,1,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:18:01Z,2.7263631815907954,usage_user,cpu,cpu-total,tahoecity.prod
,,1,2020-02-25T22:17:54.068926364Z,2020-02-25T22:22:54.068926364Z,2020-02-25T22:18:11Z,2.247752247752248,usage_user,cpu,cpu-total,tahoecity.prod

line protocol data:

cpu,cpu=cpu1,host=rsavage.prod time_steal=0 1582669077000000000
cpu,cpu=cpu1,host=rsavage.prod time_steal=0 1582669087000000000
cpu,cpu=cpu-total,host=tahoecity.prod usage_user=2.7263631815907954 1582669081000000000
cpu,cpu=cpu-total,host=tahoecity.prod usage_user=2.247752247752248 1582669091000000000

Example 2 - Simple Annotated CSV file

influx write --dry-run --file doc/examples/annotatedLinepart.csv

#linepart measurement,tag,tag,field,field,time
m,cpu,host,time_steal,usage_user,time
cpu,cpu1,rsavage.prod,0,2.7,1482669077000000000
cpu,cpu1,rsavage.prod,0,2.2,1482669087000000000

line protocol data:

cpu,cpu=cpu1,host=rsavage.prod time_steal=0,usage_user=2.7 1482669077000000000
cpu,cpu=cpu1,host=rsavage.prod time_steal=0,usage_user=2.2 1482669087000000000

Note that all fields are of type double.

Example 3 - Annotated CSV file with Data Types

influx write --dry-run --file doc/examples/annotatedDatatype.csv

#datatype ,,string,double,boolean,long,unsignedLong,duration,
#linepart measurement,tag,,,,,,,time
#default test,annotatedDatatypes,,,,,,
m,name,s,d,b,l,ul,dur,time
,,str1,1.0,true,1,1,1ms,1
,,str2,2.0,false,2,2,2us,2020-01-11T10:10:10Z

line protocol data:

test,name=annotatedDatatypes s="str1",d=1,b=true,l=1i,ul=1u,dur=1000000i 1
test,name=annotatedDatatypes s="str2",d=2,b=false,l=2i,ul=2u,dur=2000i 1578737410000000000

Documentation

Overview

Copyright © 2020 pavel.zavora@bonitoo.io

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.