Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
View Source
var Cmd = &charm.Spec{ Name: "zq", Usage: "zq [ options ] [ zed-query ] file [ file ... ]", Short: "process data with Zed queries", HiddenFlags: "cpuprofile,memprofile,pathregexp", Long: ` "zq" is a command-line tool for processing data in diverse input formats, providing search, analytics, and extensive transormations using the Zed query language. A query typically applies Boolean logic or keyword search to filter the input and then transforms or analyzes the filtered stream. Output is written to one or more files or to standard output. A Zed query is comprised of one or more operators interconnected into a pipeline using the Unix pipe charcter "|". See https://github.com/brimdata/zed/tree/main/docs/language for details. Supported input formats include CSV, JSON, NDJSON, Parquet, ZSON, ZNG, ZST, and Zeek TSV. Supported output formats include all the input formats along with a SQL-like table format. "zq" must be run with at least one input. Input files can be file system paths; "-" for standard input; or HTTP, HTTPS, or S3 URLs. For most types of data, the input format is automatically detected. If multiple files are specified, each file format is determined independently so you can mix and match input types. If multiple files are concatenated into a stream and presented as standard input, the files must all be of the same type as the beginning of stream will determine the format. Output is sent to standard output unless an output file is specified with -o. Some output formats like Parquet are based on schemas and require all data in the output to conform to the same schema. To handle this, you can either fuse the data into a union of all the record types present (presuming all the output values are records) or you can specify the -split flag to indicate a destination directory for separate output files for each output type. This flag may be used in combination with -o, which provides the prefix for the file path, e.g., zq -f parquet -split out -o example-output input.zng When writing to stdout and stdout is a terminal, the default output format is ZSON. Otherwise, the default format is binary ZNG. In either case, the default may be overridden with -f, -z, or -Z. After the options, a Zed "query" string may be specified as a single argument conforming to the Zed language syntax; i.e., it should be quoted as a single string in the shell. If the first argument is a path to a valid file rather than a Zed query, then the Zed query is assumed to be "*", i.e., match and output all of the input. If the first argument is both a valid Zed query and an existing file, then the file overrides. The Zed query text may include source files using -I, which is particularly convenient when a large, complex query spans multiple lines. In this case, these source files are concatenated together along with the command-line query text in the order appearing on the command line. The "zq" engine processes data natively in Zed so if you intend to run many queries over the same data, you will see substantial performance gains by converting your data to the efficient binary form of Zed called ZNG, e.g., zq -f zng input.json > fast.zng zq <query> fast.zng ... Please see https://github.com/brimdata/zq and https://github.com/brimdata/zed for more information. `, New: New, }
Functions ¶
Types ¶
Click to show internal directories.
Click to hide internal directories.