miller

module

v6.13.0 Latest Latest Go to latest Published: Oct 5, 2024 License: BSD-2-Clause

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/johnkerl/miller

Links

Open Source Insights

README ¶

What is Miller?

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.

What can Miller do for me?

With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more.

cover-art

Miller operates on key-value-pair data while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map.
Miller handles a variety of data formats, including but not limited to the familiar CSV, TSV, and JSON/JSON Lines. (Miller can handle positionally-indexed data too!)

In the above image you can see how Miller embraces the common themes of key-value-pair data in a variety of data formats.

Getting started

Installing

There's a good chance you can get Miller pre-built for your system:

OS	Installation command
Linux	`yum install miller` `apt-get install miller`
Mac	`brew install miller` `port install miller`
Windows	`choco install miller` `winget install Miller.Miller`

See also README-versions.md for a full list of package versions. Note that long-term-support (LtS) releases will likely be on older versions.

Community

Discussion forum: https://github.com/johnkerl/miller/discussions
Feature requests / bug reports: https://github.com/johnkerl/miller/issues
How to contribute: https://miller.readthedocs.io/en/latest/contributing/

Build status

Building from source

First:
- cd /where/you/want/to/put/the/source
- git clone https://github.com/johnkerl/miller
- cd miller
With make:
- To build: make. This takes just a few seconds and produces the Miller executable, which is ./mlr (or .\mlr.exe on Windows).
- To run tests: make check.
- To install: make install. This installs the executable /usr/local/bin/mlr and manual page /usr/local/share/man/man1/mlr.1 (so you can do man mlr).
- You can do ./configure --prefix=/some/install/path before make install if you want to install somewhere other than /usr/local.
Without make:
- To build: go build github.com/johnkerl/miller/v6/cmd/mlr.
- To run tests: go test github.com/johnkerl/miller/v6/pkg/... and mlr regtest.
- To install: go install github.com/johnkerl/miller/v6/cmd/mlr will install to GOPATH/bin/mlr.
See also the doc page on building from source.
For more developer information please see README-dev.md.

For developers

License

License: BSD2

Features

Miller is multi-purpose: it's useful for data cleaning, data reduction, statistical reporting, devops, system administration, log-file processing, format conversion, and database-query post-processing.
You can use Miller to snarf and munge log-file data, including selecting out relevant substreams, then produce CSV format and load that into all-in-memory/data-frame utilities for further statistical and/or graphical processing.
Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data. While you can do basic statistics entirely in Miller, its streaming-data feature and single-pass algorithms enable you to reduce very large data sets.
Miller complements SQL databases: you can slice, dice, and reformat data on the client side on its way into or out of a database. You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
Miller also goes beyond the classic Unix tools by stepping fully into our modern, no-SQL world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.
Miller is streaming: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (sort, tac, stats1), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system’s available RAM, and you can use Miller in tail -f contexts.
Miller is pipe-friendly and interoperates with the Unix toolkit.
Miller's I/O formats include tabular pretty-printing, positionally indexed (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others.
Miller does conversion between formats.
Miller's processing is format-aware: e.g. CSV sort and tac keep header lines first.
Miller has high-throughput performance on par with the Unix toolkit.
Miller is written in portable, modern Go, with zero runtime dependencies. You can download or compile a single binary, scp it to a faraway machine, and expect it to work.

What people are saying about Miller

Today I discovered Miller—it's like jq but for CSV: https://t.co/pn5Ni241KM

Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." @GreatBlueC @nfmcclure
— Adrien Trouillaud (@adrienjt) September 24, 2020

Underappreciated swiss-army command-line chainsaw.

"Miller is like awk, sed, cut, join, and sort for [...] CSV, TSV, and [...] JSON." https://t.co/TrQqSUK3KK
— Dirk Eddelbuettel (@eddelbuettel) February 28, 2017

Miller looks like a great command line tool for working with CSV data. Sed, awk, cut, join all rolled into one: http://t.co/9BBb6VCZ6Y
— Mike Loukides (@mikeloukides) August 16, 2015

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV: http://t.co/1zPbfg6B2W - handy tool!
— Ilya Grigorik (@igrigorik) August 22, 2015

Btw, I think Miller is the best CLI tool to deal with CSV. I used to use this when I need to preprocess too big CSVs to load into R (now we have vroom, so such cases might be rare, though...)https://t.co/kUjrSSGJoT
— Hiroaki Yutani (@yutannihilat_en) April 21, 2020

Miller: a *format-aware* data munging tool By @__jo_ker__ to overcome limitations with *line-aware* workshorses like awk, sed et al https://t.co/LCyPkhYvt9

The project website is a fantastic example of good software documentation!!
— Donny Daniel (@dnnydnl) September 9, 2018

Holy holly data swiss army knife batman! How did no one suggest Miller https://t.co/JGQpmRAZLv for solving database cleaning / ETL issues to me before

Congrats to @__jo_ker__ for amazingly intuitive tool for critical data management tasks!#DataScienceandLaw #ComputationalLaw
— James Miller (@japanlawprof) June 12, 2018

🤯@__jo_ker__'s Miller easily reads, transforms, + writes all sorts of tabular data. It's standalone, fast, and built for streaming data (operating on one line at a time, so you can work on files larger than memory).

And the docs are dream. I've been reading them all morning! https://t.co/Be2pGPZK6t
— Benjamin Wolfe (he/him) (@BenjaminWolfe) September 9, 2021

Contributors ✨

Thanks to all the fine people who help make Miller better (emoji key):

_{Andrea Borruso} 🤔 🎨	_{Shaun Jackman} 🤔	_{Fred Trotter} 🤔 🎨	_komosa 🤔	_{jungle-boogie} 🤔	_{Thomas Klausner} 🚇	_{Stephen Kitt} 📦
_{Leah Neukirchen} 🤔	_{Luigi Baldoni} 📦	_{Hiroaki Yutani} 🤔	_{Daniel M. Drucker} 🤔	_{Nikos Alexandris} 🤔	_kundeng 📦	_{Victor Sergienko} 📦
_{Adrian Ho} 🎨	_zachp 📦	_{David Selassie} 🤔	_{Joel Parker Henderson} 🤔	_{Michel Ace} 🤔	_{Matus Goljer} 🤔	_{Richard Patel} 📦
_{Jakub Podlaha} 🎨	_{Miodrag Milić} 📦	_{Derek Mahar} 🤔	_spmundi 🤔	_{Peter Körner} 🛡️	_rubyFeedback 🤔	_rbolsius 📦
_awildturtok 🤔	_agguser 🤔	_jganong 🤔	_{Fulvio Scapin} 🤔	_{Jordan Torbiak} 🤔	_{Andreas Weber} 🤔	_vapniks 📦
_Zombo 📦	_{Brian Fulton-Howard} 📦	_ChCyrill 🤔	_{Jauder Ho} 💻	_{Paweł Sacawa} 🐛	_schragge 📖	_Jordi 📖 🤔

This project follows the all-contributors specification. Contributions of any kind are welcome!

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
experiments/colors This is the entry point for the mlr executable.	This is the entry point for the mlr executable.
experiments/line_parser
mlr This is the entry point for the mlr executable.	This is the entry point for the mlr executable.
scan
sizes
docs
src/miller-as-library This is an example of using Miller as a library.	This is an example of using Miller as a library.
pkg
auxents Package auxents implements little helper-tools embedded within Miller, such as `mlr hex` and `mlr termcvt`.	Package auxents implements little helper-tools embedded within Miller, such as `mlr hex` and `mlr termcvt`.
bifs
cli Package cli contains the flags table and associated datatypes for parsing the Miller command line.	Package cli contains the flags table and associated datatypes for parsing the Miller command line.
climain Package climain contains high-level logic for parsing the main Miller command line.	Package climain contains high-level logic for parsing the main Miller command line.
colorizer Package colorizer implements ANSI terminal colors for key/value color highlighting, test pass/fail, etc.	Package colorizer implements ANSI terminal colors for key/value color highlighting, test pass/fail, etc.
dsl Package dsl contains support routines used between package parsing and package cst.	Package dsl contains support routines used between package parsing and package cst.
dsl/cst Package cst implements the Miller programming language.	Package cst implements the Miller programming language.
entrypoint Package entrypoint has all the usual contents of main(), which are put into this package for ease of unit-testing.	Package entrypoint has all the usual contents of main(), which are put into this package for ease of unit-testing.
go-csv
input Package input contains logic for parsing disk-file/standard-input byte streams into records.	Package input contains logic for parsing disk-file/standard-input byte streams into records.
lib Package lib contains basic library routines for Miller.	Package lib contains basic library routines for Miller.
mlrval
output Package output contains logic for formatting output records as strings to standard output.	Package output contains logic for formatting output records as strings to standard output.
parsing Package parsing contains the single source file mlr.bnf, which is the lexical/semantic grammar file for the Miller put/filter DSL using the GOCC framework.	Package parsing contains the single source file mlr.bnf, which is the lexical/semantic grammar file for the Miller put/filter DSL using the GOCC framework.
parsing/errors Package errors contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.	Package errors contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.
parsing/lexer Package lexer contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.	Package lexer contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.
parsing/parser Package parser contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.	Package parser contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.
parsing/token Package token contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.	Package token contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.
parsing/util Package util contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.	Package util contains source code autogenerated by GOCC from the Miller DSL's grammar file pkg/parsing/mlr.bnf.
pbnjay-strptime
platform Package platform is Miller's platform-dependent code -- as of April 2021, this is divided into Windows and not-Windows.	Package platform is Miller's platform-dependent code -- as of April 2021, this is divided into Windows and not-Windows.
runtime Package runtime contains state for the concrete-syntax-tree executor which runs user-specifed statements in the Miller domain-specific language.	Package runtime contains state for the concrete-syntax-tree executor which runs user-specifed statements in the Miller domain-specific language.
scan Package scan contains low-level logic for efficient type-inference of string to int/float/bool/string.	Package scan contains low-level logic for efficient type-inference of string to int/float/bool/string.
stream Package stream is the high-level view of Miller.	Package stream is the high-level view of Miller.
terminals Package terminals implements `mlr help` (on-line help), `mlr regtest` (for regressio-testing), `mlr repl` (Miller's read-evaluate-print loop), and `mlr version`.	Package terminals implements `mlr help` (on-line help), `mlr regtest` (for regressio-testing), `mlr repl` (Miller's read-evaluate-print loop), and `mlr version`.
terminals/help Package help implements Miller's on-line help.	Package help implements Miller's on-line help.
terminals/regtest Package regtest implements regression-testing for Miller.	Package regtest implements regression-testing for Miller.
terminals/repl Package repl implements Miller's read-evaluate-print loop, which is invoked using mlr repl.	Package repl implements Miller's read-evaluate-print loop, which is invoked using mlr repl.
transformers Package transformer contains logic for transforming input records into output records as requested by the user (sort, filter, etc.).	Package transformer contains logic for transforming input records into output records as requested by the user (sort, filter, etc.).
transformers/utils Package utils holds data structures used by various verbs including join,stats1, and top within package transforming.	Package utils holds data structures used by various verbs including join,stats1, and top within package transforming.
types Package types contains the implementation of the Mlrval datatype which is used for record values, as well as expression/variable values in the Miller put/filter DSL.	Package types contains the implementation of the Mlrval datatype which is used for record values, as well as expression/variable values in the Miller put/filter DSL.
version Package version hold the current Miller version string, such as "v6.0.0".	Package version hold the current Miller version string, such as "v6.0.0".
scripts
early-multi-language-timings
early-multi-language-timings/mand