script

package module

v0.9.0 Latest Latest Go to latest Published: Feb 28, 2024 License: MIT Imports: 23 Imported by: 0

README ¶

import "github.com/bitfield/script"

What is `script`?

script is a Go library for doing the kind of tasks that shell scripts are good at: reading files, executing subprocesses, counting lines, matching strings, and so on.

Why shouldn't it be as easy to write system administration programs in Go as it is in a typical shell? script aims to make it just that easy.

Shell scripts often compose a sequence of operations on a stream of data (a pipeline). This is how script works, too.

This is one absolutely superb API design. Taking inspiration from shell pipes and turning it into a Go library with syntax this clean is really impressive.
—Simon Willison

Quick start: Unix equivalents

If you're already familiar with shell scripting and the Unix toolset, here is a rough guide to the equivalent script operation for each listed Unix command.

Unix / shell	`script` equivalent
(any program name)	`Exec`
`[ -f FILE ]`	`IfExists`
`>`	`WriteFile`
`>>`	`AppendFile`
`$*`	`Args`
`basename`	`Basename`
`cat`	`File` / `Concat`
`curl`	`Do` / `Get` / `Post`
`cut`	`Column`
`dirname`	`Dirname`
`echo`	`Echo`
`find`	`FindFiles`
`grep`	`Match` / `MatchRegexp`
`grep -v`	`Reject` / `RejectRegexp`
`head`	`First`
`jq`	`JQ`
`ls`	`ListFiles`
`sed`	`Replace` / `ReplaceRegexp`
`sha256sum`	`SHA256Sum` / `SHA256Sums`
`tail`	`Last`
`tee`	`Tee`
`uniq -c`	`Freq`
`wc -l`	`CountLines`
`xargs`	`ExecForEach`

Some examples

Let's see some simple examples. Suppose you want to read the contents of a file as a string:

contents, err := script.File("test.txt").String()

That looks straightforward enough, but suppose you now want to count the lines in that file.

numLines, err := script.File("test.txt").CountLines()

For something a bit more challenging, let's try counting the number of lines in the file that match the string Error:

numErrors, err := script.File("test.txt").Match("Error").CountLines()

But what if, instead of reading a specific file, we want to simply pipe input into this program, and have it output only matching lines (like grep)?

script.Stdin().Match("Error").Stdout()

Just for fun, let's filter all the results through some arbitrary Go function:

script.Stdin().Match("Error").FilterLine(strings.ToUpper).Stdout()

That was almost too easy! So let's pass in a list of files on the command line, and have our program read them all in sequence and output the matching lines:

script.Args().Concat().Match("Error").Stdout()

Maybe we're only interested in the first 10 matches. No problem:

script.Args().Concat().Match("Error").First(10).Stdout()

What's that? You want to append that output to a file instead of printing it to the terminal? You've got some attitude, mister. But okay:

script.Args().Concat().Match("Error").First(10).AppendFile("/var/log/errors.txt")

And if we'd like to send the output to the terminal as well as to the file, we can do that:

script.Echo("data").Tee().AppendFile("data.txt")

We're not limited to getting data only from files or standard input. We can get it from HTTP requests too:

script.Get("https://wttr.in/London?format=3").Stdout()
// Output:
// London: 🌦   +13°C

That's great for simple GET requests, but suppose we want to send some data in the body of a POST request, for example. Here's how that works:

script.Echo(data).Post(URL).Stdout()

If we need to customise the HTTP behaviour in some way, such as using our own HTTP client, we can do that:

script.NewPipe().WithHTTPClient(&http.Client{
	Timeout: 10 * time.Second,
}).Get("https://example.com").Stdout()

Or maybe we need to set some custom header on the request. No problem. We can just create the request in the usual way, and set it up however we want. Then we pass it to Do, which will actually perform the request:

req, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
req.Header.Add("Authorization", "Bearer "+token)
script.Do(req).Stdout()

The HTTP server could return some non-okay response, though; for example, “404 Not Found”. So what happens then?

In general, when any pipe stage (such as Do) encounters an error, it produces no output to subsequent stages. And script treats HTTP response status codes outside the range 200-299 as errors. So the answer for the previous example is that we just won't see any output from this program if the server returns an error response.

Instead, the pipe “remembers” any error that occurs, and we can retrieve it later by calling its Error method, or by using a sink method such as String, which returns an error value along with the result.

Stdout also returns an error, plus the number of bytes successfully written (which we don't care about for this particular case). So we can check that error, which is always a good idea in Go:

_, err := script.Do(req).Stdout()
if err != nil {
	log.Fatal(err)
}

If, as is common, the data we get from an HTTP request is in JSON format, we can use JQ queries to interrogate it:

data, err := script.Do(req).JQ(".[0] | {message: .commit.message, name: .commit.committer.name}").String()

We can also run external programs and get their output:

script.Exec("ping 127.0.0.1").Stdout()

Note that Exec runs the command concurrently: it doesn't wait for the command to complete before returning any output. That's good, because this ping command will run forever (or until we get bored).

Instead, when we read from the pipe using Stdout, we see each line of output as it's produced:

PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.056 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.054 ms
...

In the ping example, we knew the exact arguments we wanted to send the command, and we just needed to run it once. But what if we don't know the arguments yet? We might get them from the user, for example.

We might like to be able to run the external command repeatedly, each time passing it the next line of data from the pipe as an argument. No worries:

script.Args().ExecForEach("ping -c 1 {{.}}").Stdout()

That {{.}} is standard Go template syntax; it'll substitute each line of data from the pipe into the command line before it's executed. You can write as fancy a Go template expression as you want here (but this simple example probably covers most use cases).

If there isn't a built-in operation that does what we want, we can just write our own, using Filter:

script.Echo("hello world").Filter(func (r io.Reader, w io.Writer) error {
	n, err := io.Copy(w, r)
	fmt.Fprintf(w, "\nfiltered %d bytes\n", n)
	return err
}).Stdout()
// Output:
// hello world
// filtered 11 bytes

The func we supply to Filter takes just two parameters: a reader to read from, and a writer to write to. The reader reads the previous stages of the pipe, as you might expect, and anything written to the writer goes to the next stage of the pipe.

If our func returns some error, then, just as with the Do example, the pipe's error status is set, and subsequent stages become a no-op.

Filters run concurrently, so the pipeline can start producing output before the input has been fully read, as it did in the ping example. In fact, most built-in pipe methods, including Exec, are implemented using Filter.

If we want to scan input line by line, we could do that with a Filter function that creates a bufio.Scanner on its input, but we don't need to:

script.Echo("a\nb\nc").FilterScan(func(line string, w io.Writer) {
	fmt.Fprintf(w, "scanned line: %q\n", line)
}).Stdout()
// Output:
// scanned line: "a"
// scanned line: "b"
// scanned line: "c"

And there's more. Much more. Read the docs for full details, and more examples.

A realistic use case

Let's use script to write a program that system administrators might actually need. One thing I often find myself doing is counting the most frequent visitors to a website over a given period of time. Given an Apache log in the Common Log Format like this:

212.205.21.11 - - [30/Jun/2019:17:06:15 +0000] "GET / HTTP/1.1" 200 2028 "https://example.com/ "Mozilla/5.0 (Linux; Android 8.0.0; FIG-LX1 Build/HUAWEIFIG-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.156 Mobile Safari/537.36"

we would like to extract the visitor's IP address (the first column in the logfile), and count the number of times this IP address occurs in the file. Finally, we might like to list the top 10 visitors by frequency. In a shell script we might do something like:

cut -d' ' -f 1 access.log |sort |uniq -c |sort -rn |head

There's a lot going on there, and it's pleasing to find that the equivalent script program is quite brief:

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Stdin().Column(1).Freq().First(10).Stdout()
}

Let's try it out with some sample data:

16 176.182.2.191
 7 212.205.21.11
 1 190.253.121.1
 1 90.53.111.17

Documentation

See pkg.go.dev for the full documentation, or read on for a summary.

Sources

These are functions that create a pipe with a given contents:

Source	Contents
`Args`	command-line arguments
`Do`	HTTP response
`Echo`	a string
`Exec`	command output
`File`	file contents
`FindFiles`	recursive file listing
`Get`	HTTP response
`IfExists`	do something only if some file exists
`ListFiles`	file listing (including wildcards)
`Post`	HTTP response
`Slice`	slice elements, one per line
`Stdin`	standard input

Filters

Filters are methods on an existing pipe that also return a pipe, allowing you to chain filters indefinitely. The filters modify each line of their input according to the following rules:

Filter	Results
`Basename`	removes leading path components from each line, leaving only the filename
`Column`	Nth column of input
`Concat`	contents of multiple files
`Dirname`	removes filename from each line, leaving only leading path components
`Do`	response to supplied HTTP request
`Echo`	all input replaced by given string
`Exec`	filtered through external command
`ExecForEach`	execute given command template for each line of input
`Filter`	user-supplied function filtering a reader to a writer
`FilterLine`	user-supplied function filtering each line to a string
`FilterScan`	user-supplied function filtering each line to a writer
`First`	first N lines of input
`Freq`	frequency count of unique input lines, most frequent first
`Get`	response to HTTP GET on supplied URL
`Join`	replace all newlines with spaces
`JQ`	result of `jq` query
`Last`	last N lines of input
`Match`	lines matching given string
`MatchRegexp`	lines matching given regexp
`Post`	response to HTTP POST on supplied URL
`Reject`	lines not matching given string
`RejectRegexp`	lines not matching given regexp
`Replace`	matching text replaced with given string
`ReplaceRegexp`	matching text replaced with given string
`SHA256Sums`	SHA-256 hashes of each listed file
`Tee`	input copied to supplied writers

Note that filters run concurrently, rather than producing nothing until each stage has fully read its input. This is convenient for executing long-running commands, for example. If you do need to wait for the pipeline to complete, call Wait.

Sinks

Sinks are methods that return some data from a pipe, ending the pipeline and extracting its full contents in a specified way:

Sink	Destination	Results
`AppendFile`	appended to file, creating if it doesn't exist	bytes written, error
`Bytes`		data as `[]byte`, error
`CountLines`		number of lines, error
`Read`	given `[]byte`	bytes read, error
`SHA256Sum`		SHA-256 hash, error
`Slice`		data as `[]string`, error
`Stdout`	standard output	bytes written, error
`String`		data as `string`, error
`Wait`		none
`WriteFile`	specified file, truncating if it exists	bytes written, error

What's new

Version	New
v0.22.0	`Tee`, `WithStderr`
v0.21.0	HTTP support: `Do`, `Get`, `Post`
v0.20.0	`JQ`

Contributing

See the contributor's guide for some helpful tips if you'd like to contribute to the script project.

Links

Gopher image by MariaLetta

Documentation ¶

Rendered for

Overview ¶

Package script aims to make it easy to write shell-type scripts in Go, for general system administration purposes: reading files, counting lines, matching strings, and so on.

Index ¶

type Pipe
type ReadAutoCloser
- func NewReadAutoCloser(r io.Reader) ReadAutoCloser
- func (ra ReadAutoCloser) Close() error
- func (ra ReadAutoCloser) Read(b []byte) (n int, err error)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Pipe ¶

type Pipe struct {
	// Reader is the underlying reader.
	Reader ReadAutoCloser
	// contains filtered or unexported fields
}

Pipe represents a pipe object with an associated ReadAutoCloser.

func Args ¶

func Args() *Pipe

Args creates a pipe containing the program's command-line arguments from os.Args, excluding the program name, one per line.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Args().Stdout()
	// prints command-line arguments
}

Output:

func Cat ¶ added in v0.9.0

func Cat(path string) *Pipe

func Do ¶ added in v0.4.0

func Do(req *http.Request) *Pipe

Do creates a pipe that makes the HTTP request req and produces the response. See Pipe.Do for how the HTTP response status is interpreted.

Example ¶

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"

	"github.com/bitfield/script"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "some data")
	}))
	defer ts.Close()
	req, err := http.NewRequest(http.MethodGet, ts.URL, http.NoBody)
	if err != nil {
		log.Println(err)
		return
	}
	script.Do(req).Stdout()
}

Output:

some data

func Echo ¶

func Echo(s string) *Pipe

Echo creates a pipe containing the string s.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("Hello, world!").Stdout()
}

Output:

Hello, world!

func Exec ¶

func Exec(cmdLine string) *Pipe

Exec creates a pipe that runs cmdLine as an external command and produces its combined output (interleaving standard output and standard error). See Pipe.Exec for error handling details.

Use Pipe.Exec to send the contents of an existing pipe to the command's standard input.

Example (Exit_status_not_zero) ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	p := script.Exec("false")
	p.Wait()
	fmt.Println(p.ExitStatus())
}

Output:

1

Example (Exit_status_zero) ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	p := script.Exec("echo")
	p.Wait()
	fmt.Println(p.ExitStatus())
}

Output:

0

Example (Ok) ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Exec("echo Hello, world!").Stdout()
}

Output:

Hello, world!

func File ¶

func File(path string) *Pipe

File creates a pipe that reads from the file path.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.File("testdata/hello.txt").Stdout()
}

Output:

hello world

func FindFiles ¶

func FindFiles(dir string) *Pipe

FindFiles creates a pipe listing all the files in the directory dir and its subdirectories recursively, one per line, like Unix find(1). If dir doesn't exist or can't be read, the pipe's error status will be set.

Each line of the output consists of a slash-separated path, starting with the initial directory. For example, if the directory looks like this:

test/
        1.txt
        2.txt

the pipe's output will be:

test/1.txt
test/2.txt

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.FindFiles("testdata/multiple_files_with_subdirectory").Stdout()
}

Output:

testdata/multiple_files_with_subdirectory/1.txt
testdata/multiple_files_with_subdirectory/2.txt
testdata/multiple_files_with_subdirectory/3.tar.zip
testdata/multiple_files_with_subdirectory/dir/.hidden
testdata/multiple_files_with_subdirectory/dir/1.txt
testdata/multiple_files_with_subdirectory/dir/2.txt

func Get ¶ added in v0.4.0

func Get(url string) *Pipe

Get creates a pipe that makes an HTTP GET request to url, and produces the response. See Pipe.Do for how the HTTP response status is interpreted.

Example ¶

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/bitfield/script"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "some data")
	}))
	defer ts.Close()
	script.Get(ts.URL).Stdout()
}

Output:

some data

func IfExists ¶

func IfExists(path string) *Pipe

IfExists tests whether path exists, and creates a pipe whose error status reflects the result. If the file doesn't exist, the pipe's error status will be set, and if the file does exist, the pipe will have no error status. This can be used to do some operation only if a given file exists:

IfExists("/foo/bar").Exec("/usr/bin/something")

Example (Exec) ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.IfExists("./testdata/hello.txt").Exec("echo hello").Stdout()
}

Output:

hello

Example (False) ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.IfExists("doesntexist").Echo("found it").Stdout()
}

Output:

Example (NoExec) ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.IfExists("doesntexist").Exec("echo hello").Stdout()
}

Output:

Example (True) ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.IfExists("./testdata/hello.txt").Echo("found it").Stdout()
}

Output:

found it

func ListFiles ¶

func ListFiles(path string) *Pipe

ListFiles creates a pipe containing the files or directories specified by path, one per line. path can be a glob expression, as for filepath.Match. For example:

ListFiles("/data/*").Stdout()

ListFiles does not recurse into subdirectories; use FindFiles instead.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.ListFiles("testdata/multiple_files_with_subdirectory").Stdout()
}

Output:

testdata/multiple_files_with_subdirectory/1.txt
testdata/multiple_files_with_subdirectory/2.txt
testdata/multiple_files_with_subdirectory/3.tar.zip
testdata/multiple_files_with_subdirectory/dir

func NewPipe ¶

func NewPipe() *Pipe

NewPipe creates a new pipe with an empty reader (use Pipe.WithReader to attach another reader to it).

func Post ¶ added in v0.4.0

func Post(url string) *Pipe

Post creates a pipe that makes an HTTP POST request to url, with an empty body, and produces the response. See Pipe.Do for how the HTTP response status is interpreted.

func Slice ¶

func Slice(s []string) *Pipe

Slice creates a pipe containing each element of s, one per line.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	input := []string{"1", "2", "3"}
	script.Slice(input).Stdout()
}

Output:

1
2
3

func Stdin ¶

func Stdin() *Pipe

Stdin creates a pipe that reads from os.Stdin.

func ZFile ¶ added in v0.8.0

func ZFile(path string) *Pipe

func (*Pipe) AppendFile ¶

func (p *Pipe) AppendFile(path string) (int64, error)

AppendFile appends the contents of the pipe to the file path, creating it if necessary, and returns the number of bytes successfully written, or an error.

func (*Pipe) Basename ¶

func (p *Pipe) Basename() *Pipe

Basename reads paths from the pipe, one per line, and removes any leading directory components from each. So, for example, /usr/local/bin/foo would become just foo. This is the complementary operation to Pipe.Dirname.

If any line is empty, Basename will transform it to a single dot. Trailing slashes are removed. The behaviour of Basename is the same as filepath.Base (not by coincidence).

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	input := []string{
		"",
		"/",
		"/root",
		"/tmp/example.php",
		"/var/tmp/",
		"./src/filters",
		"C:/Program Files",
	}
	script.Slice(input).Basename().Stdout()
}

Output:

.
/
root
example.php
tmp
filters
Program Files

func (*Pipe) Bytes ¶

func (p *Pipe) Bytes() ([]byte, error)

Bytes returns the contents of the pipe as a []byte, or an error.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	data, err := script.Echo("hello").Bytes()
	if err != nil {
		panic(err)
	}
	fmt.Println(data)
}

Output:

[104 101 108 108 111]

func (*Pipe) Close ¶

func (p *Pipe) Close() error

Close closes the pipe's associated reader. This is a no-op if the reader is not an io.Closer.

func (*Pipe) Column ¶

func (p *Pipe) Column(col int) *Pipe

Column produces column col of each line of input, where the first column is column 1, and columns are delimited by Unicode whitespace. Lines containing fewer than col columns will be skipped.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	input := []string{
		"PID   TT  STAT      TIME COMMAND",
		"  1   ??  Ss   873:17.62 /sbin/launchd",
		" 50   ??  Ss    13:18.13 /usr/libexec/UserEventAgent (System)",
		" 51   ??  Ss    22:56.75 /usr/sbin/syslogd",
	}
	script.Slice(input).Column(1).Stdout()
}

Output:

PID
1
50
51

func (*Pipe) Concat ¶

func (p *Pipe) Concat() *Pipe

Concat reads paths from the pipe, one per line, and produces the contents of all the corresponding files in sequence. If there are any errors (for example, non-existent files), these will be ignored, execution will continue, and the pipe's error status will not be set.

This makes it convenient to write programs that take a list of paths on the command line. For example:

script.Args().Concat().Stdout()

The list of paths could also come from a file:

script.File("filelist.txt").Concat()

Or from the output of a command:

script.Exec("ls /var/app/config/").Concat().Stdout()

Each input file will be closed once it has been fully read. If any of the files can't be opened or read, Concat will simply skip these and carry on, without setting the pipe's error status. This mimics the behaviour of Unix cat(1).

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	input := []string{
		"testdata/test.txt",
		"testdata/doesntexist.txt",
		"testdata/hello.txt",
	}
	script.Slice(input).Concat().Stdout()
}

Output:

This is the first line in the file.
Hello, world.
This is another line in the file.
hello world

func (*Pipe) CountLines ¶

func (p *Pipe) CountLines() (lines int, err error)

CountLines returns the number of lines of input, or an error.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	n, err := script.Echo("a\nb\nc\n").CountLines()
	if err != nil {
		panic(err)
	}
	fmt.Println(n)
}

Output:

3

func (*Pipe) Dirname ¶

func (p *Pipe) Dirname() *Pipe

Dirname reads paths from the pipe, one per line, and produces only the parent directories of each path. For example, /usr/local/bin/foo would become just /usr/local/bin. This is the complementary operation to Pipe.Basename.

If a line is empty, Dirname will transform it to a single dot. Trailing slashes are removed, unless Dirname returns the root folder. Otherwise, the behaviour of Dirname is the same as filepath.Dir (not by coincidence).

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	input := []string{
		"",
		"/",
		"/root",
		"/tmp/example.php",
		"/var/tmp/",
		"./src/filters",
		"C:/Program Files",
	}
	script.Slice(input).Dirname().Stdout()
}

Output:

.
/
/
/tmp
/var
./src
C:

func (*Pipe) Do ¶ added in v0.4.0

func (p *Pipe) Do(req *http.Request) *Pipe

Do performs the HTTP request req using the pipe's configured HTTP client, as set by Pipe.WithHTTPClient, or http.DefaultClient otherwise. The response body is streamed concurrently to the pipe's output. If the response status is anything other than HTTP 200-299, the pipe's error status is set.

Example ¶

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
	"strings"

	"github.com/bitfield/script"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		data, err := io.ReadAll(r.Body)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Fprintf(w, "You said: %s", data)
	}))
	defer ts.Close()
	req, err := http.NewRequest(http.MethodGet, ts.URL, strings.NewReader("hello"))
	if err != nil {
		log.Println(err)
		return
	}
	script.NewPipe().Do(req).Stdout()
}

Output:

You said: hello

func (*Pipe) EachLine deprecated

func (p *Pipe) EachLine(process func(string, *strings.Builder)) *Pipe

EachLine calls the function process on each line of input, passing it the line as a string, and a *strings.Builder to write its output to.

Deprecated: use Pipe.FilterLine or Pipe.FilterScan instead, which run concurrently and don't do unnecessary reads on the input.

Example ¶

package main

import (
	"strings"

	"github.com/bitfield/script"
)

func main() {
	script.File("testdata/test.txt").EachLine(func(line string, out *strings.Builder) {
		out.WriteString("> " + line + "\n")
	}).Stdout()
}

Output:

> This is the first line in the file.
> Hello, world.
> This is another line in the file.

func (*Pipe) Echo ¶

func (p *Pipe) Echo(s string) *Pipe

Echo sets the pipe's reader to one that produces the string s, detaching any existing reader without draining or closing it.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.NewPipe().Echo("Hello, world!").Stdout()
}

Output:

Hello, world!

func (*Pipe) Error ¶

func (p *Pipe) Error() error

Error returns any error present on the pipe, or nil otherwise.

func (*Pipe) Exec ¶

func (p *Pipe) Exec(cmdLine string) *Pipe

Exec runs cmdLine as an external command, sending it the contents of the pipe as input, and produces the command's standard output (see below for error output). The effect of this is to filter the contents of the pipe through the external command.

Error handling ¶

If the command had a non-zero exit status, the pipe's error status will also be set to the string “exit status X”, where X is the integer exit status. Even in the event of a non-zero exit status, the command's output will still be available in the pipe. This is often helpful for debugging. However, because Pipe.String is a no-op if the pipe's error status is set, if you want output you will need to reset the error status before calling Pipe.String.

If the command writes to its standard error stream, this will also go to the pipe, along with its standard output. However, the standard error text can instead be redirected to a supplied writer, using Pipe.WithStderr.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("Hello, world!").Exec("tr a-z A-Z").Stdout()
}

Output:

HELLO, WORLD!

func (*Pipe) ExecForEach ¶

func (p *Pipe) ExecForEach(cmdLine string) *Pipe

ExecForEach renders cmdLine as a Go template for each line of input, running the resulting command, and produces the combined output of all these commands in sequence. See Pipe.Exec for error handling details.

This is mostly useful for substituting data into commands using Go template syntax. For example:

ListFiles("*").ExecForEach("touch {{.}}").Wait()

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc\n").ExecForEach("echo {{.}}").Stdout()
}

Output:

a
b
c

func (*Pipe) ExitStatus ¶

func (p *Pipe) ExitStatus() int

ExitStatus returns the integer exit status of a previous command (for example run by Pipe.Exec). This will be zero unless the pipe's error status is set and the error matches the pattern “exit status %d”.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	p := script.Exec("echo")
	fmt.Println(p.ExitStatus())
}

Output:

0

func (*Pipe) Fields ¶ added in v0.7.0

func (p *Pipe) Fields(inDelim, outDelim string, a ...int) *Pipe

Fields splits the input string into fields based on the specified input delimiter, selects the fields at the specified indices, and then joins these fields with the specified output delimiter. The indices are 1-based and negative indices count from the end of the fields. If an index is out of range, it is ignored.

For example, Fields("|", ",", 1, 3) applied to the string "field1|field2|field3|field4" would return the string "field1,field3".

Fields returns a new Pipe that will write the transformed string when its output is read.

func (*Pipe) Filter ¶

func (p *Pipe) Filter(filter func(io.Reader, io.Writer) error) *Pipe

Filter sends the contents of the pipe to the function filter and produces the result. filter takes an io.Reader to read its input from and an io.Writer to write its output to, and returns an error, which will be set on the pipe.

filter runs concurrently, so its goroutine will not exit until the pipe has been fully read. Use Pipe.Wait to wait for all concurrent filters to complete.

Example ¶

package main

import (
	"fmt"
	"io"

	"github.com/bitfield/script"
)

func main() {
	script.Echo("hello world").Filter(func(r io.Reader, w io.Writer) error {
		n, err := io.Copy(w, r)
		fmt.Fprintf(w, "\nfiltered %d bytes\n", n)
		return err
	}).Stdout()
}

Output:

hello world
filtered 11 bytes

func (*Pipe) FilterLine ¶

func (p *Pipe) FilterLine(filter func(string) string) *Pipe

FilterLine sends the contents of the pipe to the function filter, a line at a time, and produces the result. filter takes each line as a string and returns a string as its output. See Pipe.Filter for concurrency handling.

Example (Stdlib) ¶

package main

import (
	"strings"

	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc").FilterLine(strings.ToUpper).Stdout()
}

Output:

A
B
C

Example (User) ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc").FilterLine(func(line string) string {
		return "> " + line
	}).Stdout()
}

Output:

> a
> b
> c

func (*Pipe) FilterScan ¶

func (p *Pipe) FilterScan(filter func(string, io.Writer)) *Pipe

FilterScan sends the contents of the pipe to the function filter, a line at a time, and produces the result. filter takes each line as a string and an io.Writer to write its output to. See Pipe.Filter for concurrency handling.

Example ¶

package main

import (
	"fmt"
	"io"

	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc").FilterScan(func(line string, w io.Writer) {
		fmt.Fprintf(w, "scanned line: %q\n", line)
	}).Stdout()
}

Output:

scanned line: "a"
scanned line: "b"
scanned line: "c"

func (*Pipe) First ¶

func (p *Pipe) First(n int) *Pipe

First produces only the first n lines of the pipe's contents, or all the lines if there are less than n. If n is zero or negative, there is no output at all.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc\n").First(2).Stdout()
}

Output:

a
b

func (*Pipe) Freq ¶

func (p *Pipe) Freq() *Pipe

Freq produces only the unique lines from the pipe's contents, each prefixed with a frequency count, in descending numerical order (most frequent lines first). Lines with equal frequency will be sorted alphabetically.

For example, we could take a common shell pipeline like this:

sort input.txt |uniq -c |sort -rn

and replace it with:

File("input.txt").Freq().Stdout()

Or to get only the ten most common lines:

File("input.txt").Freq().First(10).Stdout()

Like Unix uniq(1), Freq right-justifies its count values in a column for readability, padding with spaces if necessary.

Example ¶

package main

import (
	"strings"

	"github.com/bitfield/script"
)

func main() {
	input := strings.Join([]string{
		"apple",
		"orange",
		"banana",
		"banana",
		"apple",
		"orange",
		"kumquat",
		"apple",
		"orange",
		"apple",
		"banana",
		"banana",
		"apple",
		"apple",
		"orange",
		"apple",
		"apple",
		"apple",
		"apple",
	}, "\n")
	script.Echo(input).Freq().Stdout()
}

Output:

10 apple
 4 banana
 4 orange
 1 kumquat

func (*Pipe) Get ¶ added in v0.4.0

func (p *Pipe) Get(url string) *Pipe

Get makes an HTTP GET request to url, sending the contents of the pipe as the request body, and produces the server's response. See Pipe.Do for how the HTTP response status is interpreted.

Example ¶

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"

	"github.com/bitfield/script"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		data, err := io.ReadAll(r.Body)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Fprintf(w, "You said: %s", data)
	}))
	defer ts.Close()
	script.Echo("hello").Get(ts.URL).Stdout()
}

Output:

You said: hello

func (*Pipe) JQ ¶ added in v0.2.0

func (p *Pipe) JQ(query string) *Pipe

JQ executes query on the pipe's contents (presumed to be JSON), producing the result. An invalid query will set the appropriate error on the pipe.

The exact dialect of JQ supported is that provided by github.com/itchyny/gojq, whose documentation explains the differences between it and standard JQ.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	kernel := "Darwin"
	arch := "x86_64"
	query := fmt.Sprintf(".assets[] | select(.name | endswith(\"%s_%s.tar.gz\")).browser_download_url", kernel, arch)
	script.File("testdata/releases.json").JQ(query).Stdout()
}

Output:

"https://github.com/mbarley333/blackjack/releases/download/v0.3.3/blackjack_0.3.3_Darwin_x86_64.tar.gz"

func (*Pipe) Join ¶

func (p *Pipe) Join() *Pipe

Join joins all the lines in the pipe's contents into a single space-separated string, which will always end with a newline.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("hello\nworld\n").Join().Stdout()
}

Output:

hello world

func (*Pipe) Last ¶

func (p *Pipe) Last(n int) *Pipe

Last produces only the last n lines of the pipe's contents, or all the lines if there are less than n. If n is zero or negative, there is no output at all.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc\n").Last(2).Stdout()
}

Output:

b
c

func (*Pipe) Match ¶

func (p *Pipe) Match(s string) *Pipe

Match produces only the input lines that contain the string s.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc\n").Match("b").Stdout()
}

Output:

b

func (*Pipe) MatchRegexp ¶

func (p *Pipe) MatchRegexp(re *regexp.Regexp) *Pipe

MatchRegexp produces only the input lines that match the compiled regexp re.

Example ¶

package main

import (
	"regexp"

	"github.com/bitfield/script"
)

func main() {
	re := regexp.MustCompile("w.*d")
	script.Echo("hello\nworld\n").MatchRegexp(re).Stdout()
}

Output:

world

func (*Pipe) Post ¶ added in v0.4.0

func (p *Pipe) Post(url string) *Pipe

Post makes an HTTP POST request to url, using the contents of the pipe as the request body, and produces the server's response. See Pipe.Do for how the HTTP response status is interpreted.

Example ¶

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"

	"github.com/bitfield/script"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		data, err := io.ReadAll(r.Body)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Fprintf(w, "You said: %s", data)
	}))
	defer ts.Close()
	script.Echo("hello").Post(ts.URL).Stdout()
}

Output:

You said: hello

func (*Pipe) Read ¶

func (p *Pipe) Read(b []byte) (int, error)

Read reads up to len(b) bytes from the pipe into b. It returns the number of bytes read and any error encountered. At end of file, or on a nil pipe, Read returns 0, io.EOF.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	buf := make([]byte, 12)
	n, err := script.Echo("hello world\n").Read(buf)
	if err != nil {
		panic(err)
	}
	fmt.Println(n)
}

Output:

12

func (*Pipe) Reject ¶

func (p *Pipe) Reject(s string) *Pipe

Reject produces only lines that do not contain the string s.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc\n").Reject("b").Stdout()
}

Output:

a
c

func (*Pipe) RejectRegexp ¶

func (p *Pipe) RejectRegexp(re *regexp.Regexp) *Pipe

RejectRegexp produces only lines that don't match the compiled regexp re.

Example ¶

package main

import (
	"regexp"

	"github.com/bitfield/script"
)

func main() {
	re := regexp.MustCompile("w.*d")
	script.Echo("hello\nworld\n").RejectRegexp(re).Stdout()
}

Output:

hello

func (*Pipe) Replace ¶

func (p *Pipe) Replace(search, replace string) *Pipe

Replace replaces all occurrences of the string search with the string replace.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("a\nb\nc\n").Replace("b", "replacement").Stdout()
}

Output:

a
replacement
c

func (*Pipe) ReplaceRegexp ¶

func (p *Pipe) ReplaceRegexp(re *regexp.Regexp, replace string) *Pipe

ReplaceRegexp replaces all matches of the compiled regexp re with the string replace. $x variables in the replace string are interpreted as by [regexp#Regexp.Expand]; for example, $1 represents the text of the first submatch.

Example ¶

package main

import (
	"regexp"

	"github.com/bitfield/script"
)

func main() {
	re := regexp.MustCompile("w.*d")
	script.Echo("hello\nworld\n").ReplaceRegexp(re, "replacement").Stdout()
}

Output:

hello
replacement

func (*Pipe) SHA256Sum ¶

func (p *Pipe) SHA256Sum() (string, error)

SHA256Sum returns the hex-encoded SHA-256 hash of the entire contents of the pipe, or an error.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	sum, err := script.Echo("hello world").SHA256Sum()
	if err != nil {
		panic(err)
	}
	fmt.Println(sum)
}

Output:

b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9

func (*Pipe) SHA256Sums ¶

func (p *Pipe) SHA256Sums() *Pipe

SHA256Sums reads paths from the pipe, one per line, and produces the hex-encoded SHA-256 hash of each corresponding file, one per line. Any files that cannot be opened or read will be ignored.

Example ¶

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Echo("testdata/test.txt").SHA256Sums().Stdout()
}

Output:

a562c9c95e2ff3403e7ffcd8508c6b54d47d5f251387758d3e63dbaaa8296341

func (*Pipe) SetError ¶

func (p *Pipe) SetError(err error)

SetError sets the error err on the pipe.

func (*Pipe) Slice ¶

func (p *Pipe) Slice() ([]string, error)

Slice returns the pipe's contents as a slice of strings, one element per line, or an error.

An empty pipe will produce an empty slice. A pipe containing a single empty line (that is, a single \n character) will produce a slice containing the empty string as its single element.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	s, err := script.Echo("a\nb\nc\n").Slice()
	if err != nil {
		panic(err)
	}
	fmt.Println(s)
}

Output:

[a b c]

func (*Pipe) Stdout ¶

func (p *Pipe) Stdout() (int, error)

Stdout copies the pipe's contents to its configured standard output (using Pipe.WithStdout), or to os.Stdout otherwise, and returns the number of bytes successfully written, together with any error.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	n, err := script.Echo("a\nb\nc\n").Stdout()
	if err != nil {
		panic(err)
	}
	fmt.Println(n)
}

Output:

a
b
c
6

func (*Pipe) String ¶

func (p *Pipe) String() (string, error)

String returns the pipe's contents as a string, together with any error.

Example ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	s, err := script.Echo("hello\nworld").String()
	if err != nil {
		panic(err)
	}
	fmt.Println(s)
}

Output:

hello
world

func (*Pipe) Table ¶ added in v0.7.0

func (p *Pipe) Table(options ...string) error

Table takes CSV from the input pipeline, and creates a formatted table from the provided CSV string. The input from the PIPE is expected to be a CSV string, or it may not output at all. You can convert fields to CSV by using the p.Fields function. The function accepts an optional list of flags that can be used to customize the table's formatting. The flags can be used to specify the table's render style, alignment, header presence, wrapping, indentation, line separation between rows, spacing, columns, and delimiter. If the flags are not provided, the table is created with default settings. The function returns the formatted table as a string and an error if any occurred during the table creation. Try these with <render>-nohead when there is no header. Options are comma separated, "Render=grid","Header=one,two,three"

	+-------------+-------------------------------------------+---------------------------+
	| Option      | Description                               | Usage                     |
	+=============+===========================================+===========================+
	| Render      | How you want the table to be formatted    | Render=See Below          |
	| Align       | Left Center Right                         | Align=Left                |
	| NoHeader    | True/False                                | NoHeader=False            |
	| Wrap        | Wrap Text in cells                        | Wrap=true                 |
	| Header      | Specify Header                            | Header=One,Two,Three    |
	+-------------+-------------------------------------------+---------------------------+
 Ex: script.Echo("one,two,three").Table("Render=grid","Header=one,two,three", "Align=right", "NoHeader=false", "Wrap=true")

Render Formats:

+-----------+-------------------------------------+
| Render    | Output Format                       |
+===========+=====================================+
| mysql     | Looks like a MySQL Client Query     |
| grid      | Spreadsheet using Graphical Grid    |
| gridt     | Spreadsheet using text grid         |
| simple    | Simple Table                        |
| html      | Output in HTML Table                |
| tab       | Just text tab separated             |
| csv       | Output in CSV format                |
| plain     | Plain Table output                  |
+-----------+-------------------------------------+

func (*Pipe) Tee ¶ added in v0.3.0

func (p *Pipe) Tee(writers ...io.Writer) *Pipe

Tee copies the pipe's contents to each of the supplied writers, like Unix tee(1). If no writers are supplied, the default is the pipe's standard output.

Example (Stdout) ¶

package main

import (
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	s, err := script.Echo("hello\n").Tee().String()
	if err != nil {
		panic(err)
	}
	fmt.Println(s)
}

Output:

hello
hello

Example (Writers) ¶

package main

import (
	"bytes"
	"fmt"

	"github.com/bitfield/script"
)

func main() {
	buf1, buf2 := new(bytes.Buffer), new(bytes.Buffer)
	s, err := script.Echo("hello\n").Tee(buf1, buf2).String()
	if err != nil {
		panic(err)
	}
	fmt.Print(s)
	fmt.Print(buf1.String())
	fmt.Print(buf2.String())
}

Output:

hello
hello
hello

func (*Pipe) Wait ¶

func (p *Pipe) Wait()

Wait reads the pipe to completion and discards the result. This is mostly useful for waiting until concurrent filters have completed (see Pipe.Filter).

func (*Pipe) WithError ¶

func (p *Pipe) WithError(err error) *Pipe

WithError sets the error err on the pipe.

func (*Pipe) WithHTTPClient ¶ added in v0.4.0

func (p *Pipe) WithHTTPClient(c *http.Client) *Pipe

WithHTTPClient sets the HTTP client c for use with subsequent requests via Pipe.Do, Pipe.Get, or Pipe.Post. For example, to make a request using a client with a timeout:

NewPipe().WithHTTPClient(&http.Client{
        Timeout: 10 * time.Second,
}).Get("https://example.com").Stdout()

func (*Pipe) WithReader ¶

func (p *Pipe) WithReader(r io.Reader) *Pipe

WithReader sets the pipe's input reader to r. Once r has been completely read, it will be closed if necessary.

func (*Pipe) WithStderr ¶ added in v0.8.0

func (p *Pipe) WithStderr(w io.Writer) *Pipe

WithStderr redirects the standard error output for commands run via Pipe.Exec or Pipe.ExecForEach to the writer w, instead of going to the pipe as it normally would.

Example ¶

package main

import (
	"bytes"
	"fmt"
	"strings"

	"github.com/bitfield/script"
)

func main() {
	buf := new(bytes.Buffer)
	script.NewPipe().WithStderr(buf).Exec("go").Wait()
	fmt.Println(strings.Contains(buf.String(), "Usage"))
}

Output:

true

func (*Pipe) WithStdout ¶

func (p *Pipe) WithStdout(w io.Writer) *Pipe

WithStdout sets the pipe's standard output to the writer w, instead of the default os.Stdout.

func (*Pipe) WriteFile ¶

func (p *Pipe) WriteFile(path string) (int64, error)

WriteFile writes the pipe's contents to the file path, truncating it if it exists, and returns the number of bytes successfully written, or an error.

type ReadAutoCloser ¶

type ReadAutoCloser struct {
	// contains filtered or unexported fields
}

ReadAutoCloser wraps an io.ReadCloser so that it will be automatically closed once it has been fully read.

func NewReadAutoCloser ¶

func NewReadAutoCloser(r io.Reader) ReadAutoCloser

NewReadAutoCloser returns a ReadAutoCloser wrapping the reader r.

func (ReadAutoCloser) Close ¶

func (ra ReadAutoCloser) Close() error

Close closes ra's reader, returning any resulting error.

func (ReadAutoCloser) Read ¶

func (ra ReadAutoCloser) Read(b []byte) (n int, err error)

Read reads up to len(b) bytes from ra's reader into b. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF. If end-of-file is reached, the reader will be closed.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
Example

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

README ¶

What is script?

Quick start: Unix equivalents

Some examples

A realistic use case

Documentation

Sources

Filters

Sinks

What's new

Contributing

Links

Documentation ¶

Overview ¶

Index ¶

Examples ¶

Constants ¶

Variables ¶

Functions ¶

Types ¶

type Pipe ¶

func Args ¶

func Cat ¶ added in v0.9.0

func Do ¶ added in v0.4.0

func Echo ¶

func Exec ¶

func File ¶

func FindFiles ¶

func Get ¶ added in v0.4.0

func IfExists ¶

func ListFiles ¶

func NewPipe ¶

func Post ¶ added in v0.4.0

func Slice ¶

func Stdin ¶

func ZFile ¶ added in v0.8.0

func (*Pipe) AppendFile ¶

func (*Pipe) Basename ¶

func (*Pipe) Bytes ¶

func (*Pipe) Close ¶

func (*Pipe) Column ¶

func (*Pipe) Concat ¶

func (*Pipe) CountLines ¶

func (*Pipe) Dirname ¶

func (*Pipe) Do ¶ added in v0.4.0

func (*Pipe) EachLine deprecated

func (*Pipe) Echo ¶

func (*Pipe) Error ¶

func (*Pipe) Exec ¶

Error handling ¶

func (*Pipe) ExecForEach ¶

func (*Pipe) ExitStatus ¶

func (*Pipe) Fields ¶ added in v0.7.0

func (*Pipe) Filter ¶

func (*Pipe) FilterLine ¶

func (*Pipe) FilterScan ¶

func (*Pipe) First ¶

func (*Pipe) Freq ¶

func (*Pipe) Get ¶ added in v0.4.0

func (*Pipe) JQ ¶ added in v0.2.0

func (*Pipe) Join ¶

func (*Pipe) Last ¶

func (*Pipe) Match ¶

func (*Pipe) MatchRegexp ¶

func (*Pipe) Post ¶ added in v0.4.0

func (*Pipe) Read ¶

func (*Pipe) Reject ¶

func (*Pipe) RejectRegexp ¶

func (*Pipe) Replace ¶

func (*Pipe) ReplaceRegexp ¶

func (*Pipe) SHA256Sum ¶

func (*Pipe) SHA256Sums ¶

func (*Pipe) SetError ¶

func (*Pipe) Slice ¶

func (*Pipe) Stdout ¶

func (*Pipe) String ¶

func (*Pipe) Table ¶ added in v0.7.0

func (*Pipe) Tee ¶ added in v0.3.0

func (*Pipe) Wait ¶

func (*Pipe) WithError ¶

What is `script`?