README
¶
go-reader
There are many interfaces for reading files. This one is ours. It returns io.ReadSeekCloser instances.
Documentation
Example
Readers are instantiated with the reader.NewReader method which takes as its arguments a context.Context instance and a URI string. The URI's scheme represents the type of reader it implements and the remaining (URI) properties are used by that reader type to instantiate itself.
For example to read files from a directory on the local filesystem you would write:
package main
import (
"context"
"github.com/whosonfirst/go-reader"
"io"
"os"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "file:///usr/local/data")
fh, _ := r.Read(ctx, "example.txt")
defer fh.Close()
io.Copy(os.Stdout, fh)
}
There is also a handy "null" reader in case you need a "pretend" reader that doesn't actually do anything:
package main
import (
"context"
"github.com/whosonfirst/go-reader"
"io"
"os"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "null://")
fh, _ := r.Read(ctx, "example.txt")
defer fh.Close()
io.Copy(os.Stdout, fh)
}
Interfaces
reader.Reader
type Reader interface {
Read(context.Context, string) (io.ReadSeekCloser, error)
ReaderURI(context.Context, string) string
}
Custom readers
Custom readers need to:
- Implement the interface above.
- Announce their availability using the
go-reader.RegisterReadermethod on initialization, passing in an initialization function implementing thego-reader.ReaderInitializationFuncinterface.
For example, this is how the go-reader-http reader is implemented:
package reader
import (
"context"
"errors"
wof_reader "github.com/whosonfirst/go-reader"
"github.com/whosonfirst/go-ioutil"
"io"
_ "log"
"net/http"
"net/url"
"path/filepath"
"time"
)
type HTTPReader struct {
wof_reader.Reader
url *url.URL
throttle <-chan time.Time
}
func init() {
ctx := context.Background()
schemes := []string{
"http",
"https",
}
for _, s := range schemes {
err := wof_reader.RegisterReader(ctx, s, NewHTTPReader)
if err != nil {
panic(err)
}
}
}
func NewHTTPReader(ctx context.Context, uri string) (wof_reader.Reader, error) {
u, err := url.Parse(uri)
if err != nil {
return nil, err
}
rate := time.Second / 3
throttle := time.Tick(rate)
r := HTTPReader{
throttle: throttle,
url: u,
}
return &r, nil
}
func (r *HTTPReader) Read(ctx context.Context, uri string) (io.ReadSeekCloser, error) {
<-r.throttle
u, _ := url.Parse(r.url.String())
u.Path = filepath.Join(u.Path, uri)
url := u.String()
rsp, err := http.Get(url)
if err != nil {
return nil, err
}
if rsp.StatusCode != 200 {
return nil, errors.New(rsp.Status)
}
fh, err := ioutil.NewReadSeekCloser(rsp.Body)
if err != nil {
return nil, err
}
return fh, nil
}
func (r *HTTPReader) ReaderURI(ctx context.Context, uri string) string {
return uri
}
And then to use it you would do this:
package main
import (
"context"
"github.com/whosonfirst/go-reader"
_ "github.com/whosonfirst/go-reader-http"
"io"
"os"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "https://data.whosonfirst.org")
fh, _ := r.Read(ctx, "101/736/545/101736545.geojson")
defer fh.Close()
io.Copy(os.Stdout, fh)
}
Available readers
"blob"
Read files from any registered Go Cloud Blob source. For example:
import (
"context"
"github.com/whosonfirst/go-reader"
_ "github.com/whosonfirst/go-reader-blob"
_ "gocloud.dev/blob/s3blob"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "s3://whosonfirst-data?region=us-west-1")
}
github://
Read files from a GitHub repository.
import (
"context"
"github.com/whosonfirst/go-reader"
_ "github.com/whosonfirst/go-reader-github"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "github://{GITHUB_OWNER}/{GITHUB_REPO}")
// to specify a specific branch you would do this:
// r, _ := reader.NewReader(ctx, "github://{GITHUB_OWNER}/{GITHUB_REPO}?branch={GITHUB_BRANCH}")
}
githubapi://
Read files from a GitHub repository using the GitHub API.
import (
"context"
"github.com/whosonfirst/go-reader"
_ "github.com/whosonfirst/go-reader-github"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "githubapi://{GITHUB_OWNER}/{GITHUB_REPO}?access_token={GITHUBAPI_ACCESS_TOKEN}")
// to specify a specific branch you would do this:
// r, _ := reader.NewReader(ctx, "githubapi://{GITHUB_OWNER}/{GITHUB_REPO}/{GITHUB_BRANCH}?access_token={GITHUBAPI_ACCESS_TOKEN}")
}
http:// and https://
Read files from an HTTP(S) endpoint.
import (
"context"
"github.com/whosonfirst/go-reader"
_ "github.com/whosonfirst/go-reader-http"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "https://{HTTP_HOST_AND_PATH}")
}
file://
Read files from a local filesystem.
import (
"context"
"github.com/whosonfirst/go-reader"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "file://{PATH_TO_DIRECTORY}")
}
If you are importing the go-reader-blob package and using the GoCloud's fileblob driver then instantiating the file:// scheme will fail since it will have already been registered. You can work around this by using the fs:// scheme. For example:
r, _ := reader.NewReader(ctx, "fs://{PATH_TO_DIRECTORY}")
null://
Pretend to read files.
import (
"context"
"github.com/whosonfirst/go-reader"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "null://")
}
repo://
This is a convenience scheme for working with Who's On First data repositories.
It will update a URI by appending a data directory to its path and changing its scheme to fs:// before invoking reader.NewReader with the updated URI.
import (
"context"
"github.com/whosonfirst/go-reader"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "repo:///usr/local/data/whosonfirst-data-admin-ca")
}
stdin://
Read "files" from STDIN
import (
"context"
"github.com/whosonfirst/go-reader"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "stdin://")
}
And then to use, something like:
> cat README.md | ./bin/read -reader-uri stdin:// - | wc -l
339
Note the use of - for a URI. This is the convention (when reading from STDIN) but it can be whatever you want it to be.
See also
Documentation
¶
Overview ¶
Example:
package main
import (
"context"
"github.com/whosonfirst/go-reader"
"io"
"os"
)
func main() {
ctx := context.Background()
r, _ := reader.NewReader(ctx, "fs:///usr/local/data")
fh, _ := r.Read(ctx, "example.txt")
defer fh.Close()
io.Copy(os.Stdout, fh)
}
Package reader provides a common interface for reading from a variety of sources. It has the following interface:
type Reader interface {
Read(context.Context, string) (io.ReadSeekCloser, error)
ReaderURI(string) string
}
Reader intstances are created either by calling a package-specific New{SOME_READER}Reader method or by invoking the reader.NewReader method passing in a context.Context instance and a URI specific to the reader class. For example:
r, _ := reader.NewReader(ctx, "fs:///usr/local/data")
Custom reader packages implement the reader.Reader interface and register their availability by calling the reader.RegisterRegister method on initialization. For example:
func init() {
ctx := context.Background()
err = RegisterReader(ctx, "file", NewFileReader)
if err != nil {
panic(err)
}
}
Index ¶
- Constants
- func RegisterReader(ctx context.Context, scheme string, init_func ReaderInitializationFunc) error
- func Schemes() []string
- type FileReader
- type MultiReader
- type NullReader
- type Reader
- func NewFileReader(ctx context.Context, uri string) (Reader, error)
- func NewMultiReader(ctx context.Context, readers ...Reader) (Reader, error)
- func NewMultiReaderFromURIs(ctx context.Context, uris ...string) (Reader, error)
- func NewNullReader(ctx context.Context, uri string) (Reader, error)
- func NewReader(ctx context.Context, uri string) (Reader, error)
- func NewRepoReader(ctx context.Context, uri string) (Reader, error)
- func NewStdinReader(ctx context.Context, uri string) (Reader, error)
- type ReaderInitializationFunc
- type StdinReader
Constants ¶
const STDIN string = "-"
Constant string value representing STDIN.
Variables ¶
This section is empty.
Functions ¶
func RegisterReader ¶ added in v0.1.1
func RegisterReader(ctx context.Context, scheme string, init_func ReaderInitializationFunc) error
RegisterReader registers 'scheme' as a key pointing to 'init_func' in an internal lookup table used to create new `Reader` instances by the `NewReader` method.
Types ¶
type FileReader ¶
type FileReader struct {
Reader
// contains filtered or unexported fields
}
FileReader is a struct that implements the `Reader` interface for reading documents from files on a local disk.
func (*FileReader) Read ¶
func (r *FileReader) Read(ctx context.Context, path string) (io.ReadSeekCloser, error)
Read will open an `io.ReadSeekCloser` for a file matching 'path'.
type MultiReader ¶ added in v0.2.1
type MultiReader struct {
Reader
// contains filtered or unexported fields
}
MultiReader is a struct that implements the `Reader` interface for reading documents from one or more `Reader` instances.
func (*MultiReader) Read ¶ added in v0.2.1
func (mr *MultiReader) Read(ctx context.Context, path string) (io.ReadSeekCloser, error)
Read will open an `io.ReadSeekCloser` for a file matching 'path'. In the case of multiple underlying `Reader` instances the first instance to successfully load 'path' will be returned.
type NullReader ¶
type NullReader struct {
Reader
}
NullReader is a struct that implements the `Reader` interface for reading documents from nowhere.
func (*NullReader) Read ¶
func (r *NullReader) Read(ctx context.Context, path string) (io.ReadSeekCloser, error)
Read will open and return an empty `io.ReadSeekCloser` for any value of 'path'.
type Reader ¶
type Reader interface {
// Reader returns a `io.ReadSeekCloser` instance for a URI resolved by the instance implementing the `Reader` interface.
Read(context.Context, string) (io.ReadSeekCloser, error)
// The absolute path for the file is determined by the instance implementing the `Reader` interface.
ReaderURI(context.Context, string) string
}
Reader is an interface for reading data from multiple sources or targets.
func NewFileReader ¶
NewFileReader returns a new `FileReader` instance for reading documents from local files on disk, configured by 'uri' in the form of:
fs://{PATH}
Where {PATH} is an absolute path to an existing directory where files will be read from.
func NewMultiReader ¶ added in v0.2.1
NewMultiReaderFromURIs returns a new `Reader` instance for reading documents from one or more `Reader` instances.
func NewMultiReaderFromURIs ¶ added in v0.2.1
NewMultiReaderFromURIs returns a new `Reader` instance for reading documents from one or more `Reader` instances. 'uris' is assumed to be a list of URIs each of which will be used to invoke the `NewReader` method.
func NewNullReader ¶
NewNullReader returns a new `FileReader` instance for reading documents from nowhere, configured by 'uri' in the form of:
null://
Technically 'uri' can also be an empty string.
func NewReader ¶
NewReader returns a new `Reader` instance configured by 'uri'. The value of 'uri' is parsed as a `url.URL` and its scheme is used as the key for a corresponding `ReaderInitializationFunc` function used to instantiate the new `Reader`. It is assumed that the scheme (and initialization function) have been registered by the `RegisterReader` method.
func NewRepoReader ¶ added in v0.10.0
NewRepoReader is a convenience method to update 'uri' by appending a `data` directory to its path and changing its scheme to `fs://` before invoking NewReader with the updated URI.
type ReaderInitializationFunc ¶ added in v0.1.1
ReaderInitializationFunc is a function defined by individual reader package and used to create an instance of that reader
type StdinReader ¶ added in v0.8.0
type StdinReader struct {
Reader
}
StdinReader is a struct that implements the `Reader` interface for reading documents from STDIN.
func (*StdinReader) Read ¶ added in v0.8.0
func (r *StdinReader) Read(ctx context.Context, uri string) (io.ReadSeekCloser, error)
Read will open a `io.ReadSeekCloser` instance wrapping `os.Stdin`.
Source Files
Directories