README
¶
HDFS for Go
hdfs is a native go client for hdfs, using the protocol buffers interface the
namenode provides. It implements protocol version 9, which means it supports
Hadoop 2.0.0 and up (including CDH5).
It tries to be idiomatic by aping the stdlib os package where possible. This
includes implementing os.FileInfo for file status, and returning errors of
type os.PathErrors for missing files, for example.
The best place to get started is the Godoc.
client, _ := hdfs.New("namenode:8020")
file, _ := client.Open("/mobydick.txt")
buf := make([]byte, 59)
file.ReadAt(buf, 48847)
fmt.Println(string(buf))
// => Abominable are the tumblers into which he pours his poison.
The hdfs Binary
The library also ships with a command line client for hdfs. Like the library, its primary aim is to be idiomatic, by enabling your favorite unix verbs:
$ hdfs --help
Usage: ./hdfs COMMAND
The flags available are a subset of the POSIX ones, but should behave similarly.
Valid commands:
ls [-la] [FILE]...
rm [-r] FILE...
mv [-fT] SOURCE... DEST
mkdir [-p] FILE...
touch [-amc] FILE...
chmod [-R] OCTAL-MODE FILE...
chown [-R] OWNER[:GROUP] FILE...
cat SOURCE...
head [-n LINES | -c BYTES] SOURCE...
tail [-n LINES | -c BYTES] SOURCE...
get SOURCE [DEST]
getmerge SOURCE DEST
It's also pretty fast compared to hadoop -fs. How much faster, you ask?
$ time hadoop fs -ls / > /dev/null
real 0m2.218s
user 0m2.500s
sys 0m0.376s
$ time hdfs ls / > /dev/null
real 0m0.015s
user 0m0.004s
sys 0m0.004s
Best of all, it comes with bash tab completion!
Installing
To install the library, once you have Go all set up:
$ go get github.com/colinmarc/hdfs
And to install the commandline client:
$ go get github.com/colinmarc/hdfs/cmd/hdfs
Alternatively, to create an hdfs binary that you can install wherever you
want:
$ git clone https://github.com/colinmarc/hdfs
$ cd hdfs
$ make
Finally, you'll want to add two lines to your .bashrc or .profile:
source $GOPATH/src/github.com/colinmarc/hdfs/cmd/hdfs/bash_completion
HADOOP_NAMENODE="namenode:8020"
Or, to install tab completion globally on linux:
ln -sT $GOPATH/src/github.com/colinmarc/hdfs/cmd/hdfs/bash_completion /etc/bash_completion.d/gohdfs
Acknowledgements
This library is heavily indebted to snakebite.
Documentation
¶
Overview ¶
Package hdfs provides a native, idiomatic interface to HDFS. Where possible, it mimics the functionality and signatures of the standard `os` package.
Example:
client, _ := hdfs.New("namenode:8020")
file, _ := client.Open("/mobydick.txt")
buf := make([]byte, 59)
file.ReadAt(buf, 48847)
fmt.Println(string(buf))
// => Abominable are the tumblers into which he pours his poison.
Index ¶
- type Client
- func (c *Client) Chmod(name string, perm os.FileMode) error
- func (c *Client) Chown(name string, user, group string) error
- func (c *Client) Chtimes(name string, atime time.Time, mtime time.Time) error
- func (c *Client) CopyToLocal(src string, dst string) error
- func (c *Client) CreateEmptyFile(filename string) error
- func (c *Client) Mkdir(dirname string, perm os.FileMode) error
- func (c *Client) MkdirAll(dirname string, perm os.FileMode) error
- func (c *Client) Open(name string) (file *FileReader, err error)
- func (c *Client) ReadDir(dirname string) ([]os.FileInfo, error)
- func (c *Client) ReadFile(filename string) ([]byte, error)
- func (c *Client) Remove(name string) error
- func (c *Client) Rename(oldpath, newpath string) error
- func (c *Client) Stat(name string) (os.FileInfo, error)
- type FileInfo
- func (fi *FileInfo) AccessTime() time.Time
- func (fi *FileInfo) IsDir() bool
- func (fi *FileInfo) ModTime() time.Time
- func (fi *FileInfo) Mode() os.FileMode
- func (fi *FileInfo) Name() string
- func (fi *FileInfo) Owner() string
- func (fi *FileInfo) OwnerGroup() string
- func (fi *FileInfo) Size() int64
- func (fi *FileInfo) Sys() interface{}
- type FileReader
- func (f *FileReader) Checksum() ([]byte, error)
- func (f *FileReader) Close() error
- func (f *FileReader) Name() string
- func (f *FileReader) Read(b []byte) (int, error)
- func (f *FileReader) ReadAt(b []byte, off int64) (int, error)
- func (f *FileReader) Readdir(n int) ([]os.FileInfo, error)
- func (f *FileReader) Readdirnames(n int) ([]string, error)
- func (f *FileReader) Seek(offset int64, whence int) (int64, error)
- func (f *FileReader) Stat() os.FileInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
A Client represents a connection to an HDFS cluster
func New ¶
New returns a connected Client, or an error if it can't connect. The user will be the user the code is running under.
func NewForUser ¶
NewForUser returns a connected Client with the user specified, or an error if it can't connect.
func (*Client) Chown ¶
Chown changes the user and group of the file. Unlike os.Chown, this takes a string username and group (since that's what HDFS uses.)
If an empty string is passed for user or group, that field will not be changed remotely.
func (*Client) CopyToLocal ¶
CopyToLocal copies the HDFS file specified by src to the local file at dst. If dst already exists, it will be overwritten.
func (*Client) CreateEmptyFile ¶
CreateEmptyFile creates a empty file named by filename, with the permissions 0644.
func (*Client) MkdirAll ¶
MkdirAll creates a directory for dirname, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm are used for all directories that MkdirAll creates. If dirname is already a directory, MkdirAll does nothing and returns nil.
func (*Client) Open ¶
func (c *Client) Open(name string) (file *FileReader, err error)
Open returns an FileReader which can be used for reading.
func (*Client) ReadDir ¶
ReadDir reads the directory named by dirname and returns a list of sorted directory entries.
type FileInfo ¶
type FileInfo struct {
// contains filtered or unexported fields
}
FileInfo implements os.FileInfo, and provides information about a file or directory in HDFS.
func (*FileInfo) AccessTime ¶
AccessTime returns the last time the file was accessed. It's not part of the os.FileInfo interface.
func (*FileInfo) Owner ¶
Owner returns the name of the user that owns the file or directory. It's not part of the os.FileInfo interface.
func (*FileInfo) OwnerGroup ¶
OwnerGroup returns the name of the group that owns the file or directory. It's not part of the os.FileInfo interface.
type FileReader ¶
type FileReader struct {
// contains filtered or unexported fields
}
A FileReader represents an existing file or directory in HDFS. It implements Reader, ReaderAt, Seeker, and Closer, and can only be used for reads.
func (*FileReader) Checksum ¶
func (f *FileReader) Checksum() ([]byte, error)
Checksum returns HDFS's internal "MD5MD5CRC32C" checksum for a given file.
Internally to HDFS, it works by calculating the MD5 of all the CRCs (which are stored alongside the data) for each block, and then calculating the MD5 of all of those.
func (*FileReader) Read ¶
func (f *FileReader) Read(b []byte) (int, error)
Read implements io.Reader.
func (*FileReader) ReadAt ¶
func (f *FileReader) ReadAt(b []byte, off int64) (int, error)
ReadAt implements io.ReaderAt.
func (*FileReader) Readdir ¶
func (f *FileReader) Readdir(n int) ([]os.FileInfo, error)
Readdir reads the contents of the directory associated with file and returns a slice of up to n os.FileInfo values, as would be returned by Stat, in directory order. Subsequent calls on the same file will yield further os.FileInfos.
If n > 0, Readdir returns at most n os.FileInfo values. In this case, if Readdir returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.
If n <= 0, Readdir returns all the os.FileInfo from the directory in a single slice. In this case, if Readdir succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdir returns the os.FileInfo read until that point and a non-nil error.
func (*FileReader) Readdirnames ¶
func (f *FileReader) Readdirnames(n int) ([]string, error)
Readdirnames reads and returns a slice of names from the directory f.
If n > 0, Readdirnames returns at most n names. In this case, if Readdirnames returns an empty slice, it will return a non-nil error explaining why. At the end of a directory, the error is io.EOF.
If n <= 0, Readdirnames returns all the names from the directory in a single slice. In this case, if Readdirnames succeeds (reads all the way to the end of the directory), it returns the slice and a nil error. If it encounters an error before the end of the directory, Readdirnames returns the names read until that point and a non-nil error.
func (*FileReader) Seek ¶
func (f *FileReader) Seek(offset int64, whence int) (int64, error)
Seek implements io.Seeker.
The seek is virtual - it starts a new block read at the new position.
func (*FileReader) Stat ¶
func (f *FileReader) Stat() os.FileInfo
Stat returns the FileInfo structure describing file.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
protocol
|
|
|
hadoop_common
Package hadoop_common is a generated protocol buffer package.
|
Package hadoop_common is a generated protocol buffer package. |
|
hadoop_hdfs
Package hadoop_hdfs is a generated protocol buffer package.
|
Package hadoop_hdfs is a generated protocol buffer package. |
|
Package rpc implements some of the lower-level functionality required to communicate with the namenode and datanodes.
|
Package rpc implements some of the lower-level functionality required to communicate with the namenode and datanodes. |