Documentation
¶
Overview ¶
Package rpc implements some of the lower-level functionality required to communicate with the namenode and datanodes.
Index ¶
Constants ¶
const (
ClientName = "go-hdfs"
)
ClientName is passed into the namenode on requests, and identifies this client to the namenode.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BlockReader ¶
type BlockReader struct {
// contains filtered or unexported fields
}
BlockReader implements io.ReadCloser, for reading a block. It abstracts over reading from multiple datanodes, in order to be robust to connection failures, timeouts, and other shenanigans.
func NewBlockReader ¶
func NewBlockReader(block *hdfs.LocatedBlockProto, offset int64) *BlockReader
NewBlockReader returns a new BlockReader, given the block information and security token from the namenode. It will connect (lazily) to one of the provided datanode locations based on which datanodes have seen failures.
func (*BlockReader) Read ¶
func (br *BlockReader) Read(b []byte) (int, error)
Read implements io.Reader.
In the case that a failure (such as a disconnect) occurs while reading, the BlockReader will failover to another datanode and continue reading transparently. In the case that all the datanodes fail, the error from the most recent attempt will be returned.
Any datanode failures are recorded in a global cache, so subsequent reads, even reads for different blocks, will prioritize them lower.
type ChecksumReader ¶
type ChecksumReader struct {
// contains filtered or unexported fields
}
ChecksumReader provides an interface for reading the "MD5CRC32" checksums of individual blocks. It abstracts over reading from multiple datanodes, in order to be robust to failures.
func NewChecksumReader ¶
func NewChecksumReader(block *hdfs.LocatedBlockProto) *ChecksumReader
NewChecksumReader creates a new ChecksumReader for the given block.
func (*ChecksumReader) ReadChecksum ¶
func (cr *ChecksumReader) ReadChecksum() ([]byte, error)
ReadChecksum returns the checksum of the block.
type NamenodeConnection ¶
type NamenodeConnection struct {
// contains filtered or unexported fields
}
NamenodeConnection represents an open connection to a namenode.
func NewNamenodeConnection ¶
func NewNamenodeConnection(address, user string) (*NamenodeConnection, error)
NewNamenodeConnection creates a new connection to a Namenode, and preforms an initial handshake.
You probably want to use hdfs.New instead, which provides a higher-level interface.
func WrapNamenodeConnection ¶
func WrapNamenodeConnection(conn net.Conn, user string) (*NamenodeConnection, error)
WrapNamenodeConnection wraps an existing net.Conn to a Namenode, and preforms an initial handshake.
You probably want to use hdfs.New instead, which provides a higher-level interface.
type NamenodeError ¶
NamenodeError represents an interepreted error from the Namenode, including the error code and the java backtrace.
func (*NamenodeError) Desc ¶
func (err *NamenodeError) Desc() string
Desc returns the long form of the error code, as defined in the RpcErrorCodeProto in RpcHeader.proto
func (*NamenodeError) Error ¶
func (err *NamenodeError) Error() string
Source Files