Documentation
¶
Overview ¶
Package rpc implements some of the lower-level functionality required to communicate with the namenode and datanodes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrEndOfBlock = errors.New("The amount of data to be written is more than is left in the block.")
var ErrInvalidSeqno = errors.New("Invalid ack sequence number")
Functions ¶
This section is empty.
Types ¶
type BlockReader ¶
type BlockReader struct {
// contains filtered or unexported fields
}
BlockReader implements io.ReadCloser, for reading a block. It abstracts over reading from multiple datanodes, in order to be robust to connection failures, timeouts, and other shenanigans.
func NewBlockReader ¶
func NewBlockReader(block *hdfs.LocatedBlockProto, offset int64, clientName string) *BlockReader
NewBlockReader returns a new BlockReader, given the block information and security token from the namenode. It will connect (lazily) to one of the provided datanode locations based on which datanodes have seen failures.
func (*BlockReader) Read ¶
func (br *BlockReader) Read(b []byte) (int, error)
Read implements io.Reader.
In the case that a failure (such as a disconnect) occurs while reading, the BlockReader will failover to another datanode and continue reading transparently. In the case that all the datanodes fail, the error from the most recent attempt will be returned.
Any datanode failures are recorded in a global cache, so subsequent reads, even reads for different blocks, will prioritize them lower.
type BlockWriter ¶ added in v1.0.0
type BlockWriter struct {
// contains filtered or unexported fields
}
BlockWriter implements io.WriteCloser for writing a block to a datanode. Given a block location, it handles pipeline construction and failures, including communicating with the namenode if need be.
func NewBlockWriter ¶ added in v1.0.0
func NewBlockWriter(block *hdfs.LocatedBlockProto, namenode *NamenodeConnection, blockSize int64) *BlockWriter
NewBlockWriter returns a BlockWriter for the given block. It will lazily set up a replication pipeline, and connect to the "best" datanode based on any previously seen failures.
func (*BlockWriter) Close ¶ added in v1.0.0
func (bw *BlockWriter) Close() error
Close implements io.Closer. It flushes any unwritten packets out to the datanode, and sends a final packet indicating the end of the block.
func (*BlockWriter) Write ¶ added in v1.0.0
func (bw *BlockWriter) Write(b []byte) (int, error)
Write implements io.Writer.
Unlike BlockReader, BlockWriter currently has no ability to recover from write failures (timeouts, datanode failure, etc). Once it returns an error from Write or Close, it may be in an invalid state.
This will hopefully be fixed in a future release.
type ChecksumReader ¶
type ChecksumReader struct {
// contains filtered or unexported fields
}
ChecksumReader provides an interface for reading the "MD5CRC32" checksums of individual blocks. It abstracts over reading from multiple datanodes, in order to be robust to failures.
func NewChecksumReader ¶
func NewChecksumReader(block *hdfs.LocatedBlockProto) *ChecksumReader
NewChecksumReader creates a new ChecksumReader for the given block.
func (*ChecksumReader) ReadChecksum ¶
func (cr *ChecksumReader) ReadChecksum() ([]byte, error)
ReadChecksum returns the checksum of the block.
type NamenodeConnection ¶
type NamenodeConnection struct {
// contains filtered or unexported fields
}
NamenodeConnection represents an open connection to a namenode.
func NewNamenodeConnection ¶
func NewNamenodeConnection(address, user string) (*NamenodeConnection, error)
NewNamenodeConnection creates a new connection to a Namenode, and preforms an initial handshake.
You probably want to use hdfs.New instead, which provides a higher-level interface.
func WrapNamenodeConnection ¶
func WrapNamenodeConnection(conn net.Conn, user string) (*NamenodeConnection, error)
WrapNamenodeConnection wraps an existing net.Conn to a Namenode, and preforms an initial handshake.
You probably want to use hdfs.New instead, which provides a higher-level interface.
func (*NamenodeConnection) ClientName ¶ added in v1.0.0
func (c *NamenodeConnection) ClientName() string
ClientName provides a unique identifier for this client, which is required for various RPC calls. Confusingly, it's separate from clientID, which is used in the RPC header; to make things simpler, it reuses the random bytes from that, but adds a prefix to make it human-readable.
func (*NamenodeConnection) Close ¶ added in v1.0.0
func (c *NamenodeConnection) Close() error
Close terminates all underlying socket connections to remote server.
type NamenodeError ¶
NamenodeError represents an interepreted error from the Namenode, including the error code and the java backtrace.
func (*NamenodeError) Desc ¶
func (err *NamenodeError) Desc() string
Desc returns the long form of the error code, as defined in the RpcErrorCodeProto in RpcHeader.proto
func (*NamenodeError) Error ¶
func (err *NamenodeError) Error() string
Source Files