filesystem

README

Azure Storage Client

This program provides filesystem interfaces that make it easy to access Azure Storage Client.

Installation

• Install Golang: https://golang.org/doc/install
• Get Azure SDK package:

go get github.com/MSOpenTech/azure-sdk-for-go

• Install:

go install github.com/MSOpenTech/azure-sdk-for-go

Deployment

• follow http://azure.microsoft.com/en-gb/documentation/articles/storage-create-storage-account/ to create Azure Storage.
• Get your Azure Storage Accounts, Keys and BlobServiceBaseUrl. Like http://mystorageaccount.file.core.windows.net, the BlobServiceBaserUrl should be core.windows.net
• Use NewClient function, specify whether to use HTTPS, a specific REST API version

Example:

    import "github.com/taskgraph/taskgraph/filesystem"
    ...
    ...
    cli, err := filesystem.NewAzureClient(
        AzurestorageAccountName, 
        AzurestorageAccountKey, 
        "core.chinacloudapi.cn", 
        "2014-02-14", 
        true
    )

License

Apache 2.0

Documentation

Index

• type AzureClient
  • func NewAzureClient(accountName, accountKey, blobServiceBaseUrl, apiVersion string, useHttps bool) (*AzureClient, error)
  • func (c *AzureClient) Exists(name string) (bool, error)
  • func (c *AzureClient) Glob(pattern string) (matches []string, err error)
  • func (c *AzureClient) OpenReadCloser(name string) (io.ReadCloser, error)
  • func (c *AzureClient) OpenWriteCloser(name string) (io.WriteCloser, error)
  • func (c *AzureClient) Remove(name string) error
  • func (c *AzureClient) Rename(oldpath, newpath string) error
• type AzureFile
  • func (f *AzureFile) Close() error
  • func (f *AzureFile) Write(b []byte) (int, error)
• type Client
  • func NewHdfsClient(namenodeAddr, webHdfsAddr, user string) (Client, error)
  • func NewLocalFSClient() Client
• type HdfsClient
  • func (c *HdfsClient) Exists(name string) (bool, error)
  • func (c *HdfsClient) Glob(pattern string) (matches []string, err error)
  • func (c *HdfsClient) OpenReadCloser(name string) (io.ReadCloser, error)
  • func (c *HdfsClient) OpenWriteCloser(name string) (io.WriteCloser, error)
  • func (c *HdfsClient) Remove(name string) error
  • func (c *HdfsClient) Rename(oldpath, newpath string) error
• type HdfsFile
  • func (f *HdfsFile) Close() error
  • func (f *HdfsFile) Write(b []byte) (int, error)

Types

type AzureClient

type AzureClient struct {
	// contains filtered or unexported fields
}

func NewAzureClient

func NewAzureClient(accountName, accountKey, blobServiceBaseUrl, apiVersion string, useHttps bool) (*AzureClient, error)

NewAzureClient function NewClient constructs a StorageClient and blobStorageClinet. This should be used if the caller wants to specify whether to use HTTPS, a specific REST API version or a custom storage endpoint than Azure Public Cloud. Recommended API version "2014-02-14" synax : AzurestorageAccountName, AzurestorageAccountKey, "core.chinacloudapi.cn", "2014-02-14", true

func (*AzureClient) Exists

func (c *AzureClient) Exists(name string) (bool, error)

AzureClient -> Exist function support check the contianer or blob if exist or not

func (*AzureClient) Glob

func (c *AzureClient) Glob(pattern string) (matches []string, err error)

AzureClient -> Glob function only supports '*', '?' Syntax: cntName?/part.*

func (*AzureClient) OpenReadCloser

func (c *AzureClient) OpenReadCloser(name string) (io.ReadCloser, error)

AzureClient -> OpenReadCloser function implement by the providing function

func (*AzureClient) OpenWriteCloser

func (c *AzureClient) OpenWriteCloser(name string) (io.WriteCloser, error)

AzureClient -> OpenWriteCloser function If not exist, Create corresponding Container and blob.

func (*AzureClient) Remove

func (c *AzureClient) Remove(name string) error

AzureClient -> Delete function Delete specific Blob for input path

func (*AzureClient) Rename

func (c *AzureClient) Rename(oldpath, newpath string) error

AzureClient -> Rename function support user renaming contianer/blob

type AzureFile

type AzureFile struct {
	// contains filtered or unexported fields
}

func (*AzureFile) Close

func (f *AzureFile) Close() error

func (*AzureFile) Write

func (f *AzureFile) Write(b []byte) (int, error)

Follow io.Writer rules use BlockList mechanism of Azure Blob to achieve this interface BlockList divided block into committed block and uncommitted ones For every write op of user, it create new block list with old commited block and user write content, and commit this one to modifies the blob commited block list If failed in the process, it would write nothing to blob return 0, error to user

type Client

type Client interface {
	// Opens the file for reading.
	OpenReadCloser(name string) (io.ReadCloser, error)
	// Opens the file for writing.
	// It creates the file if it didn't exist.
	OpenWriteCloser(name string) (io.WriteCloser, error)
	Exists(name string) (bool, error)
	Rename(oldpath, newpath string) error
	// Glob returns the names of all files matching pattern or
	// nil if there is no matching file.
	Glob(pattern string) (matches []string, err error)
	// Remove specific file in filesystem
	Remove(name string) error
}

func NewHdfsClient

func NewHdfsClient(namenodeAddr, webHdfsAddr, user string) (Client, error)

func NewLocalFSClient

func NewLocalFSClient() Client

type HdfsClient

type HdfsClient struct {
	// contains filtered or unexported fields
}

func (*HdfsClient) Exists

func (c *HdfsClient) Exists(name string) (bool, error)

func (*HdfsClient) Glob

func (c *HdfsClient) Glob(pattern string) (matches []string, err error)

only supports '*', '?' Syntax:

/user/hdfs/etl*/part.*

func (*HdfsClient) OpenReadCloser

func (c *HdfsClient) OpenReadCloser(name string) (io.ReadCloser, error)

func (*HdfsClient) OpenWriteCloser

func (c *HdfsClient) OpenWriteCloser(name string) (io.WriteCloser, error)

func (*HdfsClient) Remove

func (c *HdfsClient) Remove(name string) error

func (*HdfsClient) Rename

func (c *HdfsClient) Rename(oldpath, newpath string) error

type HdfsFile

type HdfsFile struct {
	// contains filtered or unexported fields
}

func (*HdfsFile) Close

func (f *HdfsFile) Close() error

func (*HdfsFile) Write

func (f *HdfsFile) Write(b []byte) (int, error)

REST docs: http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Append_to_a_File