filesystem

README

Azure Storage Client

This program provides filesystem interfaces that make it easy to access Azure Storage Client.

Installation

• Install Golang: https://golang.org/doc/install
• Get Azure SDK package:

go get github.com/MSOpenTech/azure-sdk-for-go

• Install:

go install github.com/MSOpenTech/azure-sdk-for-go

Deployment

• follow http://azure.microsoft.com/en-gb/documentation/articles/storage-create-storage-account/ to create Azure Storage.
• Get your Azure Storage Accounts, Keys and BlobServiceBaseUrl. Like http://mystorageaccount.file.core.windows.net, the BlobServiceBaserUrl should be core.windows.net
• Use NewClient function, specify whether to use HTTPS, a specific REST API version

Example:

    import "github.com/taskgraph/taskgraph/filesystem"
    ...
    ...
    cli, err := filesystem.NewAzureClient(
        AzurestorageAccountName, 
        AzurestorageAccountKey, 
        "core.chinacloudapi.cn", 
        "2014-02-14", 
        true
    )

License

Apache 2.0

Documentation

Index

• type AzureClient
  • func NewAzureClient(accountName, accountKey, blobServiceBaseUrl, apiVersion string, useHttps bool) (*AzureClient, error)
  • func (c *AzureClient) Exists(name string) (bool, error)
  • func (c *AzureClient) Glob(pattern string) (matches []string, err error)
  • func (c *AzureClient) OpenReadCloser(name string) (io.ReadCloser, error)
  • func (c *AzureClient) OpenWriteCloser(name string) (io.WriteCloser, error)
  • func (c *AzureClient) Remove(name string) error
  • func (c *AzureClient) Rename(oldpath, newpath string) error
• type AzureFile
  • func (f *AzureFile) Close() error
  • func (f *AzureFile) Write(b []byte) (int, error)
• type Client
  • func NewHdfsClient(namenodeAddr, webHdfsAddr, user string) (Client, error)
  • func NewLocalFSClient() Client
• type HdfsClient
  • func (c *HdfsClient) Exists(name string) (bool, error)
  • func (c *HdfsClient) Glob(pattern string) (matches []string, err error)
  • func (c *HdfsClient) OpenReadCloser(name string) (io.ReadCloser, error)
  • func (c *HdfsClient) OpenWriteCloser(name string) (io.WriteCloser, error)
  • func (c *HdfsClient) Remove(name string) error
  • func (c *HdfsClient) Rename(oldpath, newpath string) error
• type HdfsFile
  • func (f *HdfsFile) Close() error
  • func (f *HdfsFile) Write(b []byte) (int, error)

Types

type AzureClient

type AzureClient struct {
	// contains filtered or unexported fields
}

func NewAzureClient

func NewAzureClient(accountName, accountKey, blobServiceBaseUrl, apiVersion string, useHttps bool) (*AzureClient, error)

NewAzureClient function NewClient constructs a StorageClient and blobStorageClinet. This should be used if the caller wants to specify whether to use HTTPS, a specific REST API version or a custom storage endpoint than Azure Public Cloud. Recommended API version "2014-02-14" synax : AzurestorageAccountName, AzurestorageAccountKey, "core.chinacloudapi.cn", "2014-02-14", true

func (*AzureClient) Exists

func (c *AzureClient) Exists(name string) (bool, error)

AzureClient -> Exist function support check the contianer or blob if exist or not

func (*AzureClient) Glob

func (c *AzureClient) Glob(pattern string) (matches []string, err error)

AzureClient -> Glob function only supports '*', '?' Syntax: cntName?/part.*

func (*AzureClient) OpenReadCloser

func (c *AzureClient) OpenReadCloser(name string) (io.ReadCloser, error)

AzureClient -> OpenReadCloser function implement by the providing function

func (*AzureClient) OpenWriteCloser

func (c *AzureClient) OpenWriteCloser(name string) (io.WriteCloser, error)

AzureClient -> OpenWriteCloser function If not exist, Create corresponding Container and blob.

func (*AzureClient) Remove

func (c *AzureClient) Remove(name string) error

AzureClient -> Delete function Delete specific Blob for input path

func (*AzureClient) Rename

func (c *AzureClient) Rename(oldpath, newpath string) error

AzureClient -> Rename function support rename contianer and blob

type AzureFile

type AzureFile struct {
	// contains filtered or unexported fields
}

func (*AzureFile) Close

func (f *AzureFile) Close() error

func (*AzureFile) Write

func (f *AzureFile) Write(b []byte) (int, error)

type Client

type Client interface {
	// Opens the file for reading.
	OpenReadCloser(name string) (io.ReadCloser, error)
	// Opens the file for writing.
	// It creates the file if it didn't exist.
	OpenWriteCloser(name string) (io.WriteCloser, error)
	Exists(name string) (bool, error)
	Rename(oldpath, newpath string) error
	// Glob returns the names of all files matching pattern or
	// nil if there is no matching file.
	Glob(pattern string) (matches []string, err error)
	// Remove specific file in filesystem
	Remove(name string) error
}

func NewHdfsClient

func NewHdfsClient(namenodeAddr, webHdfsAddr, user string) (Client, error)

func NewLocalFSClient

func NewLocalFSClient() Client

type HdfsClient

type HdfsClient struct {
	// contains filtered or unexported fields
}

func (*HdfsClient) Exists

func (c *HdfsClient) Exists(name string) (bool, error)

func (*HdfsClient) Glob

func (c *HdfsClient) Glob(pattern string) (matches []string, err error)

only supports '*', '?' Syntax:

/user/hdfs/etl*/part.*

func (*HdfsClient) OpenReadCloser

func (c *HdfsClient) OpenReadCloser(name string) (io.ReadCloser, error)

func (*HdfsClient) OpenWriteCloser

func (c *HdfsClient) OpenWriteCloser(name string) (io.WriteCloser, error)

func (*HdfsClient) Remove

func (c *HdfsClient) Remove(name string) error

func (*HdfsClient) Rename

func (c *HdfsClient) Rename(oldpath, newpath string) error

type HdfsFile

type HdfsFile struct {
	// contains filtered or unexported fields
}

func (*HdfsFile) Close

func (f *HdfsFile) Close() error

func (*HdfsFile) Write

func (f *HdfsFile) Write(b []byte) (int, error)

REST docs: http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Append_to_a_File