webhdfs

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• type Client
  • func NewClient(config Config, namenodes []string, username string) (Client, error)
• type Config
• type FileStatus

Types

type Client

type Client interface {
	Create(path string, src io.Reader) error
	Rename(from, to string) error
	Mkdirs(path string) error
	Open(path string, dst io.Writer) error
	GetFileStatus(path string) (FileStatus, error)
	ListFileStatus(path string) ([]FileStatus, error)
}

Client wraps webhdfs operations. All paths must be absolute.

func NewClient

func NewClient(config Config, namenodes []string, username string) (Client, error)

NewClient creates a new Client.

type Config

type Config struct {
	// BufferSize is the transfer block size.
	BufferSize datasize.ByteSize `yaml:"buffer_size"`

	// BufferGuard protects upload from draining the src reader into an oversized
	// buffer when io.Seeker is not implemented.
	BufferGuard datasize.ByteSize `yaml:"buffer_guard"`
}

Config defines Client configuration.

type FileStatus

type FileStatus struct {
	PathSuffix string `json:"pathSuffix"`
	Type       string `json:"type"`
	Length     int64  `json:"length"`
}

FileStatus defines FILESTATUS response body.