cloudpath

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 21, 2020 License: Apache-2.0 Imports: 3 Imported by: 5

Documentation

Overview

Package cloudpath provides utility routines for working with paths across both local and distributed storage systems. The set of schemes supported can be extended by providing additional implementations of the Matcher function. A cloudpath encodes two types of information:

  1. the path name itself which can be used to access the data it names.
  2. metadata about the where that filename is hosted.

For example, s3://my-bucket/a/b, contains the path '/my-bucket/a/b' as well the indication that this path is hosted on S3. Most cloud storage systems either use URI formats natively or support their use. Both AWS S3 and Google Cloud Storage support URLs: eg. storage.cloud.google.com/bucket/obj.

cloudpath provides operations for extracting both metadata and the path component, and operations for working with the extracted path directly. A common usage is to determine the 'scheme' (eg. s3, windows, unix etc) of a filename and to then operate on it appropriately. cloudpath represents a 'path' as a slice of strings to simplify often performed operations such as finding common prefixes, suffixes that are aware of the structure of the path. For example it should be possible to easily determine that s3://bucket/a/b is a prefix of https://s3.us-west-2.amazonaws.com/bucket/a/b/c.

All of the metadata for a path is represented using the Match type.

For manipulation, the path is converted to a string slice, the contents of which are documented by the Split function below.

Index

Examples

Constants

View Source 
const (
	// AWSS3 is the scheme for Amazon Web Service's S3 object store.
	AWSS3 = "s3"
	// GoogleCloudStorage is the scheme for Google's Cloud Storage object store.
	GoogleCloudStorage = "GoogleCloudStorage"
	// LocalUnixFileSystem is the scheme for unix like systems such as linux, macos etc.
	UnixFileSystem = "unix"
	// WindowsFileSystem is the scheme for msdos and windows filesystems.
	WindowsFileSystem = "windows"
)

Variables

This section is empty.

Functions

func AsPrefix 

func AsPrefix(path ...string) []string

AsPrefix returns path as a path prefix if it is not already one.

func Base 

func Base(path []string) string

Base returns the 'base', or 'filename' component of path, ie. the last one.

func HasPrefix 

func HasPrefix(path, prefix []string) bool

HasPrefix returns true if path has the specified prefix.

func HasSuffix 

func HasSuffix(path, suffix []string) bool

HasSuffix returns true if path has the specified suffix.

func Host 

func Host(path string) string

Host calls DefaultMatchers.Host(path).

func IsAbsolute 

func IsAbsolute(components []string) bool

IsAbsolute returns true if the components were derived from an absolute path.

func IsFilepath 

func IsFilepath(components []string) bool

IsFilepath returns true if the components were derived from a filepath.

func IsLocal 

func IsLocal(path string) bool

IsLocal calls DefaultMatchers.IsLocal(path).

func Join 

func Join(separator rune, components ...string) string

Join creates a string path from the supplied components. It follows the rules specified for Join. It is the inverse of Split, that is, newPath == origPath for: 

newPath = Join(sep, Split(origPath,sep)...)

func LongestCommonPrefix 

func LongestCommonPrefix(paths ...[]string) []string

func LongestCommonSuffix 

func LongestCommonSuffix(paths ...[]string) []string

func Parameters 

func Parameters(path string) map[string][]string

Parameters calls DefaultMatchers.Parameters(path).

func Path 

func Path(path string) (string, rune)

Path calls DefaultMatchers.Path(path).

func Prefix 

func Prefix(path []string) []string

Prefix returns prefix components of a path.

Example 
package main

import (
	"fmt"

	"cloudeng.io/path/cloudpath"
)

func main() {
	date := cloudpath.AsPrefix("2012-11-27")
	for _, fullname := range []string{
		"s3://my-bucket/2012-11-27/shard-0000-of-0001.json",
		"/my-local-copy/2012-11-27/shard-0000-of-0001.json",
		"https://storage.cloud.google.com/google-copy/2012-11-27/shard-0001-of-0001.json",
	} {
		components := cloudpath.SplitPath(fullname)
		fmt.Printf("%v\n", cloudpath.HasSuffix(cloudpath.Prefix(components), date))

	}
}

Output:

true
true
true

func Scheme 

func Scheme(path string) string

Scheme calls DefaultMatchers.Scheme(path).

Example 
package main

import (
	"fmt"

	"cloudeng.io/path/cloudpath"
)

func main() {
	for _, example := range []string{
		"s3://my-bucket/object",
		"https://storage.cloud.google.com/bucket/obj",
		"gs://my-bucket",
		`c:\root\file`,
	} {
		scheme := cloudpath.Scheme(example)
		local := cloudpath.IsLocal(example)
		host := cloudpath.Host(example)
		volume := cloudpath.Volume(example)
		path, sep := cloudpath.Path(example)
		parameters := cloudpath.Parameters(example)
		fmt.Printf("%v %q %q %q %q %c %v\n", local, scheme, host, volume, path, sep, parameters)
	}
}

Output:

false "s3" "" "my-bucket" "/my-bucket/object" / map[]
false "GoogleCloudStorage" "storage.cloud.google.com" "bucket" "/bucket/obj" / map[]
false "GoogleCloudStorage" "" "my-bucket" "/my-bucket" / map[]
true "windows" "localhost" "c" "c:\\root\\file" \ map[]

func Split 

func Split(path string, separator rune) []string

Split slices path into substrings according to rules designed to retain the following information:

  1. the path is absolute vs relative.
  2. the path is a prefix or a filepath.
  3. a path of zero length is represented as a nil slice and not an empty slice.

Redundant information is discarded:

  1. multiple consecutive instances of separator are treated as a single separator.

The resulting format is as follows:

  1. an empty path is represented by nil
  2. a relative path, ie. one that does not start with a separator has an empty string as the first item in the slice
  3. a path that ends with a separator has an empty string as the final component of the path

For example: 

""         => nil                // empty
"/"        => ["", ""]           // absolute, prefix
"./"       => [""]               // relative, prefix
"/abc"     => ["", "abc"]        // absolute, filepath
"abc"      => ["abc"]            // relative, filepath
"/abc/"    => ["", "abc", ""]    // absolute, prefix
"abc/"     => ["abc", ""]        // relative, prefix

func SplitPath 

func SplitPath(path string) []string

SplitPath calls Split with the results of cloudpath.Split(path).

func TrimPrefix 

func TrimPrefix(path, prefix []string) []string

TrimPrefix removes the specified prefix from path. It returns nil if path and suffix are identical.

func TrimSuffix 

func TrimSuffix(path, suffix []string) []string

TrimSuffix removes the specified suffix from path. It returns nil if path and suffix are identical.

func Volume 

func Volume(path string) string

Volume calls DefaultMatchers.Volume(path).

Types

type Match 

type Match struct {
	// Scheme uniquely identifies the filesystem being used, eg. s3 or windows.
	Scheme string
	// Local is true for local filesystems.
	Local bool
	// Host will be 'localhost' for local filesystems, the host encoded
	// in a URL or otherwise empty if there is no notion of a host.
	Host string
	// Volume will be the bucket or file system share for systems that support
	// that concept, or an empty string otherwise.
	Volume string
	// Path is the filesystem path or filename to the data. It may be a prefix
	// on a cloud based system or a directory on a local one.
	Path string
	// Separator is the filesystem separator (e.g / or \ for windows).
	Separator rune
	// Parameters are any parameters encoded in a URL/URI based name.
	Parameters map[string][]string
}

Match is the result of a successful match.

func AWSS3Matcher 

func AWSS3Matcher(p string) *Match

AWSS3Matcher implements Matcher for AWS S3 object names. It returns AWSS3 for its scheme result.

func GoogleCloudStorageMatcher 

func GoogleCloudStorageMatcher(p string) *Match

GoogleCloudStorageMatcher implements Matcher for Google Cloud Storage object names. It returns GoogleCloudStorage for its scheme result.

func UnixMatcher 

func UnixMatcher(p string) *Match

UnixMatcher implements Matcher for unix filenames. It returns UnixFileSystem for its scheme result.

func WindowsMatcher 

func WindowsMatcher(p string) *Match

WindowsMatcher implements Matcher for Windows filenames. It returns WindowsFileSystem for its scheme result.

type Matcher 

type Matcher func(p string) *Match

Matcher is the prototype for functions that parse the supplied path to determine if it matches a specific scheme and then breaks out the metadata encoded in the path. Matchers for local filesystems should return "localhost" for the host.

type MatcherSpec 

type MatcherSpec []Matcher

MatcherSpec represents a set of Matchers that will be applied in order. The ordering is important, the most specific matchers need to be applied first. For example a matcher for Windows should precede that for a Unix filesystem since the latter can accept filenames in Windows format. 

var DefaultMatchers MatcherSpec = []Matcher{
	AWSS3Matcher,
	GoogleCloudStorageMatcher,
	WindowsMatcher,
	UnixMatcher,
}

DefaultMatchers represents the built in set of Matchers.

func (MatcherSpec) Host 

func (ms MatcherSpec) Host(path string) string

Hoost returns the host component of the path if there is one.

func (*MatcherSpec) IsLocal 

func (ms *MatcherSpec) IsLocal(path string) bool

IsLocal returns true if the path is for a local filesystem.

func (MatcherSpec) Match 

func (ms MatcherSpec) Match(p string) *Match

Match applies all of the matchers in turn to match the supplied path.

func (*MatcherSpec) Parameters 

func (ms *MatcherSpec) Parameters(path string) map[string][]string

Parameters returns the parameters in path, if any. If no parameters are present an empty (rather than nil), map is returned.

func (MatcherSpec) Path 

func (ms MatcherSpec) Path(path string) (string, rune)

Path returns the path component of path and the separator to use for it.

func (MatcherSpec) Scheme 

func (ms MatcherSpec) Scheme(path string) string

Scheme returns the portion of the path that precedes a leading '//' or "" otherwise.

func (MatcherSpec) Volume 

func (ms MatcherSpec) Volume(path string) string

Volume returns the filesystem specific volume, if any, encoded in the path.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.