Documentation
¶
Overview ¶
Package fileblob provides a blob implementation that uses the filesystem. Use OpenBucket to construct a *blob.Bucket.
To avoid partial writes, fileblob writes to a temporary file and then renames the temporary file to the final path on Close. By default, it creates these temporary files in `os.TempDir`. If `os.TempDir` is on a different mount than your base bucket path, the `os.Rename` will fail with `invalid cross-device link`. To avoid this, either configure the temp dir to use by setting the environment variable `TMPDIR`, or set `Options.NoTempDir` to `true` (fileblob will create the temporary files next to the actual files instead of in a temporary directory).
By default fileblob stores blob metadata in "sidecar" files under the original filename with an additional ".attrs" suffix. This behaviour can be changed via `Options.Metadata`; writing of those metadata files can be suppressed by setting it to `MetadataDontWrite` or its equivalent "metadata=skip" in the URL for the opener. In either case, absent any stored metadata many `blob.Attributes` fields will be set to default values.
URLs ¶
For blob.OpenBucket, fileblob registers for the scheme "file". To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.
Escaping ¶
Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for fileblob:
- Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__". If os.PathSeparator != "/", it is also escaped. Additionally, the "/" in "../", the trailing "/" in "//", and a trailing "/" is key names are escaped in the same way. On Windows, the characters "<>:"|?*" are also escaped.
As ¶
fileblob exposes the following types for As:
- Bucket: os.FileInfo
- Error: *os.PathError
- ListObject: os.FileInfo
- Reader: io.Reader
- ReaderOptions.BeforeRead: *os.File
- Attributes: os.FileInfo
- CopyOptions.BeforeCopy: *os.File
- WriterOptions.BeforeWrite: *os.File
Example (OpenBucketFromURL) ¶
package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
"strings"
"gocloud.dev/blob"
)
func main() {
// Create a temporary directory.
dir, err := ioutil.TempDir("", "go-cloud-fileblob-example")
if err != nil {
log.Fatal(err)
}
defer os.RemoveAll(dir)
// On Unix, append the dir to "file://".
// On Windows, convert "\" to "/" and add a leading "/":
dirpath := filepath.ToSlash(dir)
if os.PathSeparator != '/' && !strings.HasPrefix(dirpath, "/") {
dirpath = "/" + dirpath
}
// blob.OpenBucket creates a *blob.Bucket from a URL.
ctx := context.Background()
b, err := blob.OpenBucket(ctx, "file://"+dirpath)
if err != nil {
log.Fatal(err)
}
defer b.Close()
// Now we can use b to read or write files to the container.
err = b.WriteAll(ctx, "my-key", []byte("hello world"), nil)
if err != nil {
log.Fatal(err)
}
data, err := b.ReadAll(ctx, "my-key")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
}
Output: hello world
Index ¶
Examples ¶
Constants ¶
const ( // Metadata gets written to a separate file. MetadataInSidecar metadataOption = "" // Writes won't carry metadata, as per the package docstring. MetadataDontWrite metadataOption = "skip" )
Settings for Options.Metadata.
const Scheme = "file"
Scheme is the URL scheme fileblob registers its URLOpener under on blob.DefaultMux.
Variables ¶
This section is empty.
Functions ¶
func OpenBucket ¶
OpenBucket creates a *blob.Bucket backed by the filesystem and rooted at dir, which must exist. See the package documentation for an example.
Example ¶
package main
import (
"log"
"os"
"gocloud.dev/blob/fileblob"
)
func main() {
// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// The directory you pass to fileblob.OpenBucket must exist first.
const myDir = "path/to/local/directory"
if err := os.MkdirAll(myDir, 0777); err != nil {
log.Fatal(err)
}
// Create a file-based bucket.
bucket, err := fileblob.OpenBucket(myDir, nil)
if err != nil {
log.Fatal(err)
}
defer bucket.Close()
}
Output:
Types ¶
type Options ¶
type Options struct {
// URLSigner implements signing URLs (to allow access to a resource without
// further authorization) and verifying that a given URL is unexpired and
// contains a signature produced by the URLSigner.
// URLSigner is only required for utilizing the SignedURL API.
URLSigner URLSigner
// If true, create the directory backing the Bucket if it does not exist
// (using os.MkdirAll).
CreateDir bool
// If true, don't use os.TempDir for temporary files, but instead place them
// next to the actual files. This may result in "stranded" temporary files
// (e.g., if the application is killed before the file cleanup runs).
//
// If your bucket directory is on a different mount than os.TempDir, you will
// need to set this to true, as os.Rename will fail across mount points.
NoTempDir bool
// Refers to the strategy for how to deal with metadata (such as blob.Attributes).
// For supported values please see the Metadata* constants.
// If left unchanged, 'MetadataInSidecar' will be used.
Metadata metadataOption
}
Options sets options for constructing a *blob.Bucket backed by fileblob.
type URLOpener ¶ added in v0.10.0
type URLOpener struct {
// Options specifies the default options to pass to OpenBucket.
Options Options
}
URLOpener opens file bucket URLs like "file:///foo/bar/baz".
The URL's host is ignored unless it is ".", which is used to signal a relative path. For example, "file://./../.." uses "../.." as the path.
If os.PathSeparator != "/", any leading "/" from the path is dropped and remaining '/' characters are converted to os.PathSeparator.
The following query parameters are supported:
- create_dir: (any non-empty value) the directory is created (using os.MkDirAll) if it does not already exist.
- no_tmp_dir: (any non-empty value) temporary files are created next to the final path instead of in os.TempDir.
- base_url: the base URL to use to construct signed URLs; see URLSignerHMAC
- secret_key_path: path to read for the secret key used to construct signed URLs; see URLSignerHMAC
- metadata: if set to "skip", won't write metadata such as blob.Attributes as per the package docstring
If either of base_url / secret_key_path are provided, both must be.
- file:///a/directory -> Passes "/a/directory" to OpenBucket.
- file://localhost/a/directory -> Also passes "/a/directory".
- file://./../.. -> The hostname is ".", signaling a relative path; passes "../..".
- file:///c:/foo/bar on Windows. -> Passes "c:\foo\bar".
- file://localhost/c:/foo/bar on Windows. -> Also passes "c:\foo\bar".
- file:///a/directory?base_url=/show&secret_key_path=secret.key -> Passes "/a/directory" to OpenBucket, and sets Options.URLSigner to a URLSignerHMAC initialized with base URL "/show" and secret key bytes read from the file "secret.key".
type URLSigner ¶ added in v0.11.0
type URLSigner interface {
// URLFromKey defines how the bucket's object key will be turned
// into a signed URL. URLFromKey must be safe to call from multiple goroutines.
URLFromKey(ctx context.Context, key string, opts *driver.SignedURLOptions) (*url.URL, error)
// KeyFromURL must be able to validate a URL returned from URLFromKey.
// KeyFromURL must only return the object if if the URL is
// both unexpired and authentic. KeyFromURL must be safe to call from
// multiple goroutines. Implementations of KeyFromURL should not modify
// the URL argument.
KeyFromURL(ctx context.Context, surl *url.URL) (string, error)
}
URLSigner defines an interface for creating and verifying a signed URL for objects in a fileblob bucket. Signed URLs are typically used for granting access to an otherwise-protected resource without requiring further authentication, and callers should take care to restrict the creation of signed URLs as is appropriate for their application.
type URLSignerHMAC ¶ added in v0.11.0
type URLSignerHMAC struct {
// contains filtered or unexported fields
}
URLSignerHMAC signs URLs by adding the object key, expiration time, and a hash-based message authentication code (HMAC) into the query parameters. Values of URLSignerHMAC with the same secret key will accept URLs produced by others as valid.
func NewURLSignerHMAC ¶ added in v0.11.0
func NewURLSignerHMAC(baseURL *url.URL, secretKey []byte) *URLSignerHMAC
NewURLSignerHMAC creates a URLSignerHMAC. If the secret key is empty, then NewURLSignerHMAC panics.
func (*URLSignerHMAC) KeyFromURL ¶ added in v0.11.0
KeyFromURL checks expiry and signature, and returns the object key only if the signed URL is both authentic and unexpired.
func (*URLSignerHMAC) URLFromKey ¶ added in v0.11.0
func (h *URLSignerHMAC) URLFromKey(ctx context.Context, key string, opts *driver.SignedURLOptions) (*url.URL, error)
URLFromKey creates a signed URL by copying the baseURL and appending the object key, expiry, and signature as a query params.
Source Files