gofakes3

package module
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 26, 2024 License: MIT Imports: 29 Imported by: 3

README

Logo

Build Status Go Report Card GoDoc

This is a fork of johannesboyne/gofakes3 mainly for use implementing the rclone serves3 command in rclone/rclone.

Notable differences:

  • Use modified xml library to handle more control chars
  • Func getVersioningConfiguration will return empty when unversioned
  • New func in backend interface: CopyObject
  • Support authentication with AWS Signature V4
  • Interfaces changed to take context

Documentation

Overview

Package s3 implements a fake s3 server for rclone

Index

Constants

View Source 
const (
	// From https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html:
	//	"The name for a key is a sequence of Unicode characters whose UTF-8
	//	encoding is at most 1024 bytes long."
	KeySizeLimit = 1024

	// From https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html:
	//	Within the PUT request header, the user-defined metadata is limited to 2
	// 	KB in size. The size of user-defined metadata is measured by taking the
	// 	sum of the number of bytes in the UTF-8 encoding of each key and value.
	//
	// As this does not specify KB or KiB, KB is used in gofakes3. The reason
	// for this is if gofakes3 is used for testing, and your tests show that
	// 2KiB works, but Amazon uses 2KB...  that's a much worse time to discover
	// the disparity!
	DefaultMetadataSizeLimit = 2000

	// Like DefaultMetadataSizeLimit, the docs don't specify MB or MiB, so we
	// will accept 5MB for now. The Go client SDK rejects 5MB with the error
	// "part size must be at least 5242880 bytes", which is a hint that it
	// has been interpreted as MiB at least _somewhere_, but we should remain
	// liberal in what we accept in the face of ambiguity.
	DefaultUploadPartSize = 5 * 1000 * 1000

	DefaultSkewLimit = 15 * time.Minute

	MaxUploadsLimit       = 1000
	DefaultMaxUploads     = 1000
	MaxUploadPartsLimit   = 1000
	DefaultMaxUploadParts = 1000

	MaxBucketKeys        = 1000
	DefaultMaxBucketKeys = 1000

	MaxBucketVersionKeys        = 1000
	DefaultMaxBucketVersionKeys = 1000

	// From the docs: "Part numbers can be any number from 1 to 10,000, inclusive."
	MaxUploadPartNumber = 10000
)
View Source 
const (
	DefaultBucketVersionKeys = 1000
)
View Source 
const RangeNoEnd = -1

Variables

This section is empty.

Functions

func BucketNotFound 

func BucketNotFound(bucket string) error

func CheckClose 

func CheckClose(c io.Closer, err *error)

CheckClose is a utility function used to check the return from Close in a defer statement.

func ErrorInvalidArgument 

func ErrorInvalidArgument(name, value, message string) error

func ErrorMessage 

func ErrorMessage(code ErrorCode, message string) error

func ErrorMessagef 

func ErrorMessagef(code ErrorCode, message string, args ...interface{}) error

func HasErrorCode 

func HasErrorCode(err error, code ErrorCode) bool

HasErrorCode asserts that the error has a specific error code: 

if HasErrorCode(err, ErrNoSuchBucket) {
	// handle condition
}

If err is nil and code is ErrNone, HasErrorCode returns true.

func IsAlreadyExists 

func IsAlreadyExists(err error) bool

IsAlreadyExists asserts that the error is a kind that indicates the resource already exists, similar to os.IsExist.

func KeyNotFound 

func KeyNotFound(key string) error

func MergeMetadata 

func MergeMetadata(ctx context.Context, db Backend, bucketName string, objectName string, meta map[string]string) error

func ReadAll 

func ReadAll(r io.Reader, size int64) (b []byte, err error)

ReadAll is a fakeS3-centric replacement for ioutil.ReadAll(), for use when the size of the result is known ahead of time. It is considerably faster to preallocate the entire slice than to allow growslice to be triggered repeatedly, especially with larger buffers.

It also reports S3-specific errors in certain conditions, like ErrIncompleteBody.

func ResourceError 

func ResourceError(code ErrorCode, resource string) error

func URLEncode 

func URLEncode(s string) string

s3URLEncode is based on Golang's url.QueryEscape() code, while considering some S3 exceptions:

  • Avoid encoding '/' and '*'
  • Force encoding of '~'

func ValidateBucketName 

func ValidateBucketName(name string) error

ValidateBucketName applies the rules from the AWS docs: https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules

1. Bucket names must comply with DNS naming conventions. 2. Bucket names must be at least 3 and no more than 63 characters long. 3. Bucket names must not contain uppercase characters or underscores. 4. Bucket names must start with a lowercase letter or number.

The DNS RFC confirms that the valid range of characters in an LDH label is 'a-z0-9-': https://tools.ietf.org/html/rfc5890#section-2.3.1

Types

type Backend 

type Backend interface {
	// ListBuckets returns a list of all buckets owned by the authenticated
	// sender of the request.
	// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTServiceGET.html
	ListBuckets(ctx context.Context) ([]BucketInfo, error)

	// ListBucket returns the contents of a bucket. Backends should use the
	// supplied prefix to limit the contents of the bucket and to sort the
	// matched items into the Contents and CommonPrefixes fields.
	//
	// ListBucket must return a gofakes3.ErrNoSuchBucket error if the bucket
	// does not exist. See gofakes3.BucketNotFound() for a convenient way to create one.
	//
	// The prefix MUST be correctly handled for the backend to be valid. Each
	// item you consider returning should be checked using prefix.Match(name),
	// even if the prefix is empty. The Backend MUST treat a nil prefix
	// identically to a zero prefix.
	//
	// At this stage, implementers MAY return gofakes3.ErrInternalPageNotImplemented
	// if the page argument is non-empty. In this case, gofakes3 may or may
	// not, depending on how it was configured, retry the same request with no page.
	// We have observed (though not yet confirmed) that simple clients tend to
	// work fine if you ignore the pagination request, but this may not suit
	// your application. Not all backends bundled with gofakes3 correctly
	// support this pagination yet, but that will change.
	ListBucket(ctx context.Context, name string, prefix *Prefix, page ListBucketPage) (*ObjectList, error)

	// CreateBucket creates the bucket if it does not already exist. The name
	// should be assumed to be a valid name.
	//
	// If the bucket already exists, a gofakes3.ResourceError with
	// gofakes3.ErrBucketAlreadyExists MUST be returned.
	CreateBucket(ctx context.Context, name string) error

	// BucketExists should return a boolean indicating the bucket existence, or
	// an error if the backend was unable to determine existence.
	BucketExists(ctx context.Context, name string) (exists bool, err error)

	// DeleteBucket deletes a bucket if and only if it is empty.
	//
	// If the bucket is not empty, gofakes3.ResourceError with
	// gofakes3.ErrBucketNotEmpty MUST be returned.
	//
	// If the bucket does not exist, gofakes3.ErrNoSuchBucket MUST be returned.
	//
	// AWS does not validate the bucket's name for anything other than existence.
	DeleteBucket(ctx context.Context, name string) error

	// GetObject must return a gofakes3.ErrNoSuchKey error if the object does
	// not exist. See gofakes3.KeyNotFound() for a convenient way to create
	// one.
	//
	// If the returned Object is not nil, you MUST call Object.Contents.Close(),
	// otherwise you will leak resources. Implementers should return a no-op
	// implementation of io.ReadCloser.
	//
	// If rnge is nil, it is assumed you want the entire object. If rnge is not
	// nil, but the underlying backend does not support range requests,
	// implementers MUST return ErrNotImplemented.
	//
	// If the backend is a VersionedBackend, GetObject retrieves the latest version.
	GetObject(ctx context.Context, bucketName, objectName string, rangeRequest *ObjectRangeRequest) (*Object, error)

	// HeadObject fetches the Object from the backend, but reading the Contents
	// will return io.EOF immediately.
	//
	// If the returned Object is not nil, you MUST call Object.Contents.Close(),
	// otherwise you will leak resources. Implementers should return a no-op
	// implementation of io.ReadCloser.
	//
	// HeadObject should return a NotFound() error if the object does not
	// exist.
	HeadObject(ctx context.Context, bucketName, objectName string) (*Object, error)

	// DeleteObject deletes an object from the bucket.
	//
	// If the backend is a VersionedBackend and versioning is enabled, this
	// should introduce a delete marker rather than actually delete the object.
	//
	// DeleteObject must return a gofakes3.ErrNoSuchBucket error if the bucket
	// does not exist. See gofakes3.BucketNotFound() for a convenient way to create one.
	// FIXME: confirm with S3 whether this is the correct behaviour.
	//
	// DeleteObject must not return an error if the object does not exist. Source:
	// https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.DeleteObject:
	//
	//	Removes the null version (if there is one) of an object and inserts a
	//	delete marker, which becomes the latest version of the object. If there
	//	isn't a null version, Amazon S3 does not remove any objects.
	//
	DeleteObject(ctx context.Context, bucketName, objectName string) (ObjectDeleteResult, error)

	// PutObject should assume that the key is valid. The map containing meta
	// may be nil.
	//
	// The size can be used if the backend needs to read the whole reader; use
	// gofakes3.ReadAll() for this job rather than ioutil.ReadAll().
	PutObject(ctx context.Context, bucketName, key string, meta map[string]string, input io.Reader, size int64) (PutObjectResult, error)

	DeleteMulti(ctx context.Context, bucketName string, objects ...string) (MultiDeleteResult, error)

	CopyObject(ctx context.Context, srcBucket, srcKey, dstBucket, dstKey string, meta map[string]string) (CopyObjectResult, error)
}

Backend provides a set of operations to be implemented in order to support gofakes3.

The Backend API is not yet stable; if you create your own Backend, breakage is likely until this notice is removed.

type BucketInfo 

type BucketInfo struct {
	Name string `xml:"Name"`

	// CreationDate is required; without it, boto returns the error "('String
	// does not contain a date:', ”)"
	CreationDate ContentTime `xml:"CreationDate"`
}

BucketInfo represents a single bucket returned by the ListBuckets response.

type Buckets 

type Buckets []BucketInfo

func (Buckets) Names 

func (b Buckets) Names() []string

Names is a deterministic convenience function returning a sorted list of bucket names.

type CommonPrefix 

type CommonPrefix struct {
	Prefix string `xml:"Prefix"`
}

CommonPrefix is used in Bucket.CommonPrefixes to list partial delimited keys that represent pseudo-directories.

type CompleteMultipartUploadRequest 

type CompleteMultipartUploadRequest struct {
	Parts []CompletedPart `xml:"Part"`
}

type CompleteMultipartUploadResult 

type CompleteMultipartUploadResult struct {
	Location string `xml:"Location"`
	Bucket   string `xml:"Bucket"`
	Key      string `xml:"Key"`
	ETag     string `xml:"ETag"`
}

type CompletedPart 

type CompletedPart struct {
	PartNumber int    `xml:"PartNumber"`
	ETag       string `xml:"ETag"`
}

type Content 

type Content struct {
	Key          string       `xml:"Key"`
	LastModified ContentTime  `xml:"LastModified"`
	ETag         string       `xml:"ETag"`
	Size         int64        `xml:"Size"`
	StorageClass StorageClass `xml:"StorageClass,omitempty"`
	Owner        *UserInfo    `xml:"Owner,omitempty"`
}

type ContentTime 

type ContentTime struct {
	time.Time
}

func NewContentTime 

func NewContentTime(t time.Time) ContentTime

func (ContentTime) MarshalXML 

func (c ContentTime) MarshalXML(e *xml.Encoder, start xml.StartElement) error

type CopyObjectResult 

type CopyObjectResult struct {
	XMLName      xml.Name    `xml:"CopyObjectResult"`
	ETag         string      `xml:"ETag,omitempty"`
	LastModified ContentTime `xml:"LastModified,omitempty"`
}

CopyObjectResult contains the response from a CopyObject operation.

type DeleteMarker 

type DeleteMarker struct {
	XMLName      xml.Name    `xml:"DeleteMarker"`
	Key          string      `xml:"Key"`
	VersionID    VersionID   `xml:"VersionId"`
	IsLatest     bool        `xml:"IsLatest"`
	LastModified ContentTime `xml:"LastModified,omitempty"`
	Owner        *UserInfo   `xml:"Owner,omitempty"`
}

func (DeleteMarker) GetVersionID 

func (d DeleteMarker) GetVersionID() VersionID

type DeleteRequest 

type DeleteRequest struct {
	Objects []ObjectID `xml:"Object"`

	// Element to enable quiet mode for the request. When you add this element,
	// you must set its value to true.
	//
	// By default, the operation uses verbose mode in which the response
	// includes the result of deletion of each key in your request. In quiet
	// mode the response includes only keys where the delete operation
	// encountered an error. For a successful deletion, the operation does not
	// return any information about the delete in the response body.
	Quiet bool `xml:"Quiet"`
}

type Error 

type Error interface {
	error
	ErrorCode() ErrorCode
}

type ErrorCode 

type ErrorCode string

ErrorCode represents an S3 error code, documented here: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html 

const (
	ErrNone ErrorCode = ""

	// The Content-MD5 you specified did not match what we received.
	ErrBadDigest ErrorCode = "BadDigest"

	ErrBucketAlreadyExists ErrorCode = "BucketAlreadyExists"

	// Raised when attempting to delete a bucket that still contains items.
	ErrBucketNotEmpty ErrorCode = "BucketNotEmpty"

	// "Indicates that the versioning configuration specified in the request is invalid"
	ErrIllegalVersioningConfiguration ErrorCode = "IllegalVersioningConfigurationException"

	// You did not provide the number of bytes specified by the Content-Length
	// HTTP header:
	ErrIncompleteBody ErrorCode = "IncompleteBody"

	// POST requires exactly one file upload per request.
	ErrIncorrectNumberOfFilesInPostRequest ErrorCode = "IncorrectNumberOfFilesInPostRequest"

	// InlineDataTooLarge occurs when using the PutObjectInline method of the
	// SOAP interface
	// (https://docs.aws.amazon.com/AmazonS3/latest/API/SOAPPutObjectInline.html).
	// This is not documented on the errors page; the error is included here
	// only for reference.
	ErrInlineDataTooLarge ErrorCode = "InlineDataTooLarge"

	ErrInvalidArgument ErrorCode = "InvalidArgument"

	// https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules
	ErrInvalidBucketName ErrorCode = "InvalidBucketName"

	// The Content-MD5 you specified is not valid.
	ErrInvalidDigest ErrorCode = "InvalidDigest"

	ErrInvalidRange         ErrorCode = "InvalidRange"
	ErrInvalidToken         ErrorCode = "InvalidToken"
	ErrKeyTooLong           ErrorCode = "KeyTooLongError" // This is not a typo: Error is part of the string, but redundant in the constant name
	ErrMalformedPOSTRequest ErrorCode = "MalformedPOSTRequest"

	// One or more of the specified parts could not be found. The part might
	// not have been uploaded, or the specified entity tag might not have
	// matched the part's entity tag.
	ErrInvalidPart ErrorCode = "InvalidPart"

	// The list of parts was not in ascending order. Parts list must be
	// specified in order by part number.
	ErrInvalidPartOrder ErrorCode = "InvalidPartOrder"

	ErrInvalidURI ErrorCode = "InvalidURI"

	ErrMetadataTooLarge ErrorCode = "MetadataTooLarge"
	ErrMethodNotAllowed ErrorCode = "MethodNotAllowed"
	ErrMalformedXML     ErrorCode = "MalformedXML"

	// You must provide the Content-Length HTTP header.
	ErrMissingContentLength ErrorCode = "MissingContentLength"

	// See BucketNotFound() for a helper function for this error:
	ErrNoSuchBucket ErrorCode = "NoSuchBucket"

	// See KeyNotFound() for a helper function for this error:
	ErrNoSuchKey ErrorCode = "NoSuchKey"

	// The specified multipart upload does not exist. The upload ID might be
	// invalid, or the multipart upload might have been aborted or completed.
	ErrNoSuchUpload ErrorCode = "NoSuchUpload"

	ErrNoSuchVersion ErrorCode = "NoSuchVersion"

	// No need to retransmit the object
	ErrNotModified ErrorCode = "NotModified"

	ErrRequestTimeTooSkewed ErrorCode = "RequestTimeTooSkewed"
	ErrTooManyBuckets       ErrorCode = "TooManyBuckets"
	ErrNotImplemented       ErrorCode = "NotImplemented"

	ErrInternal ErrorCode = "InternalError"
)

Error codes are documented here: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

If you add a code to this list, please also add it to ErrorCode.Status().

func (ErrorCode) Error 

func (e ErrorCode) Error() string

func (ErrorCode) ErrorCode 

func (e ErrorCode) ErrorCode() ErrorCode

func (ErrorCode) Message 

func (e ErrorCode) Message() string

Message tries to return the same string as S3 would return for the error response, when it is known, or nothing when it is not. If you see the status text for a code we don't have listed in here in the wild, please let us know!

func (ErrorCode) Status 

func (e ErrorCode) Status() int

type ErrorInvalidArgumentResponse 

type ErrorInvalidArgumentResponse struct {
	ErrorResponse

	ArgumentName  string `xml:"ArgumentName"`
	ArgumentValue string `xml:"ArgumentValue"`
}

type ErrorResponse 

type ErrorResponse struct {
	XMLName xml.Name `xml:"Error"`

	Code      ErrorCode
	Message   string `xml:",omitempty"`
	RequestID string `xml:"RequestId,omitempty"`
	HostID    string `xml:"HostId,omitempty"`
}

ErrorResponse is the base error type returned by S3 when any error occurs.

Some errors contain their own additional fields in the response, for example ErrRequestTimeTooSkewed, which contains the server time and the skew limit. To create one of these responses, subclass it (but please don't export it): 

type notQuiteRightResponse struct {
	ErrorResponse
	ExtraField int
}

Next, create a constructor that populates the error. Interfaces won't work for this job as the error itself does double-duty as the XML response object. Fill the struct out however you please, but don't forget to assign Code and Message: 

func NotQuiteRight(at time.Time, max time.Duration) error {
    code := ErrNotQuiteRight
    return &notQuiteRightResponse{
        ErrorResponse{Code: code, Message: code.Message()},
        123456789,
    }
}

func (*ErrorResponse) Error 

func (e *ErrorResponse) Error() string

func (*ErrorResponse) ErrorCode 

func (e *ErrorResponse) ErrorCode() ErrorCode

type ErrorResult 

type ErrorResult struct {
	XMLName   xml.Name  `xml:"Error"`
	Key       string    `xml:"Key,omitempty"`
	Code      ErrorCode `xml:"Code,omitempty"`
	Message   string    `xml:"Message,omitempty"`
	Resource  string    `xml:"Resource,omitempty"`
	RequestID string    `xml:"RequestId,omitempty"`
}

func ErrorResultFromError 

func ErrorResultFromError(err error) ErrorResult

func (ErrorResult) String 

func (er ErrorResult) String() string

type GetBucketLocation 

type GetBucketLocation struct {
	XMLName            xml.Name `xml:"LocationConstraint"`
	Xmlns              string   `xml:"xmlns,attr"`
	LocationConstraint string   `xml:",chardata"`
}

type GoFakeS3 

type GoFakeS3 struct {
	// contains filtered or unexported fields
}

GoFakeS3 implements HTTP handlers for processing S3 requests and returning S3 responses.

Logic is delegated to other components, like Backend or uploader.

func New 

func New(backend Backend, options ...Option) *GoFakeS3

New creates a new GoFakeS3 using the supplied Backend. Backends are pluggable. Several Backend implementations ship with GoFakeS3, which can be found in the gofakes3/backends package.

func (*GoFakeS3) AddAuthKeys 

func (g *GoFakeS3) AddAuthKeys(p map[string]string)

func (*GoFakeS3) DelAuthKeys 

func (g *GoFakeS3) DelAuthKeys(p []string)

func (*GoFakeS3) Server 

func (g *GoFakeS3) Server() http.Handler

Create the AWS S3 API

type InitiateMultipartUpload 

type InitiateMultipartUpload struct {
	Bucket   string   `xml:"Bucket"`
	Key      string   `xml:"Key"`
	UploadID UploadID `xml:"UploadId"`
}

type InternalErrorCode 

type InternalErrorCode string

InternalErrorCode represents an GoFakeS3 error code. It maps to ErrInternal when constructing a response. 

const (
	ErrInternalPageNotImplemented InternalErrorCode = "PaginationNotImplemented"
)

INTERNAL errors! These are not part of the S3 interface, they are codes we have declared ourselves. Should all map to a 500 status code:

func (InternalErrorCode) Error 

func (e InternalErrorCode) Error() string

func (InternalErrorCode) ErrorCode 

func (e InternalErrorCode) ErrorCode() ErrorCode

type ListBucketPage 

type ListBucketPage struct {
	// Specifies the key in the bucket that represents the last item in
	// the previous page. The first key in the returned page will be the
	// next lexicographically (UTF-8 binary) sorted key after Marker.
	// If HasMarker is true, this must be non-empty.
	Marker    string
	HasMarker bool

	// Sets the maximum number of keys returned in the response body. The
	// response might contain fewer keys, but will never contain more. If
	// additional keys satisfy the search criteria, but were not returned
	// because max-keys was exceeded, the response contains
	// <isTruncated>true</isTruncated>. To return the additional keys, see
	// key-marker and version-id-marker.
	//
	// MaxKeys MUST be > 0, otherwise it is ignored.
	MaxKeys int64
}

func (ListBucketPage) IsEmpty 

func (p ListBucketPage) IsEmpty() bool

type ListBucketResult 

type ListBucketResult struct {
	ListBucketResultBase

	// Indicates where in the bucket listing begins. Marker is included in the
	// response if it was sent with the request.
	Marker string `xml:"Marker"`

	// When the response is truncated (that is, the IsTruncated element value
	// in the response is true), you can use the key name in this field as a
	// marker in the subsequent request to get next set of objects. Amazon S3
	// lists objects in UTF-8 character encoding in lexicographical order.
	//
	// NOTE: This element is returned only if you specify a delimiter request
	// parameter. If the response does not include the NextMarker and it is
	// truncated, you can use the value of the last Key in the response as the
	// marker in the subsequent request to get the next set of object keys.
	NextMarker string `xml:"NextMarker,omitempty"`
}

type ListBucketResultBase 

type ListBucketResultBase struct {
	XMLName xml.Name `xml:"ListBucketResult"`
	Xmlns   string   `xml:"xmlns,attr"`

	// Name of the bucket.
	Name string `xml:"Name"`

	// Specifies whether (true) or not (false) all of the results were
	// returned. If the number of results exceeds that specified by MaxKeys,
	// all of the results might not be returned.
	IsTruncated bool `xml:"IsTruncated"`

	// Causes keys that contain the same string between the prefix and the
	// first occurrence of the delimiter to be rolled up into a single result
	// element in the CommonPrefixes collection. These rolled-up keys are not
	// returned elsewhere in the response.
	//
	// NOTE: Each rolled-up result in CommonPrefixes counts as only one return
	// against the MaxKeys value. (BW: been waiting to find some confirmation of
	// that for a while!)
	Delimiter string `xml:"Delimiter,omitempty"`

	Prefix string `xml:"Prefix"`

	MaxKeys int64 `xml:"MaxKeys,omitempty"`

	CommonPrefixes []CommonPrefix `xml:"CommonPrefixes,omitempty"`
	Contents       []*Content     `xml:"Contents"`
}

type ListBucketResultV2 

type ListBucketResultV2 struct {
	ListBucketResultBase

	// If ContinuationToken was sent with the request, it is included in the
	// response.
	ContinuationToken string `xml:"ContinuationToken,omitempty"`

	// Returns the number of keys included in the response. The value is always
	// less than or equal to the MaxKeys value.
	KeyCount int64 `xml:"KeyCount,omitempty"`

	// If the response is truncated, Amazon S3 returns this parameter with a
	// continuation token. You can specify the token as the continuation-token
	// in your next request to retrieve the next set of keys.
	NextContinuationToken string `xml:"NextContinuationToken,omitempty"`

	// If StartAfter was sent with the request, it is included in the response.
	StartAfter   string `xml:"StartAfter,omitempty"`
	EncodingType string `xml:"EncodingType,omitempty"`
}

type ListBucketVersionsPage 

type ListBucketVersionsPage struct {
	// Specifies the key in the bucket that you want to start listing from.
	// If HasKeyMarker is true, this must be non-empty.
	KeyMarker    string
	HasKeyMarker bool

	// Specifies the object version you want to start listing from. If
	// HasVersionIDMarker is true, this must be non-empty.
	VersionIDMarker    VersionID
	HasVersionIDMarker bool

	// Sets the maximum number of keys returned in the response body. The
	// response might contain fewer keys, but will never contain more. If
	// additional keys satisfy the search criteria, but were not returned
	// because max-keys was exceeded, the response contains
	// <isTruncated>true</isTruncated>. To return the additional keys, see
	// key-marker and version-id-marker.
	//
	// MaxKeys MUST be > 0, otherwise it is ignored.
	MaxKeys int64
}

type ListBucketVersionsResult 

type ListBucketVersionsResult struct {
	XMLName        xml.Name       `xml:"ListBucketVersionsResult"`
	Xmlns          string         `xml:"xmlns,attr"`
	Name           string         `xml:"Name"`
	Delimiter      string         `xml:"Delimiter,omitempty"`
	Prefix         string         `xml:"Prefix,omitempty"`
	CommonPrefixes []CommonPrefix `xml:"CommonPrefixes,omitempty"`
	IsTruncated    bool           `xml:"IsTruncated"`
	MaxKeys        int64          `xml:"MaxKeys"`

	// Marks the last Key returned in a truncated response.
	KeyMarker string `xml:"KeyMarker,omitempty"`

	// When the number of responses exceeds the value of MaxKeys, NextKeyMarker
	// specifies the first key not returned that satisfies the search criteria.
	// Use this value for the key-marker request parameter in a subsequent
	// request.
	NextKeyMarker string `xml:"NextKeyMarker,omitempty"`

	// Marks the last version of the Key returned in a truncated response.
	VersionIDMarker VersionID `xml:"VersionIdMarker,omitempty"`

	// When the number of responses exceeds the value of MaxKeys,
	// NextVersionIdMarker specifies the first object version not returned that
	// satisfies the search criteria. Use this value for the version-id-marker
	// request parameter in a subsequent request.
	NextVersionIDMarker VersionID `xml:"NextVersionIdMarker,omitempty"`

	// AWS responds with a list of either <Version> or <DeleteMarker> objects. The order
	// needs to be preserved and they need to be direct of ListBucketVersionsResult:
	//	<ListBucketVersionsResult>
	//		<DeleteMarker ... />
	//		<Version ... />
	//		<DeleteMarker ... />
	//		<Version ... />
	//	</ListBucketVersionsResult>
	Versions []VersionItem
	// contains filtered or unexported fields
}

func NewListBucketVersionsResult 

func NewListBucketVersionsResult(
	bucketName string,
	prefix *Prefix,
	page *ListBucketVersionsPage,
) *ListBucketVersionsResult

func (*ListBucketVersionsResult) AddPrefix 

func (b *ListBucketVersionsResult) AddPrefix(prefix string)

type ListMultipartUploadItem 

type ListMultipartUploadItem struct {
	Key          string       `xml:"Key"`
	UploadID     UploadID     `xml:"UploadId"`
	Initiator    *UserInfo    `xml:"Initiator,omitempty"`
	Owner        *UserInfo    `xml:"Owner,omitempty"`
	StorageClass StorageClass `xml:"StorageClass,omitempty"`
	Initiated    ContentTime  `xml:"Initiated,omitempty"`
}

type ListMultipartUploadPartItem 

type ListMultipartUploadPartItem struct {
	PartNumber   int         `xml:"PartNumber"`
	LastModified ContentTime `xml:"LastModified,omitempty"`
	ETag         string      `xml:"ETag,omitempty"`
	Size         int64       `xml:"Size"`
}

type ListMultipartUploadPartsResult 

type ListMultipartUploadPartsResult struct {
	XMLName xml.Name `xml:"ListPartsResult"`

	Bucket               string       `xml:"Bucket"`
	Key                  string       `xml:"Key"`
	UploadID             UploadID     `xml:"UploadId"`
	StorageClass         StorageClass `xml:"StorageClass,omitempty"`
	Initiator            *UserInfo    `xml:"Initiator,omitempty"`
	Owner                *UserInfo    `xml:"Owner,omitempty"`
	PartNumberMarker     int          `xml:"PartNumberMarker"`
	NextPartNumberMarker int          `xml:"NextPartNumberMarker"`
	MaxParts             int64        `xml:"MaxParts"`
	IsTruncated          bool         `xml:"IsTruncated,omitempty"`

	Parts []ListMultipartUploadPartItem `xml:"Part"`
}

type ListMultipartUploadsResult 

type ListMultipartUploadsResult struct {
	Bucket string `xml:"Bucket"`

	// Together with upload-id-marker, this parameter specifies the multipart upload
	// after which listing should begin.
	KeyMarker string `xml:"KeyMarker,omitempty"`

	// Together with key-marker, specifies the multipart upload after which listing
	// should begin. If key-marker is not specified, the upload-id-marker parameter
	// is ignored.
	UploadIDMarker UploadID `xml:"UploadIdMarker,omitempty"`

	NextKeyMarker      string   `xml:"NextKeyMarker,omitempty"`
	NextUploadIDMarker UploadID `xml:"NextUploadIdMarker,omitempty"`

	// Sets the maximum number of multipart uploads, from 1 to 1,000, to return
	// in the response body. 1,000 is the maximum number of uploads that can be
	// returned in a response.
	MaxUploads int64 `xml:"MaxUploads,omitempty"`

	Delimiter string `xml:"Delimiter,omitempty"`

	// Lists in-progress uploads only for those keys that begin with the specified
	// prefix.
	Prefix string `xml:"Prefix,omitempty"`

	CommonPrefixes []CommonPrefix `xml:"CommonPrefixes,omitempty"`
	IsTruncated    bool           `xml:"IsTruncated,omitempty"`

	Uploads []ListMultipartUploadItem `xml:"Upload"`
}

type LogLevel 

type LogLevel string
const (
	LogErr  LogLevel = "ERR"
	LogWarn LogLevel = "WARN"
	LogInfo LogLevel = "INFO"
)

type Logger 

type Logger interface {
	Print(level LogLevel, v ...interface{})
}

Logger provides a very minimal target for logging implementations to hit to allow arbitrary logging dependencies to be used with GoFakeS3.

Only an interface to the standard library's log package is provided with GoFakeS3, other libraries will require an adapter. Adapters are trivial to write.

For zap: 

type LogrusLog struct {
	log *zap.Logger
}

func (l LogrusLog) Print(level LogLevel, v ...interface{}) {
	switch level {
	case gofakes3.LogErr:
		l.log.Error(fmt.Sprint(v...))
	case gofakes3.LogWarn:
		l.log.Warn(fmt.Sprint(v...))
	case gofakes3.LogInfo:
		l.log.Info(fmt.Sprint(v...))
	default:
		panic("unknown level")
	}
}

For logrus: 

type LogrusLog struct {
	log *logrus.Logger
}

func (l LogrusLog) Print(level LogLevel, v ...interface{}) {
	switch level {
	case gofakes3.LogErr:
		l.log.Errorln(v...)
	case gofakes3.LogWarn:
		l.log.Warnln(v...)
	case gofakes3.LogInfo:
		l.log.Infoln(v...)
	default:
		panic("unknown level")
	}
}

func DiscardLog 

func DiscardLog() Logger

DiscardLog creates a Logger that discards all messages.

func GlobalLog 

func GlobalLog(levels ...LogLevel) Logger

GlobalLog creates a Logger that uses the global log.Println() function.

All levels are reported by default. If you pass levels to this function, it will act as a level whitelist.

func MultiLog 

func MultiLog(loggers ...Logger) Logger

func StdLog 

func StdLog(log *log.Logger, levels ...LogLevel) Logger

StdLog creates a Logger that uses the stdlib's log.Logger type.

All levels are reported by default. If you pass levels to this function, it will act as a level whitelist.

type MFADeleteStatus 

type MFADeleteStatus string

MFADeleteStatus is used by VersioningConfiguration. 

const (
	MFADeleteNone     MFADeleteStatus = ""
	MFADeleteEnabled  MFADeleteStatus = "Enabled"
	MFADeleteDisabled MFADeleteStatus = "Disabled"
)

func (MFADeleteStatus) Enabled 

func (v MFADeleteStatus) Enabled() bool

func (*MFADeleteStatus) UnmarshalXML 

func (v *MFADeleteStatus) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error

type MultiDeleteResult 

type MultiDeleteResult struct {
	XMLName xml.Name      `xml:"DeleteResult"`
	Deleted []ObjectID    `xml:"Deleted"`
	Error   []ErrorResult `xml:",omitempty"`
}

MultiDeleteResult contains the response from a multi delete operation.

func (MultiDeleteResult) AsError 

func (d MultiDeleteResult) AsError() error

type Object 

type Object struct {
	Name     string
	Metadata map[string]string
	Size     int64
	Contents io.ReadCloser
	Hash     []byte
	Range    *ObjectRange

	// VersionID will be empty if bucket versioning has not been enabled.
	VersionID VersionID

	// If versioning is enabled for the bucket, this is true if this object version
	// is a delete marker.
	IsDeleteMarker bool
}

Object contains the data retrieved from a backend for the specified bucket and object key.

You MUST always call Contents.Close() otherwise you may leak resources.

type ObjectDeleteResult 

type ObjectDeleteResult struct {
	// Specifies whether the versioned object that was permanently deleted was
	// (true) or was not (false) a delete marker. In a simple DELETE, this
	// header indicates whether (true) or not (false) a delete marker was
	// created.
	IsDeleteMarker bool

	// Returns the version ID of the delete marker created as a result of the
	// DELETE operation. If you delete a specific object version, the value
	// returned by this header is the version ID of the object version deleted.
	VersionID VersionID
}

type ObjectID 

type ObjectID struct {
	Key string `xml:"Key"`

	// Versions not supported in GoFakeS3 yet.
	VersionID string `xml:"VersionId,omitempty" json:"VersionId,omitempty"`
}

type ObjectList 

type ObjectList struct {
	CommonPrefixes []CommonPrefix
	Contents       []*Content
	IsTruncated    bool
	NextMarker     string
	// contains filtered or unexported fields
}

func NewObjectList 

func NewObjectList() *ObjectList

func (*ObjectList) Add 

func (b *ObjectList) Add(item *Content)

func (*ObjectList) AddPrefix 

func (b *ObjectList) AddPrefix(prefix string)

type ObjectRange 

type ObjectRange struct {
	Start, Length int64
}

type ObjectRangeRequest 

type ObjectRangeRequest struct {
	Start, End int64
	FromEnd    bool
}

func (*ObjectRangeRequest) Range 

func (o *ObjectRangeRequest) Range(size int64) (*ObjectRange, error)

type Option 

type Option func(g *GoFakeS3)

func WithAutoBucket 

func WithAutoBucket(enabled bool) Option

WithAutoBucket instructs GoFakeS3 to create buckets that don't exist on first use, rather than returning ErrNoSuchBucket.

func WithGlobalLog 

func WithGlobalLog() Option

WithGlobalLog configures gofakes3 to use GlobalLog() for logging, which uses the standard library's log.Println() call to log messages.

func WithHostBucket 

func WithHostBucket(enabled bool) Option

WithHostBucket enables or disables bucket rewriting in the router. If active, the URL 'http://mybucket.localhost/object' will be routed as if the URL path was '/mybucket/object'.

See https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html for details.

func WithIntegrityCheck 

func WithIntegrityCheck(check bool) Option

WithIntegrityCheck enables or disables Content-MD5 validation when putting an Object.

func WithLogger 

func WithLogger(logger Logger) Option

WithLogger allows you to supply a logger to GoFakeS3 for debugging/tracing. logger may be nil.

func WithMetadataSizeLimit 

func WithMetadataSizeLimit(size int) Option

WithMetadataSizeLimit allows you to reconfigure the maximum allowed metadata size.

See DefaultMetadataSizeLimit for the starting value, set to '0' to disable.

func WithRequestID 

func WithRequestID(id uint64) Option

WithRequestID sets the starting ID used to generate the "x-amz-request-id" header.

func WithTimeSkewLimit 

func WithTimeSkewLimit(skew time.Duration) Option

WithTimeSkewLimit allows you to reconfigure the allowed skew between the client's clock and the server's clock. The AWS client SDKs will send the "x-amz-date" header containing the time at the client, which is used to calculate the skew.

See DefaultSkewLimit for the starting value, set to '0' to disable.

func WithTimeSource 

func WithTimeSource(timeSource TimeSource) Option

WithTimeSource allows you to substitute the behaviour of time.Now() and time.Since() within GoFakeS3. This can be used to trigger time skew errors, or to ensure the output of the commands is deterministic.

See gofakes3.FixedTimeSource(), gofakes3.LocalTimeSource(tz).

func WithUnimplementedPageError 

func WithUnimplementedPageError() Option

WithUnimplementedPageError allows you to enable or disable the error that occurs if the Backend does not implement paging.

By default, GoFakeS3 will simply retry a request for a page of objects without the page if the Backend does not implement pagination. This can be used to enable an error in that condition instead.

func WithV4Auth 

func WithV4Auth(authPair map[string]string) Option

func WithoutVersioning 

func WithoutVersioning() Option

WithoutVersioning disables versioning on the passed backend, if it supported it.

type Prefix 

type Prefix struct {
	HasPrefix bool
	Prefix    string

	HasDelimiter bool
	Delimiter    string
}

func NewFolderPrefix 

func NewFolderPrefix(prefix string) (p Prefix)

func NewPrefix 

func NewPrefix(prefix, delim *string) (p Prefix)

func (Prefix) FilePrefix 

func (p Prefix) FilePrefix() (path, remaining string, ok bool)

FilePrefix returns the path portion, then the remaining portion of the Prefix if the Delimiter is "/". If the Delimiter is not set, or not "/", ok will be false.

For example: 

/foo/bar/  : path: /foo/bar  remaining: ""
/foo/bar/b : path: /foo/bar  remaining: "b"
/foo/bar   : path: /foo      remaining: "bar"

func (Prefix) Match 

func (p Prefix) Match(key string, match *PrefixMatch) (ok bool)

PrefixMatch checks whether key starts with prefix. If the prefix does not match, nil is returned.

It is a best-effort attempt to implement the prefix/delimiter matching found in S3.

To check whether the key belongs in Contents or CommonPrefixes, compare the result to key.

func (Prefix) String 

func (p Prefix) String() string

type PrefixMatch 

type PrefixMatch struct {
	// Input key passed to PrefixMatch.
	Key string

	// CommonPrefix indicates whether this key should be returned in the bucket
	// contents or the common prefixes part of the "list bucket" response.
	CommonPrefix bool

	// The longest matched part of the key.
	MatchedPart string
}

func (*PrefixMatch) AsCommonPrefix 

func (match *PrefixMatch) AsCommonPrefix() CommonPrefix

type PutObjectResult 

type PutObjectResult struct {
	// If versioning is enabled on the bucket, this should be set to the
	// created version ID. If versioning is not enabled, this should be
	// empty.
	VersionID VersionID
}

type Storage 

type Storage struct {
	XMLName xml.Name  `xml:"ListAllMyBucketsResult"`
	Xmlns   string    `xml:"xmlns,attr"`
	Owner   *UserInfo `xml:"Owner,omitempty"`
	Buckets Buckets   `xml:"Buckets>Bucket"`
}

type StorageClass 

type StorageClass string
const (
	StorageStandard StorageClass = "STANDARD"
)

func (StorageClass) MarshalXML 

func (s StorageClass) MarshalXML(e *xml.Encoder, start xml.StartElement) error

type TimeSource 

type TimeSource interface {
	Now() time.Time
	Since(time.Time) time.Duration
}

func DefaultTimeSource 

func DefaultTimeSource() TimeSource

type TimeSourceAdvancer 

type TimeSourceAdvancer interface {
	TimeSource
	Advance(by time.Duration)
}

func FixedTimeSource 

func FixedTimeSource(at time.Time) TimeSourceAdvancer

FixedTimeSource provides a source of time that always returns the specified time.

type UploadID 

type UploadID string

UploadID uses a string as the underlying type, but the string should only represent a decimal integer. See uploader.uploadID for details.

type UploadListMarker 

type UploadListMarker struct {
	// Represents the key-marker query parameter. Together with 'uploadID',
	// this parameter specifies the multipart upload after which listing should
	// begin.
	//
	// If 'uploadID' is not specified, only the keys lexicographically greater
	// than the specified key-marker will be included in the list.
	//
	// If 'uploadID' is specified, any multipart uploads for a key equal to
	// 'object'  might also be included, provided those multipart uploads have
	// upload IDs lexicographically greater than the specified uploadID.
	Object string

	// Represents the upload-id-marker query parameter to the
	// ListMultipartUploads operation. Together with 'object', specifies the
	// multipart upload after which listing should begin. If 'object' is not
	// specified, the 'uploadID' parameter is ignored.
	UploadID UploadID
}

UploadListMarker is used to seek to the start of a page in a ListMultipartUploads operation.

type UserInfo 

type UserInfo struct {
	ID          string `xml:"ID"`
	DisplayName string `xml:"DisplayName"`
}

type Version 

type Version struct {
	XMLName      xml.Name    `xml:"Version"`
	Key          string      `xml:"Key"`
	VersionID    VersionID   `xml:"VersionId"`
	IsLatest     bool        `xml:"IsLatest"`
	LastModified ContentTime `xml:"LastModified,omitempty"`
	Size         int64       `xml:"Size"`

	// According to the S3 docs, this is always STANDARD for a Version:
	StorageClass StorageClass `xml:"StorageClass"`

	ETag  string    `xml:"ETag"`
	Owner *UserInfo `xml:"Owner,omitempty"`
}

func (Version) GetVersionID 

func (v Version) GetVersionID() VersionID

type VersionID 

type VersionID string

type VersionItem 

type VersionItem interface {
	GetVersionID() VersionID
	// contains filtered or unexported methods
}

type VersionedBackend 

type VersionedBackend interface {
	// VersioningConfiguration must return a gofakes3.ErrNoSuchBucket error if the bucket
	// does not exist. See gofakes3.BucketNotFound() for a convenient way to create one.
	//
	// If the bucket has never had versioning enabled, VersioningConfiguration MUST return
	// empty strings (S300001).
	VersioningConfiguration(bucket string) (VersioningConfiguration, error)

	// SetVersioningConfiguration must return a gofakes3.ErrNoSuchBucket error if the bucket
	// does not exist. See gofakes3.BucketNotFound() for a convenient way to create one.
	SetVersioningConfiguration(bucket string, v VersioningConfiguration) error

	// GetObject must return a gofakes3.ErrNoSuchKey error if the object does
	// not exist. See gofakes3.KeyNotFound() for a convenient way to create
	// one.
	//
	// If the returned Object is not nil, you MUST call Object.Contents.Close(),
	// otherwise you will leak resources. Implementers should return a no-op
	// implementation of io.ReadCloser.
	//
	// GetObject must return gofakes3.ErrNoSuchVersion if the version does not
	// exist.
	//
	// If versioning has been enabled on a bucket, but subsequently suspended,
	// GetObjectVersion should still return the object version (S300001).
	//
	// FIXME: s3assumer test; what happens when versionID is empty? Does it
	// return the latest?
	GetObjectVersion(
		bucketName, objectName string,
		versionID VersionID,
		rangeRequest *ObjectRangeRequest) (*Object, error)

	// HeadObjectVersion fetches the Object version from the backend, but the Contents will be
	// a no-op ReadCloser.
	//
	// If the returned Object is not nil, you MUST call Object.Contents.Close(),
	// otherwise you will leak resources. Implementers should return a no-op
	// implementation of io.ReadCloser.
	//
	// HeadObjectVersion should return a NotFound() error if the object does not
	// exist.
	HeadObjectVersion(bucketName, objectName string, versionID VersionID) (*Object, error)

	// DeleteObjectVersion permanently deletes a specific object version.
	//
	// DeleteObjectVersion must return a gofakes3.ErrNoSuchBucket error if the bucket
	// does not exist. See gofakes3.BucketNotFound() for a convenient way to create one.
	//
	// If the bucket exists and either the object does not exist (S300003) or
	// the version does not exist (S300002), you MUST return an empty
	// ObjectDeleteResult and a nil error.
	DeleteObjectVersion(bucketName, objectName string, versionID VersionID) (ObjectDeleteResult, error)

	// Backend implementers can assume the ListBucketVersionsPage is valid:
	// KeyMarker and VersionIDMarker will either both be set, or both be unset. No
	// other combination will be present (S300004).
	//
	// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETVersion.html
	//
	// This MUST return the list of current versions with an empty VersionID
	// even if versioning has never been enabled for the bucket (S300005).
	//
	// The Backend MUST treat a nil prefix identically to a zero prefix, and a
	// nil page identically to a zero page.
	ListBucketVersions(bucketName string, prefix *Prefix, page *ListBucketVersionsPage) (*ListBucketVersionsResult, error)
}

VersionedBackend may be optionally implemented by a Backend in order to support operations on S3 object versions.

If you don't implement VersionedBackend, requests to GoFakeS3 that attempt to make use of versions will return ErrNotImplemented if GoFakesS3 is unable to find another way to satisfy the request.

type VersioningConfiguration 

type VersioningConfiguration struct {
	XMLName xml.Name `xml:"VersioningConfiguration"`

	Status VersioningStatus `xml:"Status,omitempty"`

	// When enabled, the bucket owner must include the x-amz-mfa request header
	// in requests to change the versioning state of a bucket and to
	// permanently delete a versioned object.
	MFADelete MFADeleteStatus `xml:"MfaDelete,omitempty"`
}

func (*VersioningConfiguration) Enabled 

func (v *VersioningConfiguration) Enabled() bool

func (*VersioningConfiguration) SetEnabled 

func (v *VersioningConfiguration) SetEnabled(enabled bool)

type VersioningStatus 

type VersioningStatus string
const (
	VersioningNone      VersioningStatus = ""
	VersioningEnabled   VersioningStatus = "Enabled"
	VersioningSuspended VersioningStatus = "Suspended"
)

func (*VersioningStatus) UnmarshalXML 

func (v *VersioningStatus) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error

Source Files

View all Source files

Directories

Path Synopsis
internal
goskipiter
s3assumer
s3io
Package signature verify request for any AWS service.
 Package signature verify request for any AWS service.
Package xml implements a simple XML 1.0 parser that understands XML name spaces.
 Package xml implements a simple XML 1.0 parser that understands XML name spaces.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.