caddys3proxy

package module
v0.5.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2021 License: Apache-2.0 Imports: 26 Imported by: 0

README

golangci-lint Actions Status Test Actions Status All Contributors

caddy-s3-proxy

caddy-s3-proxy allows you to proxy requests directly from S3.

S3 does have the website option, in which case, a normal reverse proxy could be used to display S3 data. However, it is sometimes inconvient to do that. This module lets you access S3 data even if website access is not configured on your bucket.

Making a version of caddy with this plugin

With caddy 2 you can use xcaddy to build a version of caddy with this plugin installed. To install xcaddy do:

go get -u github.com/caddyserver/xcaddy/cmd/xcaddy

This repo has a Makefile to make it easier to build a new version of caddy with this plugin. Just type:

make build

You can run make docker do build a local image you can test with.

Configuration

The Caddyfile directive would look something like this:

	s3proxy [<matcher>] {
		bucket <bucket_name>
		region <region_name>
                profile <aws profile>
		index  <list of index file names>
		endpoint <alternative S3 endpoint>
		root   <key prefix>
		enable_put
		enable_delete
 		force_path_style
		errors <http status> <S3 key to a custom error page for this http status>
		errors <S3 key to a default error page>
		browse [<path to template>]
	}
option type required default help
bucket string yes S3 bucket name
region string yes-ish env AWS_REGION S3 region - if not give in the Caddyfile then AWS_REGION env var must be set.
profile string no empty string AWS profile if using shared credentials files.
endpoint string no aws default S3 hostname
index string[] no [index.html, index.txt] Index files to look up for dir path
root string no Set a "prefix" to be added to key
enable_put bool no false Allow PUT method to be sent through proxy
enable_delete bool no false Allow DELETE method to be sent through proxy
force_path_style bool no false Set this to true to force S3 request to use path-style addressing
use_accelerate bool no false Set this to true to enable S3 Accelerate feature
errors [int, ] string no Custom error page or use "pass_through" to write nothing for errors.
browse [string] no Turns on a directory view for partial keys, an optional path to a template can be given

Credentials

This module uses the default providor chain to get credentials for access to S3. This provides several more secure options to provide credentials for accessing S3 without putting the credentials in the Caddyfile. The methods include (and are looked for in this order):

  1. Environment variables. I.e. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

  2. Shared credentials file. (Located at ~/.aws/credentials) (You may pass the optional profile directive to select specific credentials.)

  3. If your application uses an ECS task definition or RunTask API operation, IAM role for tasks.

  4. If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2.

For much more detail on the various options for setting AWS credentials see here: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Works with localstack!

The s3 proxy works great with localstack for local testing. Just set the endpoint directive to your localstack instance. You will also want to set the force_path_style directive as well since localstack currently does not support virtual style addressing. In fact, all of our examples use localstack - check out the examples directory.

Handling errors

When accessing S3 you may get errors like keyNotFound, bucket does not exist, or ACL permissions problems. By default this proxy will map those errors to an http error - like 404, 403 or 500.

However, with the errors directive you have a couple of more options. You can specify a S3 key that may contain HTML to display rather than just returning an error code. This can be done for a specific error or all errors. For example,

errors 403 /key/path/to/permissionerr.html
errors /key/path/to/defaulterr.html

This will display the page permissionerr.html for any 403 errors and defaulterr.html for all other errors.

There is a special option to "pass through" on an error and let the next Caddy handler deal with the request. For example,

errors 404 pass_through
errors /key/path/to/defaulterr.html

Will pass 404 errors onto the next handler. All other errors will show the page defaulterr.html.

Note: The errors direction only applies to GET method requests. PUT and DELETE errors just return the code.

Examples you can play with

In the examples directory is an example of using the s3proxy with localstack. Localstack contains a working version of S3 you can use for local development.

Check out the examples here. You can also just run make example to build a docker image with the plugin and launch the compose example.

Contributors

A big thank you to folks who have contributed to this project!


rayjlinden
Gilbert Gilb's
Christoph Kluge

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Item 

type Item struct {
	Name         string `json:"name"`
	IsDir        bool   `json:"is_dir"`
	Key          string `json:"key"`
	Url          string `json:"url"`
	Size         string `json:"size"`
	LastModified string `json:"last_modified"`
}

type PageObj 

type PageObj struct {
	Count    int64  `json:"count"`
	Items    []Item `json:"items"`
	MoreLink string `json:"more"`
}

func (PageObj) GenerateHtml 

func (po PageObj) GenerateHtml(w http.ResponseWriter, template *template.Template) error

GenerateHtml generates html output for the PageObj

func (PageObj) GenerateJson 

func (po PageObj) GenerateJson(w http.ResponseWriter) error

GenerateJson generates JSON output for the PageObj

type S3Proxy 

type S3Proxy struct {
	// The path to the root of the site. Default is `{http.vars.root}` if set,
	// Or if not set the value is "" - meaning use the whole path as a key.
	Root string `json:"root,omitempty"`

	// The AWS region the bucket is hosted in
	Region string `json:"region,omitempty"`

	// The AWS profile to use if mulitple profiles are specified in creds
	Profile string `json:"profile,omitempty"`

	// The name of the S3 bucket
	Bucket string `json:"bucket,omitempty"`

	// Use non-standard endpoint for S3
	Endpoint string `json:"endpoint,omitempty"`

	// The names of files to try as index files if a folder is requested.
	IndexNames []string `json:"index_names,omitempty"`

	// A glob pattern used to hide matching key paths (returning a 404)
	Hide []string

	// Flag to determine if PUT operations are allowed (default false)
	EnablePut bool

	// Flag to determine if DELETE operations are allowed (default false)
	EnableDelete bool

	// Flag to enable browsing of "directories" in S3 (paths that end with a /)
	EnableBrowse bool

	// Path to a template file to use for generating browse dir html page
	BrowseTemplate string

	// Mapping of HTTP error status to S3 keys or pass through option.
	ErrorPages map[int]string `json:"error_pages,omitempty"`

	// S3 key to a default error page or pass through option.
	DefaultErrorPage string `json:"default_error_page,omitempty"`

	// Set this to `true` to force the request to use path-style addressing.
	S3ForcePathStyle bool `json:"force_path_style,omitempty"`

	// Set this to `true` to enable S3 Accelerate feature.
	S3UseAccelerate bool `json:"use_accelerate,omitempty"`
	// contains filtered or unexported fields
}

S3Proxy implements a proxy to return, set, delete or browse objects from S3

func (S3Proxy) BrowseHandler 

func (p S3Proxy) BrowseHandler(w http.ResponseWriter, r *http.Request, key string) error

func (S3Proxy) CaddyModule 

func (S3Proxy) CaddyModule() caddy.ModuleInfo

CaddyModule returns the Caddy module information.

func (S3Proxy) ConstructListObjInput 

func (p S3Proxy) ConstructListObjInput(r *http.Request, key string) s3.ListObjectsV2Input

func (S3Proxy) DeleteHandler 

func (p S3Proxy) DeleteHandler(w http.ResponseWriter, r *http.Request, key string) error

func (S3Proxy) GetHandler 

func (p S3Proxy) GetHandler(w http.ResponseWriter, r *http.Request, fullPath string) error

func (S3Proxy) MakePageObj 

func (p S3Proxy) MakePageObj(result *s3.ListObjectsV2Output) PageObj

func (*S3Proxy) Provision 

func (p *S3Proxy) Provision(ctx caddy.Context) (err error)

func (S3Proxy) PutHandler 

func (p S3Proxy) PutHandler(w http.ResponseWriter, r *http.Request, key string) error

func (S3Proxy) ServeHTTP 

func (p S3Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error

ServeHTTP implements the main entry point for a request for the caddyhttp.Handler interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.