upload

README

Upload for HTTP servers

Enables you to upload files, such as build artifacts, to your HTTP server instance.

Licensed under a BSD-style license.

Highlights

• uses HTTP PUT and POST for uploads
• supports HTTP COPY, MOVE, and DELETE
• imposes limits on filenames:
  • rejects those that are not conforming to Unicode NFC or NFD
  • rejects any comprised of unexpected alphabets ϟ(ツ)╯
• limits to file- and transaction sizes independent from any transport encoding

Versions

Version	Change
/v4	Call to http.Handler is henceforth the preferred way to use this.
/v5	No longer limited to local filesystems as backend, the syntax for to (WriteToPath) has changed.

Warnings

This plugin reveals some errors thrown by your filesystem implementation to the uploader, for example about insufficient space on the target device..

The way Golang currently decodes MIME Multipart (which is used with POST requests) results in any files you are uploading being held in memory for the duration of the upload.

Configuration Syntax

Let this be a legible shorthand for instances of Handler:

upload <path> {
	to                    "<directory>"

	enable_webdav
	filenames_form        <none|NFC|NFD>
	filenames_in          <u0000-uff00> [<u0000-uff00>| …]
	random_suffix_len     0..N
	promise_download_from <path>

	max_filesize          0..N
	max_transaction_size  0..N
}

These settings are required:

• path is the Scope you cofigured the handler for, such as Go's ServeMux. It will be stripped and won't be part of any resulting file and directories.
• to is an existing target directory. Must be a quoted absolute path. When using Linux it is recommended to place this on a filesystem which supports O_TMPFILE and extents, such as (but not limited to) ext4 or XFS.
  Absent any other silently assumes scheme file://.

It is not advised—signed upload URLs are more efficient— you can upload directly to cloud storage buckets. Use a scheme supported by the Go CDK:

import (
  upload "blitznote.com/src/http.upload/v5"
  _ "gocloud.dev/blob/gcsblob" // Registers scheme "gs://"
  _ "gocloud.dev/blob/s3blob"  // Registers scheme "s3://"
)

// …

to := "s3://my-bucket?region=us-west-1"
to = "gs://my-bucket"
to = "/var/tmp"
h, _ := upload.NewHandler("/", to, nil)

These are optional:

• enable_webdav: Enables other methods than POST and PUT, especially MOVE and DELETE. Is a flag and has no parameters.
  (disable_webdav will no longer be recognized because it's the new default.)

• filenames_form: if given, filenames and directories that are not conforming to Unicode NFC or NFD will be rejected.
  Set this to one of either values when you get errors indicating that your filesystem does not convert names properly. (If in doubt, go with NFC; on Mac PCs with NFD.)
  The default is to not enforce anything.

• filenames_in allows you to limit filenames to specified Unicode ranges. The ranges' bounds must be given in hexadecimal, and start with letter u.
  Use this setting to prevent users from uploading files in, for example, Cyrillic when expect Latin and/or Chinese alphabets only.

• random_suffix_len, if > 0, will result in all filenames getting a randomized suffix.
  The suffix will start in a _ (underscore letter) and placed before any extension.
  For example, image.png will be written as image_a107xm.png with configuration value 6. Utilize promise_download_from to get the resulting filename.
  The default is 0 for off.

• promise_download_from is a string that represents an URI reference, such as a path.
  It will be used to indicate where the uploaded file can be downloaded, by responding with HTTP header Location (multiple times if need be) for all received files.
  You will most probably want to set this to the upload path.
  The default value is "", which means no HTTP header Location will be sent.

• By max_filesize you can limit the size of individual files. Unless set to 0, which means "unlimited" and is the default value, it's in bytes.

• max_transaction_size is similar, but applies to uploads of one or more file in one request. For example, when using MIME Multipart uploads.
  The behaviour with max_filesize > max_transaction_size is currently undefined; set max_transaction_size to a multiple of max_filesize.

Some transfer encodings, such as base64, know comments. Those, or super-long headers and the such, can be exploited to transfer many more bytes than for example max_transaction_size would otherwise allow. Mitigate this by utilizing a different plugin, http.limits, which counts incoming bytes ignorant of any encoding. Set a limit of about 1.4× to 2.05× the max_transaction_size.
This plugin writes files blockwise for a better performance. Limits are rounded up by a few kilobytes to the next full block.

Tutorial

Setup a minimal configuration like this, or go run example.go & after copying it into a separate directory and removing the line with +build ignore (mind the port number, which is 9000 there):

uploadHandler, _ := upload.NewHandler("/web/path", "/var/tmp", nil)
uploadHandler.EnableWebdav = true

… and upload one file:

# HTTP PUT
curl \
  -T /etc/os-release \
  https://127.0.0.1/web/path/from-release

… or more files in one go (sub-directories will be created as needed):

# HTTP POST
curl \
  -F gitconfig=@.gitconfig \
  -F id_ed25519.pub=@.ssh/id_ed25519.pub \
  https://127.0.0.1/web/path/

… which you then can move and delete like this:

# MOVE is 'mv'
curl -X MOVE \
  -H "Destination: /web/path/to-release" \
  https://127.0.0.1/web/path/from-release

# DELETE is 'rm -r'
curl -X DELETE \
  https://127.0.0.1/web/path/to-release

Configuration Examples

A host used by someone in Central and West Europe would be configured like this to accept filenames in Latin with some Greek runes and a few mathematical symbols:

h, _ := upload.NewHandler(
  "/college/curriculum",
  "/home/ellen_baker/inbox",
  nil)
h.UnicodeForm = &struct{ Use norm.Form }{Use: norm.NFC}
h.RestrictFilenamesTo = upload.ParseUnicodeBlockList(
  "u0000–u007F u0100–u017F u0391–u03C9 u2018–u203D u2152–u217F",
)

// upload /college/curriculum {
//	to "/home/ellen_baker/inbox"
//	filenames_form NFC
//	filenames_in u0000–u007F u0100–u017F u0391–u03C9 u2018–u203D u2152–u217F
// }

A host for Linux distribution packages can be more restrictive:

h, _ := upload.NewHandler(
  "/binhost/gentoo",
  "/var/portage/packages",
  nil)
h.RestrictFilenamesTo = upload.ParseUnicodeBlockList(
  "u0000–u007F", // ASCII
)

// upload /binhost/gentoo {
//	to "/var/portage/packages"
//	filenames_in u0000–u007F
// }

… while someone in East Asia would share space on his blog like this:

to, _ := blob.OpenBucket(context.Background(), "gs://blog.senpai.asia")
alphabet, _ := upload.ParseUnicodeBlockList(
  "u0000–u007F u0100–u017F u0391–u03C9 u2018–u203D u3000–u303f u3040–u309f u30a0–u30ff u4e00–9faf uff00–uffef"
)
h, _ := upload.Handler{
  Scope:               "/wp-uploads",
  Bucket:              to,
  EnableWebdav:        true,
  MaxFilesize:         16<<20,
  RestrictFilenamesTo: alphabet,
}

// upload /wp-uploads {
//	to "/var/www/senpai/wp-uploads"
//	enable_webdav
//	max_filesize 16777216
//	filenames_in u0000–u007F u0100–u017F u0391–u03C9 u2018–u203D u3000–u303f u3040–u309f u30a0–u30ff u4e00–9faf uff00–uffef
// }

See Also

You can find a very nice overview of Unicode Character ranges here:
http://jrgraphix.net/research/unicode_blocks.php

Here is the official list of Unicode blocks:
http://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt

Documentation

Overview

Package upload contains a HTTP handler that provides facilities for uploading files.

Example

var (
	scope     = "/" // prefix to http.Request.URL.Path
	directory = "/var/tmp"
	next      = http.FileServer(http.Dir(directory))
)

uploadHandler, _ := NewHandler(scope, directory, next)
http.Handle(scope, uploadHandler)
// http.ListenAndServe(":9000", nil)

Index

• Constants
• func InAlphabet(s string, alphabet []*unicode.RangeTable, enforceForm *norm.Form) bool
• func ParseUnicodeBlockList(str string) (*unicode.RangeTable, error)
• type Handler
  • func NewHandler(scope string, targetDirectory string, next http.Handler) (*Handler, error)
  • func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

Examples

• Package

Constants

const (
	// AlwaysRejectedRunes contains which that are not safe to use with network shares.
	// If a file name contains any, it will be rejected.
	AlwaysRejectedRunes = `"*:<>?|\`
)

Functions

func InAlphabet

func InAlphabet(s string, alphabet []*unicode.RangeTable, enforceForm *norm.Form) bool

InAlphabet is true for strings exclusively in the given alphabet and form.

Runes representing whitespace – other than U+0020 (space) and U+2009 (spatium) – as well as any non-printable will always be rejected.

Use this to filter file names.

func ParseUnicodeBlockList

func ParseUnicodeBlockList(str string) (*unicode.RangeTable, error)

ParseUnicodeBlockList naïvely translates a string with space-delimited Unicode ranges to Go's unicode.RangeTable.

All elements must fit into uint32. A Range must begin with its lower bound, and ranges must not overlap.

The format of one range is as follows, with 'stride' being set to '1' if left empty.

<low>-<high>[:<stride>]

Types

type Handler

type Handler struct {
	MaxFilesize        int64
	MaxTransactionSize int64

	// The upload destination.
	Bucket *blob.Bucket

	// Uploaded files can be gotten back from here.
	// If ≠ "" this will trigger sending headers such as "Location".
	ApparentLocation string

	// Enables MOVE, DELETE, and similar. Without this only POST and PUT will be recognized.
	EnableWebdav bool

	// Set this to reject any non-conforming filenames.
	UnicodeForm *struct{ Use norm.Form }

	// Limit the acceptable alphabet(s) for filenames by setting this value.
	RestrictFilenamesTo []*unicode.RangeTable

	// Append '_' and a randomized suffix of that length.
	RandomizedSuffixLength uint32

	// For methods that are not recognized.
	Next http.Handler
	// The path, to be stripped from the full URL and the target path swapped in.
	Scope string
}

Handler will deal with anything that manipulates files, but won't deliver a listing or serve them.

func NewHandler

func NewHandler(scope string, targetDirectory string, next http.Handler) (*Handler, error)

NewHandler creates a new instance of this plugin's upload handler, meant to be used in Go's own http server.

'scope' is the prefix of the upload destination's URL.Path, like `/dir/to/upload/destination`.

'next' is optional and can be nil.

func (Handler) ServeHTTP

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP catches methods meant for file manipulation. Anything else will be delegated to h.Next, if not nil.