fs

README

The package go-fs provides some common utilities which GoLang developers use when working with files, either system files or web files.

Installation

The only requirement is the Go Programming Language.

$ go get -u gopkg.in/kataras/go-fs.v0

Docs

Local file system helpers

// DirectoryExists returns true if a directory(or file) exists, otherwise false
DirectoryExists(dir string) bool

// GetHomePath returns the user's $HOME directory
GetHomePath() string

// GetParentDir returns the parent directory of the passed targetDirectory
GetParentDir(targetDirectory string) string

// RemoveFile removes a file or a directory
RemoveFile(filePath string) error

// RenameDir renames (moves) oldpath to newpath.
// If newpath already exists, Rename replaces it.
// OS-specific restrictions may apply when oldpath and newpath are in different directories.
// If there is an error, it will be of type *LinkError.
//
// It's a copy of os.Rename
RenameDir(oldPath string, newPath string) error

// CopyFile  accepts full path of the source and full path of destination, if file exists it's overrides it
// this function doesn't checks for permissions and all that, it returns an error
CopyFile(source string, destination string) error

// CopyDir recursively copies a directory tree, attempting to preserve permissions.
// Source directory must exist.
CopyDir(source string, dest string)  error

// Unzip extracts a zipped file to the target location
// returns the path of the created folder (if any) and an error (if any)
Unzip(archive string, target string) (string, error)

// TypeByExtension returns the MIME type associated with the file extension ext.
// The extension ext should begin with a leading dot, as in ".html".
// When ext has no associated type, TypeByExtension returns "".
//
// Extensions are looked up first case-sensitively, then case-insensitively.
//
// The built-in table is small but on unix it is augmented by the local
// system's mime.types file(s) if available under one or more of these
// names:
//
//   /etc/mime.types
//   /etc/apache2/mime.types
//   /etc/apache/mime.types
//
// On Windows, MIME types are extracted from the registry.
//
// Text types have the charset parameter set to "utf-8" by default.
TypeByExtension(fullfilename string) string

Net/http handlers

// FaviconHandler receives the favicon path and serves the favicon
FaviconHandler(favPath string) http.Handler

// StaticContentHandler returns the net/http.Handler interface to handle raw binary data,
// normally the data parameter was read by custom file reader or by variable
StaticContentHandler(data []byte, contentType string) http.Handler

// StaticFileHandler serves a static file such as css,js, favicons, static images
// it stores the file contents to the memory, doesn't supports seek because we read all-in-one the file, but seek is supported by net/http.ServeContent
StaticFileHandler(filename string) http.Handler

// SendStaticFileHandler sends a file for force-download to the client
// it stores the file contents to the memory, doesn't supports seek because we read all-in-one the file, but seek is supported by net/http.ServeContent
SendStaticFileHandler(filename string) http.Handler

// DirHandler serves a directory as web resource
// accepts a system Directory (string),
// a string which will be stripped off if not empty and
// Note 1: this is a dynamic dir handler, means that if a new file is added to the folder it will be served
// Note 2: it doesn't cache the system files, use it with your own risk, otherwise you can use the http.FileServer method, which is different of what I'm trying to do here.
// example:
// staticHandler := http.FileServer(http.Dir("static"))
// http.Handle("/static/", http.StripPrefix("/static/", staticHandler))
// converted to ->
// http.Handle("/static/", fs.DirHandler("./static", "/static/"))
DirHandler(dir string, strippedPrefix string) http.Handler

Read the http_test.go for more.

Gzip Writer

Writes gzip compressed content to an underline io.Writer. It uses sync.Pool to reduce memory allocations.

Better performance through klauspost/compress package which provides us a gzip.Writer which is faster than Go standard's gzip package's writer.

// NewGzipPool returns a new gzip writer pool, ready to use
NewGzipPool(Level int) *GzipPool

// DefaultGzipPool returns a new writer pool with Compressor's level setted to DefaultCompression
DefaultGzipPool() *GzipPool

// AcquireGzipWriter prepares a gzip writer and returns it
//
// see ReleaseGzipWriter
AcquireGzipWriter(w io.Writer) *gzip.Writer

// ReleaseGzipWriter called when flush/close and put the gzip writer back to the pool
//
// see AcquireGzipWriter
ReleaseGzipWriter(gzipWriter *gzip.Writer)

// WriteGzip writes a compressed form of p to the underlying io.Writer. The
// compressed bytes are not necessarily flushed until the Writer is closed
WriteGzip(w io.Writer, b []byte) (int, error)

• AcquireGzipWriter get a gzip writer, create new if no free writer available from inside the pool (sync.Pool).

• ReleaseGzipWriter releases puts a gzip writer to the pool (sync.Pool).

• WriteGzip gets a gzip writer, writes a compressed form of p to the underlying io.Writer. The compressed bytes are not necessarily flushed until the Writer is closed. Finally it Releases the particular gzip writer.

  if these called from package level then the default gzip writer's pool is used to get/put and write

• NewGzipPool receives a compression level and returns a new gzip writer pool

• DefaultGzipPool returns a new gzip writer pool with DefaultCompression as the Compressor's Level

New & Default are optional, use them to create more than one sync.Pool, if you expect thousands of writers working together

Using default pool's writer to compress & write content

import "gopkg.in/kataras/go-fs.v0"

var writer io.Writer

// ... using default package's Pool to get a gzip writer
n, err := fs.WriteGzip(writer, []byte("Compressed data and content here"))

Using default Pool to get a gzip writer, compress & write content and finally release manually the gzip writer to the default Pool

import "gopkg.in/kataras/go-fs.v0"

var writer io.Writer

// ... using default writer's pool to get a gzip.Writer

mygzipWriter := fs.AcquireGzipWriter(writer) // get a gzip.Writer from the default gzipwriter Pool

n, err := mygzipWriter.WriteGzip([]byte("Compressed data and content here"))

gzipwriter.ReleaseGzipWriter(mygzipWriter) // release this gzip.Writer to the default gzipwriter package's gzip writer Pool (sync.Pool)

Create and use a totally new gzip writer Pool

import "gopkg.in/kataras/go-fs.v0"

var writer io.Writer
var gzipWriterPool = fs.NewGzipPool(fs.DefaultCompression)

// ...
n, err := gzipWriterPool.WriteGzip(writer, []byte("Compressed data and content here"))

Get a gzip writer Pool with the default options(compressor's Level)

import "gopkg.in/kataras/go-fs.v0"

var writer io.Writer
var gzipWriterPool = fs.DefaultGzipPool() // returns a new default gzip writer pool

// ...
n, err := gzipWriterPool.WriteGzip(writer, []byte("Compressed data and content here"))

Acquire, Write and Release from a new(.NewGzipPool/.DefaultGzipPool) gzip writer Pool

import "gopkg.in/kataras/go-fs.v0"

var writer io.Writer

var gzipWriterPool = fs.DefaultGzipPool() // returns a new default gzip writer pool

mygzipWriter := gzipWriterPool.AcquireGzipWriter(writer) // get a gzip.Writer from the new gzipWriterPool

n, err := mygzipWriter.WriteGzip([]byte("Compressed data and content here"))

gzipWriterPool.ReleaseGzipWriter(mygzipWriter) // release this gzip.Writer to the gzipWriterPool (sync.Pool)

Working with remote zip files

// DownloadZip downloads a zip file returns the downloaded filename and an error.
DownloadZip(zipURL string, newDir string, showOutputIndication bool) (string, error)

// Install is just the flow of: downloadZip -> unzip -> removeFile(zippedFile)
// accepts 3 parameters
//
// first parameter is the remote url file zip
// second parameter is the target directory
// third paremeter is a boolean which you can set to true to print out the progress
// returns a string(installedDirectory) and an error
//
// (string) installedDirectory is the directory which the zip file had, this is the real installation path
// the installedDirectory is not empty when the installation is succed, the targetDirectory is not already exists and no error happens
// the installedDirectory is empty when the installation is already done by previous time or an error happens
Install(remoteFileZip string, targetDirectory string, showOutputIndication bool) (string, error)

Install = DownloadZip -> Unzip to the destination folder, remove the downloaded .zip, copy the inside extracted folder to the destination

Install many remote files(URI) to a single destination folder via installer instance

type Installer struct {
	// InstallDir is the directory which all zipped downloads will be extracted
	// defaults to $HOME path
	InstallDir string
	// Indicator when it's true it shows an indicator about the installation process
	// defaults to false
	Indicator bool
	// RemoteFiles is the list of the files which should be downloaded when Install() called
	RemoteFiles []string
}

// Add adds a remote file(*.zip) to the list for download
Add(...string)

// Install installs all RemoteFiles, when this function called then the RemoteFiles are being resseted
// returns all installed paths and an error (if any)
// it continues on errors and returns them when the operation completed
Install() ([]string, error)

Usage

package main

import "gopkg.in/kataras/go-fs.v0"

var testInstalledDir = fs.GetHomePath() + fs.PathSeparator + "mydir" + fs.PathSeparator

// remote file zip | expected output(installed) directory
var filesToInstall = map[string]string{
	"https://github.com/kataras/q/archive/master.zip":             testInstalledDir + "q-master",
	"https://github.com/kataras/iris/archive/master.zip":          testInstalledDir + "iris-master",
	"https://github.com/kataras/go-errors/archive/master.zip":     testInstalledDir + "go-errors-master",
	"https://github.com/kataras/go-gzipwriter/archive/master.zip": testInstalledDir + "go-gzipwriter-master",
	"https://github.com/kataras/go-events/archive/master.zip":     testInstalledDir + "go-events-master",
}

func main() {
	myInstaller := fs.NewInstaller(testInstalledDir)

	for remoteURI := range filesToInstall {
		myInstaller.Add(remoteURI)
	}

	installedDirs, err := myInstaller.Install()

	if err != nil {
		panic(err)
	}

	for _, installedDir := range installedDirs {
    println("New folder created: " + installedDir)
	}

}

When you want to install different zip files to different destination directories.

Usage

package main

import "gopkg.in/kataras/go-fs.v0"

var testInstalledDir = fs.GetHomePath() + fs.PathSeparator + "mydir" + fs.PathSeparator

// remote file zip | expected output(installed) directory
var filesToInstall = map[string]string{
	"https://github.com/kataras/q/archive/master.zip":             testInstalledDir + "q-master",
	"https://github.com/kataras/iris/archive/master.zip":          testInstalledDir + "iris-master",
	"https://github.com/kataras/go-errors/archive/master.zip":     testInstalledDir + "go-errors-master",
	"https://github.com/kataras/go-gzipwriter/archive/master.zip": testInstalledDir + "go-gzipwriter-master",
	"https://github.com/kataras/go-events/archive/master.zip":     testInstalledDir + "go-events-master",
}

func main(){
	for remoteURI, expectedInstalledDir := range filesToInstall {

		installedDir, err := fs.Install(remoteURI, testInstalledDir, false)

		if err != nil {
			panic(err)
		}
		println("Installed: "+installedDir)
	}
}

Read the installer_test.go for more.

You do not need any other special explanations for this package, just navigate to the godoc or the source code.

FAQ

Explore these questions or navigate to the community chat.

Versioning

Current: v0.0.5

People

The author of go-fs is @kataras.

If you're willing to donate, feel free to send any amount through paypal

Contributing

If you are interested in contributing to the go-fs project, please make a PR.

License

This project is licensed under the MIT License.

License can be found here.

Documentation

Overview

Package fs provides some common utilities which GoLang developers use when working with files, either system files or web files

Index

• Constants
• Variables
• func AcquireGzipWriter(w io.Writer) *gzip.Writer
• func CopyDir(source string, dest string) (err error)
• func CopyFile(source string, destination string) error
• func DirHandler(dir string, strippedPrefix string) http.Handler
• func DirectoryExists(dir string) bool
• func DownloadZip(zipURL string, newDir string, showOutputIndication bool) (string, error)
• func FaviconHandler(favPath string) http.Handler
• func GetHomePath() string
• func GetParentDir(targetDirectory string) string
• func Install(remoteFileZip string, targetDirectory string, showOutputIndication bool) (installedDirectory string, err error)
• func ReleaseGzipWriter(gzipWriter *gzip.Writer)
• func RemoveFile(filePath string) error
• func RenameDir(oldPath string, newPath string) error
• func SendStaticFileHandler(filename string) http.Handler
• func ShowIndicator(wr io.Writer, newLine bool) chan bool
• func StaticContentHandler(data []byte, contentType string) http.Handler
• func StaticFileHandler(filename string) http.Handler
• func TypeByExtension(fullfilename string) (t string)
• func Unzip(archive string, target string) (string, error)
• func WriteGzip(w io.Writer, b []byte) (int, error)
• type GzipPool
  • func DefaultGzipPool() *GzipPool
  • func NewGzipPool(Level int) *GzipPool
  • func (p *GzipPool) AcquireGzipWriter(w io.Writer) *gzip.Writer
  • func (p *GzipPool) ReleaseGzipWriter(gzipWriter *gzip.Writer)
  • func (p *GzipPool) WriteGzip(w io.Writer, b []byte) (int, error)
• type Installer
  • func NewInstaller(installDir string, remoteFilesZip ...string) *Installer
  • func (i *Installer) Add(remoteFilesZip ...string)
  • func (i *Installer) Install() ([]string, error)
• type OptionSet
  • func Silent(val bool) OptionSet
  • func Stderr(val io.Writer) OptionSet
  • func Stdin(val io.Reader) OptionSet
  • func Stdout(val io.Writer) OptionSet
  • func (o OptionSet) Set(main *Options)
• type Options
  • func (o *Options) Set(main *Options)
• type Updater
  • func GetUpdater(owner string, repo string, currentReleaseVersion string) (*Updater, error)
  • func (u *Updater) HasUpdate() (bool, string)
  • func (u *Updater) Run(setters ...optionSetter) bool

Constants

const (
	NoCompression       = 0
	BestSpeed           = 1
	BestCompression     = 9
	DefaultCompression  = -1
	ConstantCompression = -2 // Does only Huffman encoding
)

These constants are copied from the standard flate package available Compressors

const (
	// Version current version number
	Version = "0.0.5"
)

Variables

var (
	// TimeFormat default time format for any kind of datetime parsing
	TimeFormat = "Mon, 02 Jan 2006 15:04:05 GMT"
	// StaticCacheDuration expiration duration for INACTIVE file handlers
	StaticCacheDuration = 20 * time.Second
	// Charset the charset will be used to the Content-Type response header, if not given previously
	Charset = "utf-8"
)

var (
	// DefaultUpdaterAlreadyInstalledMessage "\nThe latest version '%s' was already installed."
	DefaultUpdaterAlreadyInstalledMessage = "\nThe latest version '%s' was already installed."
)

var DefaultUpdaterYesInput = [...]string{"y", "yes", "nai", "si"}

DefaultUpdaterYesInput the string or character which user should type to proceed the update, if !silent

var PathSeparator = string(os.PathSeparator)

PathSeparator is the OS-specific path separator

Functions

func AcquireGzipWriter

func AcquireGzipWriter(w io.Writer) *gzip.Writer

AcquireGzipWriter prepares a gzip writer and returns it

see ReleaseGzipWriter

func CopyDir

func CopyDir(source string, dest string) (err error)

CopyDir recursively copies a directory tree, attempting to preserve permissions. Source directory must exist.

func CopyFile

func CopyFile(source string, destination string) error

CopyFile accepts full path of the source and full path of destination, if file exists it's overrides it this function doesn't checks for permissions and all that, it returns an error

func DirHandler

func DirHandler(dir string, strippedPrefix string) http.Handler

DirHandler serves a directory as web resource accepts a system Directory (string), a string which will be stripped off if not empty and Note 1: this is a dynamic dir handler, means that if a new file is added to the folder it will be served Note 2: it doesn't cache the system files, use it with your own risk, otherwise you can use the http.FileServer method, which is different of what I'm trying to do here. example: staticHandler := http.FileServer(http.Dir("static")) http.Handle("/static/", http.StripPrefix("/static/", staticHandler)) converted to -> http.Handle("/static/", fs.DirHandler("./static", "/static/"))

func DirectoryExists

func DirectoryExists(dir string) bool

DirectoryExists returns true if a directory(or file) exists, otherwise false

func DownloadZip

func DownloadZip(zipURL string, newDir string, showOutputIndication bool) (string, error)

DownloadZip downloads a zip file returns the downloaded filename and an error.

func FaviconHandler

func FaviconHandler(favPath string) http.Handler

FaviconHandler receives the favicon path and serves the favicon

func GetHomePath

func GetHomePath() string

GetHomePath returns the user's $HOME directory

func GetParentDir

func GetParentDir(targetDirectory string) string

GetParentDir returns the parent directory(string) of the passed targetDirectory (string)

func Install

func Install(remoteFileZip string, targetDirectory string, showOutputIndication bool) (installedDirectory string, err error)

Install is just the flow of: downloadZip -> unzip -> removeFile(zippedFile) accepts 3 parameters

first parameter is the remote url file zip second parameter is the target directory third paremeter is a boolean which you can set to true to print out the progress returns a string(installedDirectory) and an error

(string) installedDirectory is the directory which the zip file had, this is the real installation path the installedDirectory is not empty when the installation is succed, the targetDirectory is not already exists and no error happens the installedDirectory is empty when the installation is already done by previous time or an error happens

func ReleaseGzipWriter

func ReleaseGzipWriter(gzipWriter *gzip.Writer)

ReleaseGzipWriter called when flush/close and put the gzip writer back to the pool

see AcquireGzipWriter

func RemoveFile

func RemoveFile(filePath string) error

RemoveFile removes a file or directory and returns an error, if any

func RenameDir

func RenameDir(oldPath string, newPath string) error

RenameDir renames (moves) oldpath to newpath. If newpath already exists, Rename replaces it. OS-specific restrictions may apply when oldpath and newpath are in different directories. If there is an error, it will be of type *LinkError.

It's a copy of os.Rename

func SendStaticFileHandler

func SendStaticFileHandler(filename string) http.Handler

SendStaticFileHandler sends a file for force-download to the client it stores the file contents to the memory, doesn't supports seek because we read all-in-one the file, but seek is supported by net/http.ServeContent

func ShowIndicator

func ShowIndicator(wr io.Writer, newLine bool) chan bool

ShowIndicator shows a silly terminal indicator for a process, close of the finish channel is done here.

func StaticContentHandler

func StaticContentHandler(data []byte, contentType string) http.Handler

StaticContentHandler returns the net/http.Handler interface to handle raw binary data, normally the data parameter was read by custom file reader or by variable

func StaticFileHandler

func StaticFileHandler(filename string) http.Handler

StaticFileHandler serves a static file such as css,js, favicons, static images it stores the file contents to the memory, doesn't supports seek because we read all-in-one the file, but seek is supported by net/http.ServeContent

func TypeByExtension

func TypeByExtension(fullfilename string) (t string)

TypeByExtension returns the MIME type associated with the file extension ext. The extension ext should begin with a leading dot, as in ".html". When ext has no associated type, TypeByExtension returns "".

Extensions are looked up first case-sensitively, then case-insensitively.

The built-in table is small but on unix it is augmented by the local system's mime.types file(s) if available under one or more of these names:

/etc/mime.types
/etc/apache2/mime.types
/etc/apache/mime.types

On Windows, MIME types are extracted from the registry.

Text types have the charset parameter set to "utf-8" by default.

func Unzip

func Unzip(archive string, target string) (string, error)

Unzip extracts a zipped file to the target location returns the path of the created folder (if any) and an error (if any)

func WriteGzip

func WriteGzip(w io.Writer, b []byte) (int, error)

WriteGzip writes a compressed form of p to the underlying io.Writer. The compressed bytes are not necessarily flushed until the Writer is closed

Types

type GzipPool

type GzipPool struct {
	sync.Pool
	Level int
}

GzipPool is a wrapper of sync.Pool, to initialize a new gzip writer pool, just create a new instance of this iteral, GzipPool{}

func DefaultGzipPool

func DefaultGzipPool() *GzipPool

DefaultGzipPool returns a new writer pool with Compressor's level setted to DefaultCompression

func NewGzipPool

func NewGzipPool(Level int) *GzipPool

NewGzipPool returns a new gzip writer pool, ready to use

func (*GzipPool) AcquireGzipWriter

func (p *GzipPool) AcquireGzipWriter(w io.Writer) *gzip.Writer

AcquireGzipWriter prepares a gzip writer and returns it

see ReleaseGzipWriter

func (*GzipPool) ReleaseGzipWriter

func (p *GzipPool) ReleaseGzipWriter(gzipWriter *gzip.Writer)

ReleaseGzipWriter called when flush/close and put the gzip writer back to the pool

see AcquireGzipWriter

func (*GzipPool) WriteGzip

func (p *GzipPool) WriteGzip(w io.Writer, b []byte) (int, error)

WriteGzip writes a compressed form of p to the underlying io.Writer. The compressed bytes are not necessarily flushed until the Writer is closed

type Installer

type Installer struct {
	// InstallDir is the directory which all zipped downloads will be extracted
	// defaults to $HOME path
	InstallDir string
	// Indicator when it's true it shows an indicator about the installation process
	// defaults to false
	Indicator bool
	// RemoteFiles is the list of the files which should be downloaded when Install() called
	RemoteFiles []string
	// contains filtered or unexported fields
}

Installer is useful when you have single output-target directory and multiple zip files to install

func NewInstaller

func NewInstaller(installDir string, remoteFilesZip ...string) *Installer

NewInstaller returns an new Installer, it's just a builder to add remote files once and call Install to install all of them first parameter is the installed directory, if empty then it uses the user's $HOME path second parameter accepts optional remote zip files to be install

func (*Installer) Add

func (i *Installer) Add(remoteFilesZip ...string)

Add adds a remote file(*.zip) to the list for download

func (*Installer) Install

func (i *Installer) Install() ([]string, error)

Install installs all RemoteFiles, when this function called then the RemoteFiles are being resseted returns all installed paths and an error (if any) it continues on errors and returns them when the operation completed

type OptionSet

type OptionSet func(*Options)

OptionSet sets an option

func Silent

func Silent(val bool) OptionSet

Silent sets the Silent option to the 'val'

func Stderr

func Stderr(val io.Writer) OptionSet

Stderr specify the process's standard output and error.

If Stdout and Stderr are the same writer, at most one goroutine at a time will call Write.

func Stdin

func Stdin(val io.Reader) OptionSet

Stdin specifies the process's standard input. If Stdin is nil, the process reads from the null device (os.DevNull). If Stdin is an *os.File, the process's standard input is connected directly to that file. Otherwise, during the execution of the command a separate goroutine reads from Stdin and delivers that data to the command over a pipe. In this case, Wait does not complete until the goroutine stops copying, either because it has reached the end of Stdin (EOF or a read error) or because writing to the pipe returned an error.

func Stdout

func Stdout(val io.Writer) OptionSet

Stdout specify the process's standard output and error.

If either is nil, Run connects the corresponding file descriptor to the null device (os.DevNull).

func (OptionSet) Set

func (o OptionSet) Set(main *Options)

Set implements the optionSetter

type Options

type Options struct {
	Silent bool
	// Stdin specifies the process's standard input.
	// If Stdin is nil, the process reads from the null device (os.DevNull).
	// If Stdin is an *os.File, the process's standard input is connected
	// directly to that file.
	// Otherwise, during the execution of the command a separate
	// goroutine reads from Stdin and delivers that data to the command
	// over a pipe. In this case, Wait does not complete until the goroutine
	// stops copying, either because it has reached the end of Stdin
	// (EOF or a read error) or because writing to the pipe returned an error.
	Stdin io.Reader

	// Stdout and Stderr specify the process's standard output and error.
	//
	// If either is nil, Run connects the corresponding file descriptor
	// to the null device (os.DevNull).
	//
	// If Stdout and Stderr are the same writer, at most one
	// goroutine at a time will call Write.
	Stdout io.Writer
	Stderr io.Writer
}

Options the available options used iside the updater.Run func

func (*Options) Set

func (o *Options) Set(main *Options)

Set implements the optionSetter

type Updater

type Updater struct {
	// contains filtered or unexported fields
}

Updater is the base struct for the Updater feature

func GetUpdater

func GetUpdater(owner string, repo string, currentReleaseVersion string) (*Updater, error)

GetUpdater returns a new Updater based on a github repository and the latest local release version(string, example: "4.2.3" or "v4.2.3-rc1")

func (*Updater) HasUpdate

func (u *Updater) HasUpdate() (bool, string)

HasUpdate returns true if a new update is available the second output parameter is the latest ,full, version

func (*Updater) Run

func (u *Updater) Run(setters ...optionSetter) bool

Run runs the update, returns true if update has been found and installed, otherwise false