README
¶
swiftlygo
Golang library for creating Static and Dynamic Large Objects in OpenStack Swift
Table of Contents
Background
OpenStack Object Storage has the ability to create two powerful kinds of aggregate file: Static and Dynamic Large Objects.
A Static Large Object (SLO) is a manifest file that lists other files within an object storage instance in a particular order. When you read the contents of an SLO, you get the contents of every file listed within its manifest in the order that they are defined within the manifest. This allows you to build large files out of small pieces.
OpenStack Object Storage does not allow single files to be larger than 5GB, and a single SLO manifest file can
only reference 1000 distinct objects. Fortunately, SLO manifest files can reference
other SLO manifest files. Swiftlygo always creates at least two manifest files (one top-level manifest
that references up to 1000 other manifests, and at least one manifest referencing chunks). If you have more
than 1000 file chunks, it can create up to 999 additional manifests to allow a maximum theoretical
object size of 5PB (1,000,000 chunks of 5GB each). If you have fewer than 1000 chunks in your file, it
may be faster to reference the uploaded file that ends with -manifest-0000 rather than the top-level manifest, as
the sub-manifest has the same contents.
A Dynamic Large Object (DLO) is equally useful, but in different circumstances. Rather than specify an ordered list of files to be treated as the DLO's contents, a DLO specifies a filename prefix within a particular Object Storage container. All files matching the prefix are treated as the contents of that DLO (sorted lexographically). Since any files that match the prefix are considered part of the DLO, you can insert data into DLOs after you create them by adding a file with the correct prefix at the correct sort order position.
Sadly, these two incredible features of the OpenStack Object Store are difficult to access and use. We wrote this library to make creating these Large Object files easier.
Install
To download the package, run
go get github.com/ibmjstart/swiftlygo
To use it within your code, add
import "github.com/ibmjstart/swiftlygo"
Usage
swiftlygo has two main sets of functionality: creating SLOs and DLOs. The API for each is slightly different, since each requires different information.
Both APIs rely on the auth.Destination interface defined in the auth subpackage.
The sections below give an overview of each API. For more thorough docs, use
godoc github.com/ibmjstart/swiftlygo
godoc github.com/ibmjstart/swiftlygo/auth
godoc github.com/ibmjstart/swiftlygo/slo
SLOs
The API for creating SLOs is based around uploading a single large file. That file will be broken into chunks, and each chunk will be uploaded to Object Storage as a separate file. After all of the chunks have been uploaded, the SLO manifest file will be uploaded. If this process is completed successfully, You will be able to reference the entire file by the name of the manifest.
If your upload is interrupted, you can ensure that the boolean onlyMissing parameter to slo.NewUploader is set to true, which will skip all uploads for which the files are already present within the targeted object storage container. This can save a lot of time if you were most of the way through a previous upload.
Here's a simple example of using the SLO API to upload a file.
package example
import (
"fmt"
"github.com/ibmjstart/swiftlygo/auth"
"github.com/ibmjstart/swiftlygo"
"os"
)
func main() {
//using object storage credentials from VCAP.json or similar
destination, err := auth.Authenticate("username", "apikey", "authurl", "domain", "tenant")
if err != nil {
// connection failed, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
uploadFile, err := os.Open("file/path")
if err != nil {
// reading file failed, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
uploader, err := swiftlygo.NewSloUploader(destination,
10000000,//file chunk size in bytes (set to something larger for multi-gigabyte files and something smaller for files < 10MB)
"container name",
"object name",//name that you want to reference the whole SLO by
uploadFile,
8,//maxiumum number of parallel uploads allowed
true,//upload only the file chunks that have not already been uploaded
os.Stdout)
if err != nil {
// there was an error preparing the upload, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
err = uploader.Upload()
if err != nil {
// there was an error uploading your SLO, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
DLOs
DLOs are slightly different from SLOs in that they allow their segments to be uploaded independently from the manifest file. DLO manifests have an attribute that defines a container and prefix. Any files in this container with the specified prefix will become segments of the DLO, regardless of whether they were uploaded before, after or at the same time as the manifest. All files meeting these criteria will be downloaded as one file (composed in lexicographical order) when the DLO is downloaded.
Here's an example of using the DLO API to create a manifest.
package example
import (
"fmt"
"github.com/ibmjstart/swiftlygo"
"github.com/ibmjstart/swiftlygo/auth"
"os"
)
func main() {
//using object storage credentials from VCAP.json or similar
destination, err := auth.Authenticate("username", "apikey", "authurl", "domain", "tenant")
if err != nil {
// connection failed, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
uploader := swiftlygo.NewDloUploader(destination,
"dlo container name", //name of the container the manifest will be created in
"manifest name",
"object container", //name of the container the DLO's segments will be in
"prefix-") //prefix for files that are segments of this DLO
if err != nil {
// there was an error preparing the upload, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
err = uploader.Upload()
if err != nil {
// there was an error uploading your DLO, handle appropriately
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Contribute
PRs accepted.
Small note: If editing the README, please conform to the standard-readme specification.
License
Apache 2.0 © IBM jStart
Documentation
¶
Overview ¶
Package swiftlygo makes working with Large Objects in OpenStack Object Storage easier.
The auth subpackage provides a convenient abstraction for talking to an Object Storage instance. See the documentation of the auth package for details on how to authenticate with Object Storage Instances. The pipeline subpackage implements a low-level API for Static Large Object creation if the SloUploader doesn't offer the level of control that your application requires.
The root swiftlygo package provides functionality for easily creating Dynamic Large Objects and Static Large Objects.
Both the SloUploader and DloUploader types are easy to use. They only have one method, Upload(), that performs a synchronous upload (it will only return after the upload is complete). The SloUploader also exposes a Status struct that can be used during an upload to query the progress up the upload.
Index ¶
- type DloUploader
- type SloUploader
- type Status
- func (s *Status) NumberUploaded() uint
- func (s *Status) PercentComplete() float64
- func (s *Status) Print()
- func (s *Status) Rate() float64
- func (s *Status) RateMBPS() float64
- func (s *Status) Start()
- func (s *Status) Stop()
- func (s *Status) String() string
- func (s *Status) TimeRemaining() time.Duration
- func (s *Status) TotalUploads() uint
- func (s *Status) UploadComplete()
- func (s *Status) UploadSize() uint
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DloUploader ¶
type DloUploader struct {
// contains filtered or unexported fields
}
DloUploader prepares and executes the upload of a Dynamic Large Object manifest.
func NewDloUploader ¶
func NewDloUploader(connection auth.Destination, dloContainer, dloName, objectContainer, prefix string) *DloUploader
NewDloUploader returns an uploader that will create a new DLO. The dloContainer determines where the DLO manifest file is stored, whereas the objectContainer determines which container the DLO will look in for files beginning with the given prefix. This allows you to store the DLO in one container while referencing files within another.
func (*DloUploader) Upload ¶
func (d *DloUploader) Upload() error
Upload actually performs the upload that creates the DLO.
type SloUploader ¶
type SloUploader struct {
Status *Status
// contains filtered or unexported fields
}
SloUploader uploads a file to object storage
func NewSloUploader ¶
func NewSloUploader(connection auth.Destination, chunkSize uint, container string, object string, source *os.File, maxUploads uint, onlyMissing bool, outputFile io.Writer) (*SloUploader, error)
NewUploader prepares an upload for an SLO by constructing a data pipeline that will read the provided file, split it into pieces of chunkSize bytes, and upload it into the provided destination in the provided container with the given object name.
func (*SloUploader) Upload ¶
func (u *SloUploader) Upload() error
Upload uploads the sloUploader's source file to object storage
type Status ¶
type Status struct {
// contains filtered or unexported fields
}
func NewStatus ¶
NewStatus creates a new Status with the number of individual uploads and the size of each upload.
func (*Status) NumberUploaded ¶
NumberUploaded returns how many file chunks have been uploaded.
func (*Status) PercentComplete ¶
PercentComplete returns much of the upload is complete.
func (*Status) Print ¶
func (s *Status) Print()
Print sends the current status of the upload to the output channel.
func (*Status) Start ¶
func (s *Status) Start()
Start begins timing the upload. Only call this once.
func (*Status) Stop ¶
func (s *Status) Stop()
Stop finalizes the duration of the upload. Only call this once.
func (*Status) TimeRemaining ¶
TimeRemaining estimates the amount of time remaining in the upload.
func (*Status) TotalUploads ¶
TotalUploads returns how many file chunks need to be uploaded total.
func (*Status) UploadComplete ¶
func (s *Status) UploadComplete()
UploadComplete marks that one chunk has been uploaded. Call this each time an upload succeeds.
func (*Status) UploadSize ¶
UploadSize returns the size of each file chunk (with the exception of the last file chunk, which can be any size less than this).
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package auth provides easy methods to authenticate with Object Storage The main fixture of the auth package is the Destination interface.
|
Package auth provides easy methods to authenticate with Object Storage The main fixture of the auth package is the Destination interface. |
|
mock
Package mock provides fake object storage endpoints for testing The structs defined here all implement the github.com/ibmjstart/swiftlygo/auth.Destination interface and are therefore useful for testing any code that uploads data via a destination.
|
Package mock provides fake object storage endpoints for testing The structs defined here all implement the github.com/ibmjstart/swiftlygo/auth.Destination interface and are therefore useful for testing any code that uploads data via a destination. |
|
Package pipeline implements a low-level pipelined API for SLO uploads.
|
Package pipeline implements a low-level pipelined API for SLO uploads. |