README
¶
sorten
A helper utility to transform data representations for deep learning input.
Rationale
Given a directory of many input files which can be .jpeg, .hdr and so on, we need an easy way to convert as quickly as possible, all of these images and outputs into more malleable / desirable formats for learning and what-not. The intention of this is to take advantage of ridiculous CPU and memory offerings from cloud providers while mounting and transforming data stored in google buckets (for example).
Install
$ go get github.com/recogni/sorten
Or grab the latest release from the releases section.
The following additional CLI tools will also be required to run some of the image conversion jobs:
imagemagick:
You will also need imagemagick installed on your system. Modern versions of imagemagick will break apart the binary into separate sub-components like convert, compose and so on. If you have installed this to a non-standard path, you can set the path to the binaries using the --magic option in the command line.
brew install imagemagick
luminence-hdr
This is needed to apply the correct and appropriate tonemapping operator on the input HDR images, which will then give us losless 8-bit tiff images. From here we use imagemagick to convert again into jpeg.
Here is an interesting discussion on tonemapping: http://osp.wikidot.com/parameters-for-photographers.
Note: On OSX 10.9 you will need an older (2.2.1) version of this program.
http://qtpfsgui.sourceforge.net/?page_id=10
Cross compile
$ cd $GOPATH/src/github.com/recogni/sorten
$ GOOS=linux go build -o sorten_linux .
Usage(s?)
$ sorten -h
Usage for sorten:
-input string
input directory to read images from
-magic string
path to imagemagick binaries (default "/usr/local/bin/")
-output string
output directory to read images from
Currently sorten only supports converting a directory (recursively) of .hdr images to .jpeg. Future workers will be crafted to convert intermediate or regular formats between each other.
.hdr -> .jpeg
Assuming that an output directory exists at /mnt/bucket/output, and a nested directory structure of .hdr images live in /mnt/bucket/input, you can convert them with as many cores as your system has using the following command:
$ sorten -input /mnt/bucket/input -output /mnt/bucket/output
$ sorten -input /mnt/bucket/input -output /mnt/bucket/output -magic /usr/local/bin
UE4 captures -> TF Record
TODO
JPEG + Bounding boxes -> TF Record
TODO
Documentation
¶
Overview ¶
Sorten is a tool used to convert large batches of images from one format to another as needed by tensorflow-based learning systems.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package jobs implements generic helpers for use in various specific job types as defined in the subdirectories that follow.
|
Package jobs implements generic helpers for use in various specific job types as defined in the subdirectories that follow. |
|
explode
Package explode wraps all needed logic to run a parallel job that will take a directory of tf records and explode them into their respective classes.
|
Package explode wraps all needed logic to run a parallel job that will take a directory of tf records and explode them into their respective classes. |
|
implode
Package implode wraps all needed logic to run a parallel job that will take a directory of tf records and implode them from their respective classes.
|
Package implode wraps all needed logic to run a parallel job that will take a directory of tf records and implode them from their respective classes. |