picdexer

README

picdexer

picdexer is a picture indexing solution :

• it extracts metadata from pictures
• metadata are indexed on elasticsearch
• pictures can be downsampled and stored using file-server
• it provides a view over indexed data using kibana

picdexer orchestrates the whole process :

• it configures elasticsearch
• it configures kibana
• it extracts and index pictures metadata
• it can store pictures (possibly resized)

Getting pixdexer

picdexer is released :

• As a compiled binary on the Github project release section. If you have a golang environment, you can compile the projet : go build -o file-server picdexer.go
• As a docker imaged on Docker Hub. If you have a [docker environment], you can build the image : docker build .

How to configure

In the picdexer docker image :

• the configuration file is located in /etc/picdexer/picdexer.json
• the "input" folder is located in /data/picdexer/in/
• the "output" folder (for simulations) is located in /data/picdexer/out/

The configuration file is a JSON file, here is a complexe example :

{
  "loggingLevel": "warn",
  "elasticsearch": {
    "url": "http://192.168.1.102:9200",
    "extractionThreadCount": 4,
    "toExtractChannelSize": 50
  },
  "binary": {
    "url": "http://192.168.1.100:8080",
    "height":480,
    "width":640,
    "resizingThreadCount": 4,
    "toResizeChannelSize": 4 
  },
  "kibana": {
    "url": "http://192.168.1.102:5601"
  }
}

• loggingLevel (optional) defines the logging level. Values :debug, info (default), warn, error, ...
• elasticsearch (required if used) configures the elasticsearch connexion and the metadata extraction process
  • url (required if documents are pushed) defines the elasticsearch endpoint
  • extractionThreadCount (optional, default : 4) defines how many thread have to be used to extract medatada
  • toExtractChannelSize (optimal, default : 50) defines the size of the file buffer (go channel) that will be consumed by the extraction goroutines
• binary (required if used) configures the interactions with file-server to store pictures
  • url (required if pictures are pushed) defines the file-server endpoint
  • height and width defines the target dimension of the pictures that will be stored. If one of the dimension is 0 then pictures will not be resized (default behaviour).
  • resizingThreadCount (optional, default : 4) defines how many thread have to be used to resize pictures
  • toResizeChannelSize (optimal, default : 4) defines the size of the file buffer (go channel) that will be consumed by the resizing goroutines
• kibana (required if user) configures the interaction with kibana (for configuration purpose)
  • url (required if kibana has to be configured) defines the kibana endpoint

Picdexer commands

picdexer have several commands that can be used. Each command is dedicated to specific purpose.

If the return code is 0, the command is successfully execute. If any problem happens, the return code will be > 0.

Setup

picdexer can setup :

• elasticsearch mapping (mapping.json)
• kibana index-pattern, visualizations, dashboards (kibana.ndjson)

The elasticsearch and kibana part of the configuration file has to be filled.

• Command line version : ./pcidexer setup -c [configurationFile]
• Docker version :

docker run --rm
  -v [hostConfigurationFile]:/etc/picdexer/picdexer.json
  barasher/picdexer:1.0.0 ./setup.sh

Full process

The full process command extracts metadata, resize (eventually) and store pictures.

• Command line version : ./picdexer full -c [configurationFile] -d [sourceFolder]
• Docker version :

docker run --rm
  -v [hostSourceFolder]:/data/picdexer/in
  -v [hostConfigurationFile]:/etc/picdexer/picdexer.json
  barasher/picdexer:1.0.0 ./full.sh

Metadata management

The metadata command deals with elasticsearch metadata indexation.

Metadata extraction simulation

This command simulates the indexation (dump the elasticsearch documents on stdout).

• Command line version : ./picdexer metadata simulate -c [configurationFile] -d [sourceFolder]
• Docker version :

docker run --rm
  -v [hostSourceFolder]:/data/picdexer/in
  -v [hostConfigurationFile]:/etc/picdexer/picdexer.json
  barasher/picdexer:1.0.0 ./metadata_simulate.sh

Metadata indexation

This command indexes metadata to elasticsearch.

• Command line version : ./picdexer metadata index -c [configurationFile] -d [sourceFolder]
• Docker version :

docker run --rm
  -v [hostSourceFolder]:/data/picdexer/in
  -v [hostConfigurationFile]:/etc/picdexer/picdexer.json
  barasher/picdexer:1.0.0 ./metadata_index.sh

Listing all metadata from a file

This command extracts all the metadata available for a picture : ./picdexer metadata display -f [pictureFile]

Pictures (binary) management

The binary command deals with the picture storage in file-server.

Picture resizing simulation

This command simulates the resizing and stores resized file in the filesystem.

• Command line version : ./picdexer binary simulate -c [configurationFile] -d [sourceFolder] -o [outputFolder].
• Docker version :

docker run --rm
  -v [hostSourceFolder]:/data/picdexer/in
  -v [hostTargetFolder]:/data/picdexer/out
  -v [hostConfigurationFile]:/etc/picdexer/picdexer.json
  barasher/picdexer:1.0.0 ./binary_simulate.sh

Picture pushing

This command resizes pictures and stores the resized version in file-server.

• Command line version : ./picdexer binary push -c [configurationFile] -d [sourceFolder].
• Docker version :

docker run --rm
  -v [hostSourceFolder]:/data/picdexer/in
  -v [hostConfigurationFile]:/etc/picdexer/picdexer.json
  barasher/picdexer:1.0.0 ./binary_push.sh