gocfl

README

Attention

New simplified config file format Look at gocfl2.toml

Installation

go install github.com/je4/gocfl/v2/gocfl@latest

Go OCFL Implementation

This library supports the Oxford Common Filesystem Layout (OCFL) and focuses on creation, update, validation and extraction of ocfl StorageRoots and Objects.

GOCFL command line tool supports the following subcommands

• init
• add
• create
• update
• validate
• info
• extract
• extractmeta
• display

There's a quickstart guide available.

Why

There are several OCFL tools & libraries which already exists. This software is build with the following motivation.

I/O Performance

Regarding performance, Storage I/O generates the main performance issues. Therefor, every file should be read and written only once. Only in case of deduplication, the checksum of a file is calculated before ingest and a second time while ingesting.

Container

Serialization of an OCFL Storage Root into a container format like ZIP must not generate overhead on disk I/O. Therefor generation of an OCFL Container is possible without an intermediary OCFL Storage Root on a filesystem.

Encryption

For storing OCFL containers in low-security locations (cloud storage, etc.), it's possible to create an AES-256 encrypted container on ingest.

Extensions

The extensions described in the OCFL standard are quite open in their functionality and may belong to the Storage Root or Object. Since there's no specification of a generic extension api, it's difficult to integrate specific extension hooks into other libraries. This library identifies 7 different extension hooks so far.

Indexer

When content is ingested into OCFL objects, technical metadata should be extracted and stored alongside the manifest data. This allows technical metadata to be extracted alongside the content. Since the OCFL structure is quite rigid, there's a need for a special extension to support this.

Functionality

• Supports local filesystems
• Supports S3 Cloud Storage (via MinIO Client SDK)
• SFTP Storage
• Google Cloud Storage
• Serialization into ZIP Container
• AES Encryption of Container
• Supports mixing of source and target storage systems
• Non blocking validation (does not stop on validation errors)
• Support for OCFL v1.0 and v1.1
• Documentation for API
• Digest Algorithms for Manifest: SHA512, SHA256
• Fixity Algorithms: SHA1, SHA256, SHA512, BLAKE2b-160, BLAKE2b-256, BLAKE2b-384, BLAKE2b-512, MD5
• Concurrent checksum generation on ingest/extract (multi-threaded)
• Minimized I/O (data is read and written only once on Object creation)
• Update strategy echo (incl. deletions) and contribute
• Deduplication (needs double read of all content files, switchable)
• Nearly full coverage of validation errors and warnings
• Content information
• Extraction with version selection
• Display of content via Webserver
• Report generation
• Community Extensions
  • 0001-digest-algorithms
  • 0002-flat-direct-storage-layout
  • 0003-hash-and-id-n-tuple-storage-layout
  • 0004-hashed-n-tuple-storage-layout
  • 0005-mutable-head
  • 0006-flat-omit-prefix-storage-layout
  • 0007-n-tuple-omit-prefix-storage-layout
  • 0008-schema-registry
• Local Extensions
  • NNNN-pairtree-storage-layout
  • NNNN-direct-clean-path-layout
  • NNNN-content-subpath (integration of non-payload files in content)
  • NNNN-metafile (integration of a metadata file)
  • NNNN-mets (generation of mets and premis files)
  • NNNN-indexer (technical metadata indexing)
  • NNNN-migration (migration of file formats)
  • NNNN-gocfl-extension-manager (initial extension for sorted exclusion and sorted execution)
  • NNNN-filesystem (filesystem metadata extension)
  • NNNN-thumbnail (generation of thumbnails)

Command Line Interface

A fast and reliable OCFL creator, extractor and validator.
https://github.com/je4/gocfl
Jürgen Enge (University Library Basel, juergen@info-age.net)
Version v1.0-beta.7

Usage:
  gocfl [flags]
  gocfl [command]

Available Commands:
  add         adds new object to existing ocfl structure
  completion  Generate the autocompletion script for the specified shell
  create      creates a new ocfl structure with initial content of one object
  display     show content of ocfl object in webbrowser
  extract     extract version of ocfl content
  extractmeta extract metadata from ocfl structure
  help        Help about any command
  init        initializes an empty ocfl structure
  stat        statistics of an ocfl structure
  update      update object in existing ocfl structure
  validate    validates an ocfl structure

Flags:
      --config string                 config file (default is $HOME/.gocfl.toml)
  -h, --help                          help for gocfl
      --log-file string               log output file (default is console)
      --log-level string              log level (CRITICAL|ERROR|WARNING|NOTICE|INFO|DEBUG) (default "ERROR")
      --s3-access-key-id string       Access Key ID for S3 Buckets
      --s3-endpoint string            Endpoint for S3 Buckets
      --s3-region string              Region for S3 Access
      --s3-secret-access-key string   Secret Access Key for S3 Buckets
      --with-indexer                  starts indexer as a local service

Use "gocfl [command] --help" for more information about a command.