df2

command module
v1.11.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 30, 2023 License: Apache-2.0 Imports: 14 Imported by: 0

README

df2

Go Report Card GitHub release (latest SemVer) GitHub

The df2 program is a terminal tool for managing plus optimising the files and database of defacto2.net that is broken down into multiple parts.

The tool to optimise and manage defacto2.net
Copyright © 2020-23 Ben Garrett
https://github.com/Defacto2/df2

Usage:
  df2 [flags]
  df2 [command]

Admin:
  approve     Approve the records that are ready to go live.
  fix         Fixes database entries and records.
  new         Manage files marked as waiting to go live (default).
  output      Generators for JSON, HTML, SQL and sitemap documents.
  proof       Manage records tagged as #releaseproof.

Drive:
  clean       Discover or clean orphan files.
  shrink      Reduces the space used in directories.

Remote:
  apis        Batch data synchronization with remote APIs.
  demozoo     Interact with Demozoo submissions.
  lookup      Lookup the file URL of a record's ID or UUID.
  test        Test various features of the website or database that cannot be fixed with automation.

Additional Commands:
  config      Configure the settings for this tool.
  help        Help about any command

Flags:
      --ascii     suppress all ANSI color feedback
  -h, --help      help for df2
      --quiet     suppress all feedback except for errors
  -v, --version   version and information for this program

Use "df2 [command] --help" for more information about a command.

Install or update

df2 is built on Go and is packaged for Debian Linux.

# download the package
wget https://github.com/Defacto2/df2/releases/latest/download/df2.deb

# install or update the package
dpkg -i df2.deb

# test the new install
df2 --version
Dependencies
Dependency installation on Ubuntu
# required dependencies
sudo apt install -y ansilove imagemagick netpbm pngquant webp

# optional file archivers
sudo apt install -y arj lhasa unrar unzip
Database dependancy

The df2 program expects local access to the Defacto2 database.

Configuration

To view and test the database and directory configurations.

df2 config info

To change the configuration.

df2 config edit

Docker container

The Docker container runs on a Go container built in Debian Linux. The main purpose is unit testing and compiling of the Go source code in a Linux environment.

# change directory to the local repository
cd df2
# synchronize any remote repository tags
git pull 
# build the current directory as an image tagged as 'df2'
docker build --tag df2 . 
# run the image tagged as 'df2' with the container name 'df2-test'
docker run -it --name df2-test df2

Use the code on your own system

Confirm the installation of Go on Linux, WSL or macOS.

$ go version

go version go1.19.5 linux/amd64

Clone the Defacto2/df2 repository.

$ git clone git@github.com:Defacto2/df2.git

Cloning into 'df2'...

Run the df2 tool. It may take a moment on the first run as it downloads dependencies.

$ cd df2
$ go run . --version

  version  0.0.0 (developer build)
     path  /tmp/go-build3011510136/b001/exe/df2
   commit  unknown
     date  unknown
       go  v1.19.5 linux/amd64

Build the df2 tool (not usually required).

$ cd df2
$ go build
$ ./df2 --version

  version  0.0.0 (developer build)
     path  /home/ben/df2/df2
   commit  3352b5393353d4e09cf19f636d4be593d961cece
     date  2023 Jan 3, 02:01 UTC
       go  v1.19.5 linux/amd64

Releasing

GitHub Actions combined with GoReleaser handles the building process when new release tags are created.

All changes should be tested with the golangci-lint Go linters aggregator.

Usage tips

import

Use rsync to upload to the remote server.

# rsync -a  optional archive mode
# rsync -P  optional progress bar and keeps the partially transferred files
# some_archive.rar is a local file to upload
# user@[ip address] is the remote destination and user account with SSH access.
# :~/downloads will place the some_archive.rar to the user account downloads directory.
rsync -aP some_archive.rar user@[ip address]:~/downloads

On the remote server.

df2 import ~/downloads/some_archive.rar --limit=10

Documentation

Overview

Package main is the command-line tool to manage and optimize defacto2.net.

Source Files

View all Source files

Directories

Path Synopsis
cmd
Package cmd uses the Cobra package to provide and handle all the interactions with the command-line interface in Linux.
 Package cmd uses the Cobra package to provide and handle all the interactions with the command-line interface in Linux.
internal/arg
Package arg has structures used to store the flag values used by the cobra.Command methods.
 Package arg has structures used to store the flag values used by the cobra.Command methods.
internal/run
Package run is a collection of work functions used by the cobra.Command methods.
 Package run is a collection of work functions used by the cobra.Command methods.
pkg
archive
Package archive handles collections of files that are either packed together or packed and compressed.
 Package archive handles collections of files that are either packed together or packed and compressed.
archive/internal/arc
Package arc handles interactions with the mholt archiver package.
 Package arc handles interactions with the mholt archiver package.
archive/internal/content
Package content stats the content of an archive.
 Package content stats the content of an archive.
archive/internal/demozoo
Package demozoo handles downloads fetched from Demozoo Productions links.
 Package demozoo handles downloads fetched from Demozoo Productions links.
archive/internal/file
Package file handles common file system operations such as move, copy and dir.
 Package file handles common file system operations such as move, copy and dir.
archive/internal/sys
Package sys uses programs installed to the host operating system to handle miscellaneous archives not usable with the Go packages.
 Package sys uses programs installed to the host operating system to handle miscellaneous archives not usable with the Go packages.
archive/internal/task
Package task handles the scanning of directories for release proofs.
 Package task handles the scanning of directories for release proofs.
assets
Package assets handles the web resources of the site such as the file downloads, thumbnails and backups.
 Package assets handles the web resources of the site such as the file downloads, thumbnails and backups.
assets/internal/file
Package file writes the results of an asset scan or walk.
 Package file writes the results of an asset scan or walk.
assets/internal/scan
Package scan handles individual directories housing specific assets.
 Package scan handles individual directories housing specific assets.
conf
Package conf sets the configurations of this program using the host system environment variables.
 Package conf sets the configurations of this program using the host system environment variables.
database
Package database interacts with the remote datastore for Defacto2.
 Package database interacts with the remote datastore for Defacto2.
database/internal/export
Package export queries records for data exports.
 Package export queries records for data exports.
database/internal/recd
Package recd has various checks for record values.
 Package recd has various checks for record values.
database/internal/templ
Package templ has the templates for the database SQL statements.
 Package templ has the templates for the database SQL statements.
database/internal/update
Package update handles edits and updates to the database records.
 Package update handles edits and updates to the database records.
database/msql
Package msql creates connections to MySQL datastores.
 Package msql creates connections to MySQL datastores.
database/psql
Package psql creates connections to Postgres datastores.
 Package psql creates connections to Postgres datastores.
demozoo
Package demozoo interacts with the demozoo.org API for data scraping and fetching user file downloads.
 Package demozoo interacts with the demozoo.org API for data scraping and fetching user file downloads.
demozoo/internal/filter
Package filter confirms a Demozoo Production is suitable for Defacto2.
 Package filter confirms a Demozoo Production is suitable for Defacto2.
demozoo/internal/fix
Package fix repairs any imported Demozoo production data that may conflict or be incompatible with the Defacto2 database.
 Package fix repairs any imported Demozoo production data that may conflict or be incompatible with the Defacto2 database.
demozoo/internal/insert
Package insert adds Demozoo productions to the Defacto2 database.
 Package insert adds Demozoo productions to the Defacto2 database.
demozoo/internal/prod
Package prod obtains a Demozoo Production.
 Package prod obtains a Demozoo Production.
demozoo/internal/prods
Package prods handles marshalling of a Demozoo Production.
 Package prods handles marshalling of a Demozoo Production.
demozoo/internal/releaser
Package releaser handles Demozoo Releasers which are objects for both scene groups and people.
 Package releaser handles Demozoo Releasers which are objects for both scene groups and people.
demozoo/internal/releases
Package releases handles collections of Demozoo Productions that share a common filter such as a platform or category type.
 Package releases handles collections of Demozoo Productions that share a common filter such as a platform or category type.
directories
Package directories interacts with the filepaths that hold both the user files for downloads and website assets.
 Package directories interacts with the filepaths that hold both the user files for downloads and website assets.
directories/internal/create
Package create handles the making of directories.
 Package create handles the making of directories.
download
Package download fetches remote files used by this program and the website.
 Package download fetches remote files used by this program and the website.
download/internal/cnter
Package cnter is an optional progress counter for the remote file downloads.
 Package cnter is an optional progress counter for the remote file downloads.
groups
Package groups deals with group names and their initialisms.
 Package groups deals with group names and their initialisms.
groups/internal/acronym
Package acronym handles the initialisms and acronyms frequently used by the groups.
 Package acronym handles the initialisms and acronyms frequently used by the groups.
groups/internal/filter
Package filter returns the groups based on a role or activity filter.
 Package filter returns the groups based on a role or activity filter.
groups/internal/rename
Package rename handles the renaming and formatting of the group title.
 Package rename handles the renaming and formatting of the group title.
groups/internal/request
Package request obtains and writes the data of the group to various formats.
 Package request obtains and writes the data of the group to various formats.
images
Package images generate thumbnails and converts these between image formats.
 Package images generate thumbnails and converts these between image formats.
images/internal/file
Package file handles the images as files.
 Package file handles the images as files.
images/internal/imagemagick
Package imagemagick interacts with ImageMagick graphic programs.
 Package imagemagick interacts with ImageMagick graphic programs.
images/internal/netpbm
Package netpbm interacts with the netpdm graphics programs and programming library.
 Package netpbm interacts with the netpdm graphics programs and programming library.
importer
Package importer parses specific .NFO and file_id.DIZ file group-packs submitted as .rar archives.
 Package importer parses specific .NFO and file_id.DIZ file group-packs submitted as .rar archives.
importer/again
importer/air
importer/amplify
importer/arcade
importer/arctic
importer/hexwars
importer/record
Package record creates an object that can be written as JSON or used as a new database record.
 Package record creates an object that can be written as JSON or used as a new database record.
importer/spirit
importer/xdb
importer/zone
importer/zwt
Package zwt handles the .rar for the group Zero Waiting Time (ZWT).
 Package zwt handles the .rar for the group Zero Waiting Time (ZWT).
internal
Package internal has shared testdata.
 Package internal has shared testdata.
logger
Package logger uses the zap logging library for application logs.
 Package logger uses the zap logging library for application logs.
logger/internal/terminal
Package terminal returns information on the host Linux terminal.
 Package terminal returns information on the host Linux terminal.
models/mysql
models/psql
people
Package people handles scene persons, their names, aliases and roles.
 Package people handles scene persons, their names, aliases and roles.
people/internal/person
Package person contains the shared Person object for individual people.
 Package person contains the shared Person object for individual people.
people/internal/role
Package role handles the job or activity a person was credited with.
 Package role handles the job or activity a person was credited with.
prompt
Package prompt reads and returns standard input from the user.
 Package prompt reads and returns standard input from the user.
proof
Package proof handles files that have the section tagged as releaseproof.
 Package proof handles files that have the section tagged as releaseproof.
proof/internal/record
Package record handles the proof database record.
 Package record handles the proof database record.
recent
Package recent is a work-in-progress, JSON generator to display the most recent files on the file.
 Package recent is a work-in-progress, JSON generator to display the most recent files on the file.
shrink
Package shrink reduces the hard drive space used by the web application.
 Package shrink reduces the hard drive space used by the web application.
shrink/internal/data
Package data are various functions for the shrink package.
 Package data are various functions for the shrink package.
sitemap
Package sitemap generates a complete list of pages available on the website.
 Package sitemap generates a complete list of pages available on the website.
sitemap/internal/urlset
Package urlset handles creation of XML formatted URLs.
 Package urlset handles creation of XML formatted URLs.
str
Package str has string and stdout functions.
 Package str has string and stdout functions.
text
Package text generates preview images and thumbnails from text files using the system installed Ansilove/C program.
 Package text generates preview images and thumbnails from text files using the system installed Ansilove/C program.
text/internal/img
Package img has the functions to convert, compress and resize the images.
 Package img has the functions to convert, compress and resize the images.
text/internal/tf
Package tf has the functions to extract and convert text files into images.
 Package tf has the functions to extract and convert text files into images.
zipcmmt
Package zipcmmt processes the text adverts that are sometimes embedded into Zip file archives.
 Package zipcmmt processes the text adverts that are sometimes embedded into Zip file archives.
zipcmmt/internal/cmmt
Package cmmt discovers and stores any found zip comments.
 Package cmmt discovers and stores any found zip comments.
zipcontent
Package zipcontent processes the directory and file content of a file archive.
 Package zipcontent processes the directory and file content of a file archive.
zipcontent/internal/record
Package record saves to the database the data collected from the file archive.
 Package record saves to the database the data collected from the file archive.
zipcontent/internal/scan
Package scan initializes and stores statistics on a file archive.
 Package scan initializes and stores statistics on a file archive.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.