tempest

README

Tempest

Leverage paste sites as a medium for discovery of objectionable/infringing materials.

Table of Contents

• Information
  • About
  • Features
  • Disclaimer
  • Installation
  • Usage
  • Cloud Storage / File Sharing Platform Modules
  • Entry Format
  • Important Notes
  • TODO

Information

About

Tempest is a simple, lightweight, and cross-platform solution designed to enable individuals to efficiently discover and extract active cloud storage/file sharing links from paste platforms such as Rentry.co. It was created to address the notable uptick in paste sites being used to distribute content that violates copyright and piracy statutes.

Features

• Scrape and extract information from multiple different cloud storage/file sharing platforms (see Cloud Storage/File Sharing Platform Modules)
• Print results to the terminal or output them to a specified JSON/CSV file
• Built in clean function for cleaning/validating/deduplicating JSON/CSV files generated by Tempest
• Design philosophy revolving around high documentation coverage and modularity, enabling easy maintenance, contribution, and integration.

Disclaimer

It is the end user's responsibility to obey all applicable local, state, and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program. By using Tempest, you agree to the previous statements.

Installation

1. Fetch the repository via git clone: git clone https://github.com/ax-i-om/tempest.git
2. Navigate to the root directory of of the cloned repository via cd: cd tempest
3. In your preferred terminal, enter and run: go run main.go

OR

1. Install the repository via go install: go install github.com/ax-i-om/tempest@latest
2. In your preferred terminal, enter and run: tempest

Usage

Display Tempest usage help in the terminal via: tempest help

Tempest supports three primary methods of output, those being JSON, CSV, and plain text (output to console). If you want to output plain text to the console, run tempest like so: tempest console.

Note: If you want to output the console results to a file, append this to the command: 2>&1 | tee results.txt
For example tempest console 2>&1 | tee results.txt (may vary depending on operating system)
CAUTION: IF THE SPECIFIED OUTPUT FILE ALREADY EXISTS, THIS WILL OVERWRITE THE CONTENTS

If you want to output the results to a JSON/CSV file, the command should be formatted like so: tempest <json/csv> <filename>
JSON Example: tempest json results VS CSV Example: tempest csv results
Note: If you exclude the file extension (.json/.csv), one will be automatically appended.

In order to gracefully shut down Tempest, press Ctrl + C in the terminal ONCE and wait until the remaining goroutines finish executing (typically <60s).
In order to forcefully shut down Tempest press Ctrl + C in the terminal TWICE.
CAUTION: FORCEFULLY SHUTTING DOWN TEMPEST MAY RESULT IN ISSUES INCLUDING, BUT NOT LIMITED TO, DATA LOSS AND FILE CORRUPTION.

If you decide to output the results to a JSON file specifically, it will not be valid JSON.
Tempest comes bundled with a function for cleaning the resulting JSON content and can be used like so: tempest clean results.json
This will be the quickest way of converting the JSON file formatting into one that is valid; however, reusing this file for results will cause further formatting issues. The clean function also removes any duplicate entries from the file. The clean function will also remove duplicate entries from CSV files generated by tempest. Note: Unlike other functions in Tempest, a file extension (.json/.csv) will not be automatically appended. When cleaning, you must specify the file extension.

Append -d or --debug flag to the command to print more detailed logs

Cloud Storage / File Sharing Platform Modules

Module	Status	Information Extracted
Bunkr	Functioning	Link, Title, Service, Type, Size, FileCount, Thumbnail, Views
CloudMailRu	Functioning	Link, Title, Service, Type, Size, MTime, Hash, Malware
Cyberdrop	Functioning	Link, Title, Service, Type, Size, FileCount, Thumbnail, Description, UploadDate
Dood	Functioning	Link, Service, Type
Gofile	Functioning	Link, Title, Service, Type, FileCount, Downloads
Google Drive	Functioning	Link, Title, Service, Type
Mega	Functioning	Link, Service, Type, Size, FileCount
Sendvid	Functioning	Link, Title, Service, Type, Thumbnail, Views

Entry Format

// Entry represents the extracted link and it's accompanying data.
type Entry struct {
	Source string `json:"source"`
	Link   string `json:"link"`

	Title       string `json:"title"`
	Description string `json:"description"`
	Service     string `json:"service"`

	Uploaded string `json:"uploaded"`
	Mtime    string `json:"mtime"`

	Type      string `json:"type"`
	Size      string `json:"size"`
	FileCount int    `json:"filecount"`

	Thumbnail string `json:"thumbnail"`
	Downloads int    `json:"downloads"`
	Views     int    `json:"views"`

	Hash    string `json:"hash"`
	Malware string `json:"malware"`
}

Important Notes

• Mega file count and size is unreliable, as the metadata specified in the Mega folder/file headers doesn't seem to accurately align with the true content's file count/size. Take with a grain of salt.
• CSV values are delimited with commas (,). Ensure that when opening/rendering/presenting the CSV file, fields are not separated via other characters/delimeters such as semicolons (;) and tabs as this may cause presentation/formatting issues.

TODO

• Add tests
• Better logging
• Implement proxy support?
• General optimization & cleanup
• Improve error handling