wayback

README

Wayback

Wayback is a tool that supports running as a command-line tool and docker container, purpose to snapshot webpage to time capsules.

Features

• Cross platform
• Batch wayback URLs
• Builtin CLI (wayback)
• Serve as Tor Hidden Service or local web entry
• Wayback to Internet Archive, archive.today, IPFS and Telegraph easier
• Interactive with IRC, Martix, Telegram bot, Mastodon and Twitter as daemon service
• Support publish wayback resutls to Telegram channel, Mastodon and GitHub Issues

Installation

From source:

$ go get -u github.com/wabarc/wayback/cmd/wayback

From GoBinaries:

$ curl -sf https://gobinaries.com/wabarc/wayback/cmd/wayback | sh

Using Snapcraft (on GNU/Linux)

$ sudo snap install wayback

See more on releases.

Usage

Command line

$ wayback -h

A CLI tool for wayback webpages.

Usage:
  wayback [flags]

Examples:
  wayback https://www.wikipedia.org
  wayback https://www.fsf.org https://www.eff.org
  wayback --ia https://www.fsf.org
  wayback --ia --is -d telegram -t your-telegram-bot-token
  WAYBACK_SLOT=pinata WAYBACK_APIKEY=YOUR-PINATA-APIKEY \
    WAYBACK_SECRET=YOUR-PINATA-SECRET wayback --ip https://www.fsf.org

Flags:
  -c, --chatid string      Telegram channel id.
  -d, --daemon strings     Run as daemon service, e.g. telegram, web
      --debug              Enable debug mode. (default false)
  -h, --help               help for wayback
      --ia                 Wayback webpages to Internet Archive.
      --ip                 Wayback webpages to IPFS. (default false)
      --ipfs-host string   IPFS daemon host, do not require, unless enable ipfs. (default "127.0.0.1")
  -m, --ipfs-mode string   IPFS mode. (default "pinner")
  -p, --ipfs-port uint     IPFS daemon port. (default 5001)
      --is                 Wayback webpages to Archive Today.
      --ph                 Wayback webpages to Telegraph. (default false)
  -t, --token string       Telegram Bot API Token.
      --tor                Snapshot webpage via Tor anonymity network.
      --tor-key string     The private key for Tor Hidden Service.
  -v, --version            version for wayback

Examples

Wayback one or more url to Internet Archive and archive.today:

$ wayback https://www.wikipedia.org

$ wayback https://www.fsf.org https://www.eff.org

Wayback url to Internet Archive or archive.today or IPFS:

// Internet Archive
$ wayback --ia https://www.fsf.org

// archive.today
$ wayback --is https://www.fsf.org

// IPFS
$ wayback --ip https://www.fsf.org

For the IPFS, also can use a specific pinner service:

$ export WAYBACK_SLOT=pinata
$ export WAYBACK_APIKEY=YOUR-PINATA-APIKEY
$ export WAYBACK_SECRET=YOUR-PINATA-SECRET
$ wayback --ip https://www.fsf.org

// or

$ WAYBACK_SLOT=pinata WAYBACK_APIKEY=YOUR-PINATA-APIKEY \
$ WAYBACK_SECRET=YOUR-PINATA-SECRET wayback --ip https://www.fsf.org

TIP: more details about pinner service.

With telegram bot:

$ wayback --ia --is --ip -d telegram -t your-telegram-bot-token

Publish message to your Telegram channel at the same time:

$ wayback --ia --is --ip -d telegram -t your-telegram-bot-token -c your-telegram-channel-name

Also can run with debug mode:

$ wayback -d telegram -t YOUR-BOT-TOKEN --debug

Both serve on Telegram and Tor hidden service:

$ wayback -d telegram -t YOUT-BOT-TOKEN -d web

Configuration Parameters

You can specify configuration options either via command flags or via environment variables, an overview of all options below.

Flags	Environment Variable	Default	Description
--debug	DEBUG	false	Enable debug mode
-	LOG_TIME	true	Display the date and time in log messages
-d, --daemon	-	-	Run as daemon service, e.g. telegram, web, mastodon, twitter
--ia	WAYBACK_ENABLE_IA	true	Wayback webpages to Internet Archive
--is	WAYBACK_ENABLE_IS	true	Wayback webpages to Archive Today
--ip	WAYBACK_ENABLE_IP	false	Wayback webpages to IPFS
--ph	WAYBACK_ENABLE_PH	false	Wayback webpages to Telegra.ph, required Chrome/Chromium
--ipfs-host	WAYBACK_IPFS_HOST	127.0.0.1	IPFS daemon service host
-p, --ipfs-port	WAYBACK_IPFS_PORT	5001	IPFS daemon service port
-m, --ipfs-mode	WAYBACK_IPFS_MODE	pinner	IPFS mode for preserve webpage, e.g. daemon, pinner
-	WAYBACK_GITHUB_TOKEN	-	GitHub Personal Access Token, required the repo scope
-	WAYBACK_GITHUB_OWNER	-	GitHub account name
-	WAYBACK_GITHUB_REPO	-	GitHub repository to publish results
-t, --token	WAYBACK_TELEGRAM_TOKEN	-	Telegram Bot API Token
-c, --chatid	WAYBACK_TELEGRAM_CHANNEL	-	The Telegram Channel name for publish archived result
-	WAYBACK_MASTODON_SERVER	-	Domain of Mastodon instance
-	WAYBACK_MASTODON_KEY	-	The client key of your Mastodon application
-	WAYBACK_MASTODON_SECRET	-	The client secret of your Mastodon application
-	WAYBACK_MASTODON_TOKEN	-	The access token of your Mastodon application
-	WAYBACK_TWITTER_CONSUMER_KEY	-	The customer key of your Twitter application
-	WAYBACK_TWITTER_CONSUMER_SECRET	-	The customer secret of your Twitter application
-	WAYBACK_TWITTER_ACCESS_TOKEN	-	The access token of your Twitter application
-	WAYBACK_TWITTER_ACCESS_SECRET	-	The access secret of your Twitter application
-	WAYBACK_IRC_NICK	-	IRC nick
-	WAYBACK_IRC_PASSWORD	-	IRC password
-	WAYBACK_IRC_CHANNEL	-	IRC channel
-	WAYBACK_IRC_SERVER	irc.freenode.net:7000	IRC server, required TLS
-	WAYBACK_MATRIX_HOMESERVER	https://matrix.org	Matrix homeserver
-	WAYBACK_MATRIX_USERID	-	Matrix unique user ID, format: @foo:example.com
-	WAYBACK_MATRIX_ROOMID	-	Matrix unique room ID, format: #bar:example.com
-	WAYBACK_MATRIX_PASSWORD	-	Matrix password
--tor	WAYBACK_USE_TOR	false	Snapshot webpage via Tor anonymity network
--tor-key	WAYBACK_TOR_PRIVKEY	-	The private key for Tor Hidden Service
-	WAYBACK_TOR_LOCAL_PORT	8964	Local port for Tor Hidden Service, also support for a reverse proxy
-	WAYBACK_TOR_REMOTE_PORTS	80	Remote ports for Tor Hidden Service, e.g. WAYBACK_TOR_REMOTE_PORTS=80,81
-	WAYBACK_TORRC	/etc/tor/torrc	Using torrc for Tor Hidden Service
-	WAYBACK_SLOT	-	Pinning service for IPFS mode of pinner, see ipfs-pinner
-	WAYBACK_APIKEY	-	API key for pinning service
-	WAYBACK_SECRET	-	API secret for pinning service

Docker/Podman

$ docker pull wabarc/wayback
$ docker run -d wabarc/wayback wayback -d telegram -t YOUR-BOT-TOKEN # without telegram channel
$ docker run -d wabarc/wayback wayback -d telegram -t YOUR-BOT-TOKEN -c YOUR-CHANNEL-USERNAME # with telegram channel

Deployment

• wabarc/on-heroku
• wabarc/on-github

TODO

Archive.org and Archive.today are currently supported, the next step mind support the followings platform:

• IPFS
• ZeroNet

Services

Telegram bot

• Bots: An introduction for developers
• How do I create a bot?
• An example bot
• An example channel

Mastodon bot

Bot friendly instance:

• botsin.space

F.A.Q

Q: How to keep the Tor hidden service hostname?

A: For the first time to run the wayback service, keep the key from the output message (the key is the part after private key: below) and next time to run the wayback service to place the key to the --tor-key option or the WAYBACK_TOR_PRIVKEY environment variable.

[INFO] Web: important to keep the private key: d005473a611d2b23e54d6446dfe209cb6c52ddd698818d1233b1d750f790445fcfb5ece556fe5ee3b4724ac6bea7431898ee788c6011febba7f779c85845ae87

Contributing

We encourage all contributions to this repository! Open an issue! Or open a Pull Request!

If you're interested in contributing to wayback itself, read our contributing guide to get started.

Note: All interaction here should conform to the Code of Conduct.

License

This software is released under the terms of the GNU General Public License v3.0. See the LICENSE file for details.

Documentation

Index

• type Archived
• type Broker
• type Collect
• type Handle
  • func (h *Handle) IA() Archived
  • func (h *Handle) IP() Archived
  • func (h *Handle) IS() Archived
  • func (h *Handle) PH() Archived

Types

type Archived

type Archived map[string]string

Archived returns result of wayback.

type Broker

type Broker interface {
	IA() Archived
	IS() Archived
	IP() Archived
	PH() Archived
}

Broker is interface of the wayback, methods returns `Archived`.

type Collect

type Collect struct {
	sync.RWMutex

	Arc string
	Dst map[string]string
	Ext string
}

Collect result that archived, Arc is name of the archive service, Dst mapping the original URL and archived destination URL, Ext is extra descriptions.

type Handle

type Handle struct {
	URLs []string

	Opts *config.Options
}

Handle URLs need to wayback and configs, Opts on `github.com/wabarc/wayback/config`.

func (*Handle) IA

func (h *Handle) IA() Archived

func (*Handle) IP

func (h *Handle) IP() Archived

func (*Handle) IS

func (h *Handle) IS() Archived

func (*Handle) PH

func (h *Handle) PH() Archived