archiveis

README

archiveis

About

archive.is is a golang package for archiving web pages via archive.is.

Please be mindful and responsible and go easy on them, we want archive.is to last forever!

Created by Jay Taylor.

Also see: archive.org golang package

TODO

• Add timeout to .Capture.
• Consider unifying to single binary

Requirements

• Go version 1.9 or newer

Installation

go get jaytaylor.com/archive.is/...

Usage

Command-line programs

archive.is <url>

Archive a fresh new copy of an HTML page

archive.is-snapshots <url>

Search for existing page snapshots

Search query examples:

• microsoft.com for snapshots from the host microsoft.com
• *.microsoft.com for snapshots from microsoft.com and all its subdomains (e.g. www.microsoft.com)
• http://twitter.com/burgerking for snapshots from exact url (search is case-sensitive)
• http://twitter.com/burg* for snapshots from urls starting with http://twitter.com/burg

Go package interfaces

Capture URL HTML Page Content

capture.go:

package main

import (
	"fmt"

	"github.com/jaytaylor/archive.is"
)

var captureURL = "https://jaytaylor.com/"

func main() {
	archiveURL, err := archiveis.Capture(captureURL)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Successfully archived %v via archive.is: %v\n", captureURL, archiveURL)
}

// Output:
//
// Successfully archived https://jaytaylor.com/ via archive.is: https://archive.is/i2PiW

Search for Existing Snapshots

search.go:

package main

import (
    "fmt"
    "time"

    "github.com/jaytaylor/archive.is"
)

var searchURL = "https://jaytaylor.com/"

func main() {
    snapshots, err := archiveis.Search(searchURL, 10*time.Second)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%# v\n", snapshots)
}

// Output:
//
//

Running the test suite

go test ./...

License

Permissive MIT license, see the LICENSE file for more information.

Documentation

Index

• Variables
• func Capture(u string, cfg ...Config) (string, error)
• type Config
• type Snapshot
  • func Search(url string, timeout time.Duration) ([]Snapshot, error)

Variables

var (
	BaseURL               = "https://archive.is"                                                                                                       // Overrideable default package value.
	HTTPHost              = "archive.is"                                                                                                               // Overrideable default package value.
	UserAgent             = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36" // Overrideable default package value.
	DefaultRequestTimeout = 10 * time.Second                                                                                                           // Overrideable default package value.
	DefaultPollInterval   = 5 * time.Second                                                                                                            // Overrideable default package value.

)

Functions

func Capture

func Capture(u string, cfg ...Config) (string, error)

Capture archives the provided URL using the archive.is service.

Types

type Config

type Config struct {
	Anyway         bool          // Force archival even if there is already a recent snapshot of the page.
	Wait           bool          // Wait until the crawl has been completed.
	WaitTimeout    time.Duration // Max time to wait for crawl completion.  Default is unlimited.
	PollInterval   time.Duration // Interval between crawl completion checks.  Defaults to 5s.
	RequestTimeout time.Duration // Overrides default request timeout.
	SubmitID       string        // Accepts a user-provided submitid.
}

Config settings for page capture client behavior.

type Snapshot

type Snapshot struct {
	URL          string
	ThumbnailURL string
	Timestamp    time.Time
}

Snapshot represents an instance of a URL page snapshot on archive.is.

func Search

func Search(url string, timeout time.Duration) ([]Snapshot, error)

Search for URL snapshots.