importer

README

Permanode type:

camliNodeType: "importer"
importerType: "twitter"
authClientID:     "xxx"    // e.g. api token
authClientSecret: "sdkojfsldfjlsdkf"

camliNodeType: "importerAccount"
importerType: "twitter"
twitterAccount: "bradfitz"

Documentation

Overview

Package importer imports content from third-party websites.

Index

• func Register(name string, im Importer)
• type Host
  • func (h *Host) BaseURL() string
  • func (h *Host) HTTPClient() *http.Client
  • func (h *Host) HTTPTransport() http.RoundTripper
  • func (h *Host) ImporterBaseURL() string
  • func (h *Host) InitHandler(hl blobserver.FindHandlerByTyper) error
  • func (h *Host) NewObject() (*Object, error)
  • func (h *Host) ObjectFromRef(permanodeRef blob.Ref) (*Object, error)
  • func (h *Host) Search() *search.Handler
  • func (h *Host) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (h *Host) Target() blobserver.StatReceiver
• type Importer
• type ImporterSetupHTMLer
• type OAuth1
  • func (OAuth1) CallbackRequestAccount(r *http.Request) (blob.Ref, error)
  • func (OAuth1) CallbackURLParameters(acctRef blob.Ref) string
• type OAuth2
  • func (OAuth2) CallbackRequestAccount(r *http.Request) (blob.Ref, error)
  • func (OAuth2) CallbackURLParameters(acctRef blob.Ref) string
  • func (OAuth2) RedirectState(imp Importer, ctx *SetupContext) (state string, err error)
  • func (OAuth2) RedirectURL(imp Importer, ctx *SetupContext) string
• type Object
  • func (o *Object) Attr(attr string) string
  • func (o *Object) Attrs(attr string) []string
  • func (o *Object) ChildPathObject(path string) (*Object, error)
  • func (o *Object) ForeachAttr(fn func(key, value string))
  • func (o *Object) PermanodeRef() blob.Ref
  • func (o *Object) SetAttr(key, value string) error
  • func (o *Object) SetAttrs(keyval ...string) error
• type ProgressMessage
• type RunContext
  • func (rc *RunContext) AccountNode() *Object
  • func (rc *RunContext) Credentials() (clientID, clientSecret string, err error)
  • func (rc *RunContext) RootNode() *Object
• type SetupContext
  • func (sc *SetupContext) AccountURL() string
  • func (sc *SetupContext) CallbackURL() string
  • func (sc *SetupContext) Credentials() (clientID, clientSecret string, err error)

Functions

func Register

func Register(name string, im Importer)

Register registers a site-specific importer. It should only be called from init, and not from concurrent goroutines.

Types

type Host

type Host struct {
	// contains filtered or unexported fields
}

Host is the HTTP handler and state for managing all the importers linked into the binary, even if they're not configured.

func (*Host) BaseURL

func (h *Host) BaseURL() string

BaseURL returns the root of the whole server, without trailing slash.

func (*Host) HTTPClient

func (h *Host) HTTPClient() *http.Client

HTTPClient returns the HTTP client to use.

func (*Host) HTTPTransport

func (h *Host) HTTPTransport() http.RoundTripper

HTTPTransport returns the HTTP transport to use.

func (*Host) ImporterBaseURL

func (h *Host) ImporterBaseURL() string

ImporterBaseURL returns the URL base of the importer handler, including trailing slash.

func (*Host) InitHandler

func (h *Host) InitHandler(hl blobserver.FindHandlerByTyper) error

func (*Host) NewObject

func (h *Host) NewObject() (*Object, error)

NewObject creates a new permanode and returns its Object wrapper.

func (*Host) ObjectFromRef

func (h *Host) ObjectFromRef(permanodeRef blob.Ref) (*Object, error)

ObjectFromRef returns the object given by the named permanode

func (*Host) Search

func (h *Host) Search() *search.Handler

func (*Host) ServeHTTP

func (h *Host) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP serves:

http://host/importer/
http://host/importer/twitter/
http://host/importer/twitter/callback
http://host/importer/twitter/sha1-abcabcabcabcabc (single account)

func (*Host) Target

func (h *Host) Target() blobserver.StatReceiver

type Importer

type Importer interface {
	// Run runs a full or incremental import.
	//
	// The importer should continually or periodically monitor the
	// context's Done channel to exit early if requested. The
	// return value should be context.ErrCanceled if the importer
	// exits for that reason.
	Run(*RunContext) error

	// NeedsAPIKey reports whether this importer requires an API key
	// (OAuth2 client_id & client_secret, or equivalent).
	// If the API only requires a username & password, or a flow to get
	// an auth token per-account without an overall API key, importers
	// can return false here.
	NeedsAPIKey() bool

	// IsAccountReady reports whether the provided account node
	// is configured.
	IsAccountReady(acctNode *Object) (ok bool, err error)
	SummarizeAccount(acctNode *Object) string

	ServeSetup(w http.ResponseWriter, r *http.Request, ctx *SetupContext) error
	ServeCallback(w http.ResponseWriter, r *http.Request, ctx *SetupContext)

	// CallbackRequestAccount extracts the blobref of the importer account from
	// the callback URL parameters of r. For example, it will be encoded as:
	// For Twitter (OAuth1), in its own URL parameter: "acct=sha1-f2b0b7da718b97ce8c31591d8ed4645c777f3ef4"
	// For Picasa: (OAuth2), in the OAuth2 "state" parameter: "state=acct:sha1-97911b1a5887eb5862d1c81666ba839fc1363ea1"
	CallbackRequestAccount(r *http.Request) (acctRef blob.Ref, err error)

	// CallbackURLParameters uses the input importer account blobRef to build
	// and return the URL parameters string (including the prefixed "?"), that
	// will be appended to the callback URL.
	CallbackURLParameters(acctRef blob.Ref) string
}

An Importer imports from a third-party site.

var TODOImporter Importer = todoImp{}

type ImporterSetupHTMLer

type ImporterSetupHTMLer interface {
	AccountSetupHTML(*Host) string
}

ImporterSetupHTMLer is an optional interface that may be implemented by Importers to return some HTML to be included on the importer setup page.

type OAuth1

type OAuth1 struct{}

OAuth1 provides methods that the importer implementations can use to help with OAuth authentication.

func (OAuth1) CallbackRequestAccount

func (OAuth1) CallbackRequestAccount(r *http.Request) (blob.Ref, error)

func (OAuth1) CallbackURLParameters

func (OAuth1) CallbackURLParameters(acctRef blob.Ref) string

type OAuth2

type OAuth2 struct{}

OAuth2 provides methods that the importer implementations can use to help with OAuth2 authentication.

func (OAuth2) CallbackRequestAccount

func (OAuth2) CallbackRequestAccount(r *http.Request) (blob.Ref, error)

func (OAuth2) CallbackURLParameters

func (OAuth2) CallbackURLParameters(acctRef blob.Ref) string

func (OAuth2) RedirectState

func (OAuth2) RedirectState(imp Importer, ctx *SetupContext) (state string, err error)

RedirectState returns the "state" query parameter that should be used for the authorization phase of OAuth2 authentication. This parameter contains the query component of the redirection URI. See http://tools.ietf.org/html/rfc6749#section-3.1.2.2

func (OAuth2) RedirectURL

func (OAuth2) RedirectURL(imp Importer, ctx *SetupContext) string

RedirectURL returns the redirect URI that imp should set in an oauth.Config for the authorization phase of OAuth2 authentication.

type Object

type Object struct {
	// contains filtered or unexported fields
}

An Object is wrapper around a permanode that the importer uses to synchronize.

func (*Object) Attr

func (o *Object) Attr(attr string) string

Attr returns the object's attribute value for the provided attr, or the empty string if unset. To distinguish between unset, an empty string, or multiple attribute values, use Attrs.

func (*Object) Attrs

func (o *Object) Attrs(attr string) []string

Attrs returns the attribute values for the provided attr.

func (*Object) ChildPathObject

func (o *Object) ChildPathObject(path string) (*Object, error)

ChildPathObject returns (creating if necessary) the child object from the permanode o, given by the "camliPath:xxxx" attribute, where xxx is the provided path.

func (*Object) ForeachAttr

func (o *Object) ForeachAttr(fn func(key, value string))

ForeachAttr runs fn for each of the object's attributes & values. There might be multiple values for the same attribute. The internal lock is held while running, so no mutations should be made or it will deadlock.

func (*Object) PermanodeRef

func (o *Object) PermanodeRef() blob.Ref

PermanodeRef returns the permanode that this object wraps.

func (*Object) SetAttr

func (o *Object) SetAttr(key, value string) error

SetAttr sets the attribute key to value.

func (*Object) SetAttrs

func (o *Object) SetAttrs(keyval ...string) error

SetAttrs sets multiple attributes. The provided keyval should be an even number of alternating key/value pairs to set.

type ProgressMessage

type ProgressMessage struct {
	ItemsDone, ItemsTotal int
	BytesDone, BytesTotal int64
}

type RunContext

type RunContext struct {
	*context.Context
	Host *Host
	// contains filtered or unexported fields
}

RunContext is the context provided for a given Run of an importer, importing a certain account on a certain importer.

func (*RunContext) AccountNode

func (rc *RunContext) AccountNode() *Object

AccountNode returns the permanode storing account information for this permanode. It will contain the attributes:

• camliNodeType = "importerAccount"
• importerType = "registered-type"

You must not change the camliNodeType or importerType.

You should use this permanode to store state about where your importer left off, if it can efficiently resume later (without missing anything).

func (*RunContext) Credentials

func (rc *RunContext) Credentials() (clientID, clientSecret string, err error)

Credentials returns the credentials for the importer. This is typically the OAuth1, OAuth2, or equivalent client ID (api token) and client secret (api secret).

func (*RunContext) RootNode

func (rc *RunContext) RootNode() *Object

RootNode returns the initially-empty permanode storing the root of this account's data. You can change anything at will. This will typically be modeled as a dynamic directory (with camliPath:xxxx attributes), where each path element is either a file, object, or another dynamic directory.

type SetupContext

type SetupContext struct {
	*context.Context
	Host        *Host
	AccountNode *Object
	// contains filtered or unexported fields
}

func (*SetupContext) AccountURL

func (sc *SetupContext) AccountURL() string

AccountURL returns the URL to an account of an importer (http://host/importer/TYPE/sha1-sd8fsd7f8sdf7).

func (*SetupContext) CallbackURL

func (sc *SetupContext) CallbackURL() string

func (*SetupContext) Credentials

func (sc *SetupContext) Credentials() (clientID, clientSecret string, err error)